Day 25: Installing GitLab on Your Own Server (Self-Hosted GitLab & Disaster Recovery)

1. Why Do We Need Self-Hosted GitLab? (Problem Statement & Status Quo)

Most modern enterprises and production environments avoid relying exclusively on public cloud SaaS platforms like gitlab.com for their core intellectual property. Instead, organizations run isolated, self-hosted instances of GitLab on their own infrastructure (such as dedicated AWS EC2 instances, private Kubernetes clusters, or on-premise Docker hosts).

Real-Life Analogy: Public Storage Locker vs. Private Bank Vault

Think of gitlab.com as a public storage facility: it's convenient, highly accessible, and managed by someone else, but you share a perimeter with the general public and depend entirely on their security guardrails. Running a self-hosted GitLab server is equivalent to building a private bank vault inside your own custom-built fortress. You control the keys, you decide who enters, your data never leaves your infrastructure, and not even the external software vendor has direct backdoor access to your repositories.

2. Core Architecture & Network Port Mapping Strategy

Before spinning up the Docker container on an Ubuntu EC2 instance, you must carefully plan your port allocation. A standard GitLab installation binds to multiple network ports for HTTP, HTTPS, and SSH traffic.

Port Allocation Breakdown

Infrastructure Flow: Client to Self-Hosted Container

3. EC2 Provisioning and Security Group Configuration

To host a robust GitLab server capable of handling multiple repositories and continuous integration pipelines, we provision an optimized AWS EC2 instance.

• Instance Name: GitLab server
• AMI (Amazon Machine Image): Ubuntu 24.04 LTS (Long Term Support)
• Instance Type: t3.medium (provides adequate CPU and memory headroom for GitLab's Puma web server and Sidekiq background workers)
• Storage: 25 GB EBS root volume
• Security Group Inbound Rules:
  • SSH (22): For admin access to the EC2 machine.
  • HTTP (80): For web traffic.
  • HTTPS (443): For secure web traffic.
  • Custom TCP (2222): For SSH-based Git clone, pull, and push operations.

4. Preparing the Host Environment & Docker Engine

Connect to your EC2 instance via SSH (using MobaxTerm or your preferred terminal) and install Docker. You also need to configure persistent host directories to store GitLab configuration files, logs, and application data outside the container lifecycle.

Step-by-Step Host Setup Commands

# 1. Update package lists and install Docker
sudo apt-get update -y
sudo apt-get install -y docker.io

# 2. Add the current 'ubuntu' user to the Docker group to run commands without sudo
sudo usermod -aG docker ubuntu

# 3. Apply group changes immediately without restarting the session
newgrp docker

# 4. Verify Docker installation
docker ps

# 5. Create persistent directories for GitLab configuration, logs, and data
sudo mkdir -p /srv/gitlab
export GITLAB_HOME=/srv/gitlab

# 6. Persist GITLAB_HOME environment variable across future terminal sessions
echo "export GITLAB_HOME=/srv/gitlab" >> ~/.bashrc
source ~/.bashrc

5. Custom Domain Configuration via AWS Route 53 & Namecheap

To make your self-hosted GitLab instance accessible via a professional URL (e.g., gitlab.myapp.art) rather than a raw IP address, delegate your domain's DNS management to AWS Route 53.

DNS Delegation Workflow

1. Purchase Domain: Acquire a cheap domain from a registrar like Namecheap (e.g., .art domains for ~$2).
2. Create AWS Route 53 Hosted Zone:
   • Navigate to AWS Route 53 -> Hosted Zones -> Create Hosted Zone.
   • Enter your domain name (e.g., myapp.art) and select Public hosted zone.
3. Update Namecheap Custom DNS:
   • Copy the 4 AWS Route 53 NS (Name Server) records generated in your hosted zone.
   • Go to your domain registrar (Namecheap), select Custom DNS, and paste the 4 AWS name servers.
4. Create DNS A-Record:
   • Inside your Route 53 hosted zone, click Create record.
   • Record name: gitlab (resulting in gitlab.myapp.art).
   • Record type: A
   • Value: Paste the Public IP of your EC2 instance.
   • Click Create records.

6. Deploying the GitLab Omnibus Docker Container

With Docker installed, directories created, and DNS propagated, deploy the official GitLab Omnibus Docker container.

The Complete Docker Run Command

sudo docker run --detach \
  --hostname gitlab.myapp.art \
  --env GITLAB_OMNIBUS_CONFIG="external_url 'http://gitlab.myapp.art';" \
  --publish 443:443 --publish 80:80 --publish 2222:22 \
  --name gitlab \
  --restart always \
  --volume $GITLAB_HOME/config:/etc/gitlab \
  --volume $GITLAB_HOME/logs:/var/log/gitlab \
  --volume $GITLAB_HOME/data:/var/opt/gitlab \
  --shm-size 256m \
  gitlab/gitlab-ce:latest

Parameter Breakdown

• --detach: Runs the container in the background.
• --hostname: Sets the internal container hostname.
• --env GITLAB_OMNIBUS_CONFIG: Injects configuration parameters into GitLab's internal gitlab.rb configuration file on startup.
• --publish: Maps host ports (443, 80, 2222) to container ports (443, 80, 22). Note the translation of port 2222 to container port 22.
• --volume: Mounts host directories into the container to ensure data persistence even if the container is destroyed.
• --shm-size 256m: Allocates shared memory to prevent database performance bottlenecks.

7. Initial Login, Project Creation, and SSH Verification

Once the container initializes (which takes 3 to 5 minutes), verify its status and log in for the first time.

1. Retrieve the Initial Root Password

GitLab automatically generates a secure initial password for the root user and stores it inside the container temporarily. Retrieve it using:

sudo docker exec -it gitlab grep 'Password:' /etc/gitlab/initial_root_password

2. Access the Web UI

Open your browser and navigate to http://gitlab.myapp.art. Log in with:

• Username: root
• Password: <paste-retrieved-password-here>

3. Create a Demo Repository

1. Click Create new project -> Create blank project.
2. Name the project demo git repo and set it to Public. Click Create project.

4. Configure SSH Keys for Git Operations

Generate an SSH key pair on your local machine or server, then add the public key to your GitLab profile (User Settings -> SSH Keys):

ssh-keygen -t rsa -b 4096
cat ~/.ssh/id_rsa.pub

Copy the output and paste it into GitLab under SSH Keys.

5. Cloning and Pushing Code via Port 2222

When cloning repositories over SSH from a self-hosted server running on a non-standard port (2222), you must specify the custom port in the Git URL format:

# Test SSH connectivity
ssh -T -p 2222 git@gitlab.myapp.art
# Expected output: Welcome to GitLab, @root!

# Clone the repository using custom port 2222
git clone ssh://git@gitlab.myapp.art:2222/root/demo-git-repo.git

# Make changes, commit, and push
cd demo-git-repo
echo "Aditya is teaching GitLab" > aditya.txt
git add aditya.txt
git commit -m "Adding aditya.txt file"
git push origin main

Refresh your browser repository view to confirm aditya.txt appears successfully.

8. Backup and Disaster Recovery Workflow

Production systems require a reliable disaster recovery strategy. If your EC2 instance or Docker container is accidentally deleted, you must be able to restore both application data (repositories/users) and configuration files (gitlab.rb, secrets).

Step A: Creating Backups Inside the Container

Execute the built-in backup utility inside the running container:

sudo docker exec -it gitlab gitlab-backup create

This generates a .tar backup archive inside the container at /var/opt/gitlab/backups/.

Step B: Copying Backups and Configurations to the Host Machine

Because containers are ephemeral, copy the backups and configuration files from the container to your persistent host machine (/srv/gitlab):

# 1. Copy the application backup tar file to host data backup directory
sudo docker cp gitlab:/var/opt/gitlab/backups/. /srv/gitlab/data_backups/

# 2. Copy the critical configuration files (gitlab.rb, secrets.json, ssh keys) to host config directory
sudo docker cp gitlab:/etc/gitlab/. /srv/gitlab/config_backup/

Note: Critical files preserved in the configuration backup include gitlab.rb, gitlab-secrets.json, SSH host keys, and SSL certificates. Without gitlab-secrets.json, encrypted database tokens cannot be decrypted after restoration.

9. Simulating Failure and Restoring Data

To prove our disaster recovery workflow, we simulate a catastrophic failure by wiping all container volumes and recreating the environment from scratch.

1. Simulate Complete Data Loss

# Stop and remove the running container
sudo docker stop gitlab
sudo docker rm gitlab

# Simulate total data deletion by wiping the mounted host volume
sudo rm -rf /srv/gitlab/data/*

2. Recreate Fresh Directories and Restore Configurations

# Recreate empty directories
sudo mkdir -p /srv/gitlab/config /srv/gitlab/logs /srv/gitlab/data_backups

# Restore configuration files back to the config mount point
sudo cp -r /path/to/saved/config_backup/* /srv/gitlab/config/

3. Restore the Application Backup

# 1. Recreate the fresh GitLab container using the same configuration and volumes
sudo docker run --detach \
  --hostname gitlab.myapp.art \
  --env GITLAB_OMNIBUS_CONFIG="external_url 'http://gitlab.myapp.art';" \
  --publish 443:443 --publish 80:80 --publish 2222:22 \
  --name gitlab \
  --restart always \
  --volume $GITLAB_HOME/config:/etc/gitlab \
  --volume $GITLAB_HOME/logs:/var/log/gitlab \
  --volume $GITLAB_HOME/data:/var/opt/gitlab \
  --shm-size 256m \
  gitlab/gitlab-ce:latest

# 2. Wait 3-5 minutes for initialization, then copy the backup tar into the new container's backup directory
sudo docker cp /srv/gitlab/data_backups/<TIMESTAMP>_gitlab_backup.tar gitlab:/var/opt/gitlab/backups/

# 3. Fix file ownership permissions so the internal 'git' user can read the archive
sudo docker exec -it gitlab chown -R git:git /var/opt/gitlab/backups/

4. Execute the Database and Repository Restoration Process

During restoration, Puma and Sidekiq background workers must be stopped to prevent active database connections from interfering with data integrity.

# 1. Stop Puma and Sidekiq inside the container
sudo docker exec -it gitlab gitlab-ctl stop puma
sudo docker exec -it gitlab gitlab-ctl stop sidekiq

# 2. Reconfigure GitLab to apply restored configuration files
sudo docker exec -it gitlab gitlab-ctl reconfigure

# 3. Run the restore command (Note: Provide the timestamp prefix only, omitting '_gitlab_backup.tar')
sudo docker exec -it gitlab gitlab-backup restore BACKUP=<TIMESTAMP>
# (Confirm 'yes' when prompted to override database data)

# 4. Reconfigure and restart all GitLab services
sudo docker exec -it gitlab gitlab-ctl reconfigure
sudo docker exec -it gitlab gitlab-ctl restart

5. Resetting Root Password Post-Restoration

After restoring an older database backup, your previous root session password might reject login attempts due to token mismatches. Reset the root password directly inside the container:

sudo docker exec -it gitlab gitlab-rails console

Inside the Ruby Rails console prompt (irb:), execute:

user = User.find_by(username: 'root')
user.password = 'NewSecurePassword123!'
user.password_confirmation = 'NewSecurePassword123!'
user.save!
exit

Log back into http://gitlab.myapp.art using user root and your new password. Your repositories, users, and commit history will be fully restored.

10. Summary and Key Takeaways

• Enterprise Security Model: Self-hosted GitLab instances eliminate reliance on public SaaS providers (gitlab.com), ensuring complete data privacy and internal governance within corporate infrastructure.
• Strategic Port Mapping: When running Dockerized GitLab alongside host SSH services, map container port 22 to an alternative host port such as 2222.
• DNS Delegation: Using AWS Route 53 Hosted Zones tied to external registrars (like Namecheap) allows custom, professional internal/external domain routing (e.g., gitlab.myapp.art).
• Persistent Volume Mounts: Mounting host directories (/etc/gitlab, /var/log/gitlab, /var/opt/gitlab) ensures configuration and application data survive container termination.
• Omnibus Backup Engine: GitLab provides built-in utilities (gitlab-backup create and gitlab-backup restore) to package repositories, databases, and attachments into portable archives.
• Configuration & Secret Preservation: Disaster recovery requires backing up both application data (.tar archives) and critical configuration files (gitlab.rb, gitlab-secrets.json), without which database decryption fails.
• Service Interruption During Restore: Restoring database backups requires temporarily halting background workers (puma and sidekiq) to prevent race conditions during table truncation.
• Credential Recovery: If authentication fails after restoring an older database state, administrative credentials can be reliably regenerated using the gitlab-rails console.