ℹ️

This page is part of a larger series on learning Infrastructure-as-Code (IaC) using Proxmox Virtual Environment. Click here to be taken back to the project home page.

Previous Step

Why GitLab?

If the whole premise of Infrastructure-as-Code is to define our IT infrastructure — and its corresponding baselines, configurations, policies, and more — as code, then it stands to reason that we'll want a Git server to commit said code and track changes to it over time.

💡

Having set up an Infisical server in a previous step, GitLab CE makes the most sense for the home lab, because Infisical and GitLab can establish an OIDC trust, allowing GitLab CE to sign JWT and inject into GitLab runners to pull secrets using ephemeral tokens.

Also, self-hosting our own Git server has the added benefit of complete data ownership and privacy.

GitOps Diagram

^{Click here to view this diagram in a new tab}

GitLab CE

Create the VM

GitLab installation requirements | GitLab Docs

Prerequisites for installation.

GitLab Docs

Reading through the guidance here, the specs I'll start with are:

Latest Debian release
80 GB disk
8 vCPU
16 GB RAM

As mentioned in the documentation, you can configure GitLab to run in resource-restricted environments.

💡

In a previous step, we created a Debian 13 template VM that I'll clone and adjust the resource allocations.

Add some extra tags that I can use for automation purposes

Make any changes and click "Regenerate Image"

✅

Give the VM a DHCP reservation if desired.

You may now power on the VM and complete the Debian installation process. I'll only be showing a couple of screenshots from setup below.

Install GitLab Community Edition

Install the Linux package on Debian | GitLab Docs

Install the Linux package on Debian

GitLab Docs

curl --location "https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh" | sudo bash

sudo apt install gitlab-ce

Generate ACME Certificate

sudo apt install -y certbot python3-certbot-nginx nginx

sudo certbot certonly \
  --nginx \
  --server https://sub-ca.pki.home.internal/acme/acme@lab.home.internal/directory \
  --domain gitlab-ce.lab.home.internal \
  --email admin@lab.home.internal \
  --agree-tos \
  --non-interactive

We'll be disabling TCP port 80 on GitLab, so that it doesn't interfere with Nginx

Configure TLS on GitLab

sudo nano /etc/gitlab/gitlab.rb

external_url 'https://gitlab-ce.lab.home.internal'
# ...
# ...
registry_external_url 'https://gitlab-ce.lab.home.internal:5050'
gitlab_rails['registry_enabled'] = true
registry['enable'] = true
# ...
# ...
nginx['redirect_http_to_https'] = false
# ...
# ...
nginx['ssl_certificate'] = "/etc/letsencrypt/live/gitlab-ce.lab.home.internal/fullchain.pem"
nginx['ssl_certificate_key'] = "/etc/letsencrypt/live/gitlab-ce.lab.home.internal/privkey.pem"
# ...
# ...
registry_nginx['ssl_certificate'] = "/etc/letsencrypt/live/gitlab-ce.lab.home.internal/fullchain.pem"
registry_nginx['ssl_certificate_key'] = "/etc/letsencrypt/live/gitlab-ce.lab.home.internal/privkey.pem"

Disable HTTP->HTTPS redirect to disable port 80, which conflicts with certbot

💡

In the change above, we've also enabled the GitLab container registry and bound it to TCP port 5050, using the same TLS certificate as the GitLab frontend.

sudo gitlab-ctl reconfigure

ℹ️

You can sign in with the username root and the password found in /etc/gitlab/initial_root_password. If the password here doesn't work, reset it following the documentation here.

Add User Account

Log into GitLab CE as the root user. Go to Admin area > Overview > Users > Click New user.

🚨

There is no SMTP configured to allow GitLab to send emails for account creation and password reset. We can reset it using the rails console as shown here.

sudo gitlab-rails console

user = User.find_by(email: 'username@domain.tld')

new_password = ::User.random_password
new_password # Print the password
user.password = new_password
user.password_confirmation = new_password
user.password_automatically_set = false
user.save!

✅

You are now ready to sign in using your new account. Save your password in your password manager.

Security and Hardening

Now that you've successfully installed GitLab CE, you should take some time to securely configure and harden your installation. For that task, I'll refer you to the GitLab documentation.

GitLab Hardening Recommendations | GitLab Docs

GitLab product documentation.

GitLab Docs

Infisical OIDC Trust

ℹ️

We'll be using the OpenID Connect authentication to create a trust between GitLab CE and Infisical. This allows GitLab CE to sign JSON Web Tokens (JWT) and inject them into the GitLab Runner. The GitLab Runner then uses the JWT to fetch short-lived secrets from Infisical, as the JWT claims can be verified thanks to this OIDC trust.

Create the Machine Identity for GitLab CE

Select Organization > Access Control > Machine Identities > Create Organization Machine Identity

Expand the menu and click "Remove Auth Method"

💡

The OIDC discovery process works because we have trusted the Intermediate CA certificate bundle on the Infisical server.

The subject and claims both reference the infrastructure group we will create in the next step. The * in the project_path identifier allows for any repository under this group. And, the Runner will executed against commits to the main branch.

Note the Machine Identity "ID" value for use later

Assign the Machine Identity to IaC Project

Open the project and click on "Access Control"

Click "Machine Identities" > "+ Add Machine Identity to Project"

"Assign Existing" > choose the correct machine ID > assign "Viewer"

Open the IaC Project created in an earlier step and note the Project ID for later

CI/CD Configuration

Create Group for IaC Repositories

💡

Using a group for your Infrastructure-as-Code repositories is the best way to handle CI/CD for this project. Since many of the variables will be shared across multiple repositories here, defining them at the group level makes the management overhead much simpler, since we only have to define them in one place.

At the GitLab home page, click "Create a group"

Configure Group Based CI/CD

Add Group Variables

💡

We're going to add some group-scoped variables for use in CI/CD jobs by the GitLab Runner (more on that later). Having it scoped to the group means that we only need to define them once here.

Good balance for variable privilege in case a trusted user is added later, click "Save Changes"

We'll add the following variables for use by the GitLab Runner, which we'll add to the environment in a later step.

INFISICAL_API_URL: https://secrets.lab.home.internal (change this to the URL of your Infisical instance)
- Type: Variable
- Environments: All
- Visibility: Visible
- Flags: ✅Protect variable
- Description: Self-hosted Infisical server
- Key: INFISICAL_API_URL
- Value: https://secrets.lab.home.internal
INFISICAL_IDENTITY_ID: 994axxxx-xxxx-xxxx-xxxx-xxxxxxx392e5 (machine ID as noted before)
- Type: Variable
- Environments: All
- Visibility: Masked
- Flags: ✅Protect variable
- Description: OIDC Machine ID from Infisical
- Key: INFISICAL_IDENTITY_ID
- Value: 994axxxx-xxxx-xxxx-xxxx-xxxxxxx392e5
INFISICAL_PROJECT_ID: bf490xxx-xxxx-xxxx-xxxx-xxxxxxxb9ae1 (Infisical project ID as noted before)
- Type: Variable
- Environments: All
- Visibility: Masked
- Flags: ✅Protect variable
- Description: Infisical "IaC Project" project ID
- Key: INFISICAL_PROJECT_ID
- Value: bf490xxx-xxxx-xxxx-xxxx-xxxxxxxb9ae1

Add Group Runner

Click Build > Runners > "Create group runner"

Restrict the runner to jobs that use only these tags

Set the timeout to 90 minutes in the event of a hung job (can increase on specific pipeline tasks if needed)

Make a note of this command, as we'll register the Runner on separate VM later

Create Group Repositories (Projects)

Under the main view for the group, click "Create Project"

ℹ️

When creating your projects, start them as Blank Projects.

Project 1: runner-images

This project will house the Docker files that identify how each tools is configured — packer, terraform, and ansible. Having these Docker files in their own project allows us to maintain clear version control of our tooling.

💡

We want to disable Auto DevOps on all of our Infrastructure-as-Code projects, as Auto DevOps targets more of an application development workflow, where there will be continuous deployments to production. If left enabled, it will interfere with our build pipelines.

Project settings > CI/CD > Auto DevOps > Disable Auto DevOps > Click "Save changes"

Project 2: packer

Project 3: terraform

Project Settings > CI/CD > Project variables > Add variable

Type: Variable
Environments: All
Visibility: Masked
Flags: ✅Protect variable
Description: GitLab CE URL for Terraform state management
Key: TF_STATE_ADDRESS
Value: https://gitlab-ce.lab.home.internal/api/v4/projects/3/terraform/state/pve-prod

💡

The URL above uses the GitLab project numeric ID -- 3 in my case (as noted above when the project was created).

Project 4: ansible

Project 5: ci-helpers

This project will store a unique pipeline with helper functions that we'll call and extend in other pipelines. Consider it the same as a repository of helper functions.

Project settings > General > "Visibility, project features, pemissions" > Disable CI/CD > Click "Save Changes"

ℹ️

We're turning off the CI/CD feature on this project, since we're only using the .gitlab-ci.yml file here to host helper functions and don't actually expect any code commits to trigger CI/CD functions.

GitLab Runner

Protected Runner

Create the VM

The system requirements for a GitLab Runner VM are highly variable depending on what kind of workload is expected. Since this is a home lab environment and I am going to be the only using it, the workload should be quite light.

💡

We'll be using the Docker executor for the Infrastructure-as-Code CI/CD pipelines. Therefore, using a VM is highly recommended due to greater isolation and simpler version dependency management.

Make a clone of the Debian 13 template VM created in a previous step and modify some settings.

Add some tags after cloning

ℹ️

A two-disk setup is recommended, so that we can easily regrow the Docker Data partition in the event the disk becomes full without having to worry about shifting partitions for the OS.

Correct DNS server for updated VLAN and click "Regenerate Image"

✅

You may now start the VM.

Configure Disk 2 as Docker Data Store

ssh -i privkey.pem debian@gitlab-runner.domain.tld

Use SSH to log into the VM

sudo mkdir -p /var/lib/docker

Create the mountpoint before installing Docker

sudo fdisk -l | grep Disk

sudo apt install -y parted

sudo parted /dev/sdb --script mklabel gpt mkpart primary ext4 0% 100%

Make a primary partition on the disk

sudo mkfs.ext4 /dev/sdb1

Format ext4, takes a few seconds to complete

echo "UUID=$(sudo blkid /dev/sdb1 | cut -d '"' -f 2)  /var/lib/docker  ext4  defaults,noatime 0 2" | sudo tee -a /etc/fstab

Add the "/etc/fstab" mapping by UUID

sudo systemctl daemon-reload

sudo mount -a

Remount all partitions to include our new Docker mount

mount | grep docker

Show Docker volume is mounted

df -h /var/lib/docker

Inspect free space

Install Docker (Debian)

Link to installation documentation for "apt" package manager

sudo usermod -aG docker debian

Add default user to "docker" group

⚠️

Log out and log back in to affect the group changes.

docker info | grep "Docker Root Dir"

Excellent! Docker is using the mount as planned.

💡

Recall from when we set up our templates, we used cloud-init to bake in the Intermediate CA certificate, which is stored at /usr/local/share/ca-certificates/internal-intermediate.crt. This allows the Debian host to trust certificates issued by our internal PKI, but does not cover the Docker engine itself.

sudo mkdir -p "/etc/docker/certs.d/gitlab-ce.lab.home.internal:5050"

Create the certificates store for Docker

sudo cp /usr/local/share/ca-certificates/internal-intermediate.crt \
  "/etc/docker/certs.d/gitlab-ce.lab.home.internal:5050/ca.crt"

Copy the Intermediate CA certificate to a directory matching the target hostname (Docker Container registry on GitLab)

sudo mkdir -p "/etc/docker/certs.d/gitlab-ce.lab.home.internal:443"

sudo cp /usr/local/share/ca-certificates/internal-intermediate.crt \
  "/etc/docker/certs.d/gitlab-ce.lab.home.internal:443/ca.crt"

Copy the Intermediate CA certificate to a directory matching the target hostname (Dependency Proxy on GitLab)

sudo systemctl restart docker

Install and Register the GitLab Runner

Install the Runner

Install GitLab Runner using the official GitLab repositories | GitLab Docs

Install GitLab Runner from a GitLab repository using your package manager.

GitLab Docs

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" -o script.deb.sh

Inspect the script source code before execution

sudo bash script.deb.sh

Adds the apt repositories

sudo apt install -y gitlab-runner

sudo usermod -aG docker gitlab-runner

Add the "gitlab-runner" service account to the Docker group

Register the Runner

sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab-ce.lab.home.internal" \
  --token "glrt-...REDACTED..." \
  --executor "docker" \
  --docker-privileged \
  --docker-image "docker:29" \
  --description "proxmox-iac-runner" \
  --docker-volumes "/certs" \
  --docker-volumes "/etc/docker/certs.d:/etc/docker/certs.d:ro" \
  --docker-volumes "/usr/local/share/ca-certificates/internal-intermediate.crt:/usr/local/share/ca-certificates/internal-intermediate.crt:ro"

Use the command from step above to register the runner

Adjust Runner Configurations

sudo nano /etc/gitlab-runner/config.toml

concurrent = 4

Change "1" to "4" here

  executor = "docker"
  tls-ca-file = "/usr/local/share/ca-certificates/internal-intermediate.crt"
  request_concurrency = 4

Add the "tls-ca-file" line below "executor" to trust the internal PKI-issued certificates Match "request_concurrency" to the "concurrent" option

✅

Save your changes to the file.

sudo systemctl restart gitlab-runner

sudo gitlab-runner verify

Test the configuration

Unprotected and Untagged Runner

Why Another Runner?

We're going to register another runner with GitLab and put it on a separate VLAN for segmentation.

ℹ️

Having a separate runner for unprotected branches and untagged jobs adds an isolation layer to the GitOps setup. What this means is that when code is committed to an unprotected branch or a pipeline job is untagged, then this runner will pick up that job.

Having it in a separate VLAN minimizes the blast radius in the event of compromise.

What If I Just Want One Runner?

Then, edit the runner registered previously and do the following:

✅ Check the box that says run untagged jobs
🔳 Uncheck the box that says protected

This will allow your single runner to pick up jobs in all circumstances. Just realize that there is less separation in the event of compromise.

Server Build Checklist

We'll be following most of the steps from above, but to be clear, I'll recap them here:

Clone the Debian 13 VM template (cloud-init enabled)
- Add tags to VM after cloning (e.g. git, runner)
- Adjust hardware resources
  - Hostname: gitlab-runner-unprotected
  - 4 GB RAM
  - 4 CPU
  - Disk 1: 64 GB (OS)
  - Disk 2: 128 GB (Docker data)
  - VLAN: Whichever VLAN you wish to isolate on
- Firewall rules:
  - Runner -> Intermediate CA on TCP port 443 (certificates)
  - Runner -> GitLab CE on TCP port 443 (job polling)
  - Runner -> GitLab CE on TCP port 5050 (container registry)
Configure Disk 2 as Docker Data Store
Install Docker
Install GitLab Runner

Add Another Group Runner

Tags: validate, lint
Run untagged jobs: ✅
Protected: 🔳
Maximum job timeout: 5400

Note the command to run on the runner for registration

Create the Docker Certificate Store

sudo mkdir -p "/etc/docker/certs.d/gitlab-ce.lab.home.internal:5050"

Create the certificates store for Docker

sudo cp /usr/local/share/ca-certificates/internal-intermediate.crt \
  "/etc/docker/certs.d/gitlab-ce.lab.home.internal:5050/ca.crt"

Copy the Intermediate CA certificate to a directory matching the target hostname (Docker Container registry on GitLab)

sudo mkdir -p "/etc/docker/certs.d/gitlab-ce.lab.home.internal:443"

sudo cp /usr/local/share/ca-certificates/internal-intermediate.crt \
  "/etc/docker/certs.d/gitlab-ce.lab.home.internal:443/ca.crt"

Copy the Intermediate CA certificate to a directory matching the target hostname (Dependency Proxy on GitLab)

sudo systemctl restart docker

Register the Runner

sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab-ce.lab.home.internal" \
  --token "glrt-...REDACTED..." \
  --executor "docker" \
  --docker-privileged \
  --docker-image "docker:29" \
  --description "unprotected-iac-runner" \
  --docker-volumes "/certs" \
  --docker-volumes "/etc/docker/certs.d:/etc/docker/certs.d:ro" \
  --docker-volumes "/usr/local/share/ca-certificates/internal-intermediate.crt:/usr/local/share/ca-certificates/internal-intermediate.crt:ro"

Update Configurations

sudo nano /etc/gitlab-runner/config.toml

concurrent = 4

Change "1" to "4" here

  executor = "docker"
  tls-ca-file = "/usr/local/share/ca-certificates/internal-intermediate.crt"
  request_concurrency = 4

Add the "tls-ca-file" line below "executor" to trust the internal PKI-issued certificates Match "request_concurrency" to the "concurrent" option

✅

Save your changes to the file.

sudo systemctl restart gitlab-runner

sudo gitlab-runner verify

Test the configuration

Infrastructure-as-Code with Proxmox: GitOps - Scaffolding

Previous Step

Why GitLab?

GitOps Diagram

GitLab CE

Create the VM

Install GitLab Community Edition

Generate ACME Certificate

Configure TLS on GitLab

Initial Sign-In

Add User Account

Security and Hardening

Infisical OIDC Trust

Create the Machine Identity for GitLab CE

Assign the Machine Identity to IaC Project

CI/CD Configuration

Create Group for IaC Repositories

Configure Group Based CI/CD

Add Group Variables

Add Group Runner

Create Group Repositories (Projects)

Project 1: runner-images

Project 2: packer

Project 3: terraform

Project 4: ansible

Project 5: ci-helpers

GitLab Runner

Protected Runner

Create the VM

Configure Disk 2 as Docker Data Store

Install Docker (Debian)

Install and Register the GitLab Runner

Install the Runner

Register the Runner

Adjust Runner Configurations

Unprotected and Untagged Runner

Why Another Runner?

What If I Just Want One Runner?

Server Build Checklist

Add Another Group Runner

Create the Docker Certificate Store

Register the Runner

Update Configurations

Next Step

0xBEN

Infrastructure-as-Code with Proxmox

Infrastructure-as-Code with Proxmox: Manual Templating

Infrastructure-as-Code with Proxmox: Dynamic DNS (DDNS)