Previous Step

Infrastructure-as-Code with Proxmox: GitOps - Dockerize Tools

In this module, we’ll template some Dockerfiles and define the build requirements for our packer, terraform, and ansible Docker images. We’ll merge into main and the pipeline will deploy the containers to the Container Registry.

0xBEN0xBEN

Quick Recap

In the previous module, we added onto the GitLab environment by:

• Defining a set of Dockerfile files in the runner-images project, specifically under the packer/, terraform, and ansible/ directories
• The purpose of the Dockerfile definitions is to define how we want build a container to run packer, terraform, and ansible, as well as any dependencies we want to package in:
  • Our internal PKI Intermediate CA certificate
  • Infisical CLI binary
  • Additional binaries required for operations
• Then, use docker build and docker push to create and store the Docker containers in our GitLab Container Registry under the runner-images project
• Then, we tested OIDC authentication by:
  • Downloading the packer image
  • Using the infisical CLI inside the packer image to ensure we could use our JWT to retrieve an access token
• Finally, we set up a CI/CD pipeline — .gitlab-ci.yml — inside the runner-images project to ensure that any changes to the Dockerfile definitions or the internal-ca.crt file would build and store the packer, terraform and/or ansible images using the latest and x.xx.x verison tag in the container registry

Packer Overview

When IT teams want to deploy hosts, they want to ensure that it conforms to specific policies, configurations, and security best practices. In order to do this, they create what is known as Golden Images.

In the past, teams would create golden images manually for different environments using specific tools for the job. They might create an AWS AMI, Azure AMI, GCP Images, or even images for their on-premise platforms such as Proxmox. Images might experience "configuration drift' and keeping them updated was time-consuming.

By keeping the configurations in source control — such as GitLab — teams can track changes and ensure that images are built consistently from the same code base.

Pipeline Diagram

Development Strategy

Local Iterative Refinement

Before we even git commit any code to the packer repository, we'll be spending most of the time locally performing iterative improvements on our build templates. The flow would go something like this.

1. git clone the packer repository
2. Add directory hierarchy
3. Add files and source code
4. Pull the docker image from GitLab Container Registry
5. Start an ephemeral container and map the directory as a volume
6. Run packer init inside a build directory
7. Run packer fmt to parse and normalize Packer files
8. Run packer validate inside a build directory
9. Run packer build inside a build directory
10. Observe any errors, bugs, or other issues during the build process and continue refining the template
11. Inspect the entire directory tree and plan a .gitignore file for things we do not want to commit to GitLab

Pipeline Builds in Production

The reason we packer build locally first before the pipeline is that we'll almost certainly be spending time fixing issues with boot_command parameters, boot_wait parameters, and other such things.

Once packer build runs successfully locally, we can let the pipeline run packer build and be the responsible party for production builds. With the pipeline doing packer build, we ensure build consistency by ensuring the production build environment is consistent and repeatable.

My Development Environment

Remote Development Box

To setup a consistent development environment, I did the following:

• Cloned my cloud-init enabled Debian template VM
• Put it on the correct VLAN and updated the cloud-init config
• Ensured my SSH key was working for authentication

The benefits to doing this is that I have a development environment where all of my tools are installed. I can access this box remotely via VPN as well.

Connect via SSH

Clone the Packer Repository

GitLab SSH Authentication | 0xBEN | Notes

Generate SSH Keypair ssh-keygen -t ed25519 -C “gitlab-user@domain.tld” -f ~/.ssh/git-ssh-key Enter y…

0xBEN | Notes

mkdir -p ~/Code/IaC_Project

cd ~/Code/IaC_Project

git clone git@gitlab-ce.lab.home.internal:infrastructure/packer.git

cd packer

Open the Packer Repository

Create a Feature Branch for New Code

git checkout -b add-initial-files

Developing Packer Templates

If you have a look around on your favorite search engine or GitHub, one thing you'll quickly realize is that there is no one uniform way to create a Packer template.

A good amount of thought has to go into the planning and continued upkeep of the code repository, but one thing is certain — don't aim for perfection on your first go through. Start with a good baseline and continue to iteratively improve.

Some basic things you'll want to start thinking about

• Which operating systems to template
• How to structure variables:
  • Which variables will go in .pkrvars.hcl file(s)
  • Which variables will be pulled from Infisical (or other secrets management)
  • Will there be GitLab group / project variables
• How to keep code DRY and create re-usable code where possible

General procedure for getting started

1. Download a target .iso file of choice to your Proxmox VE node
2. Set up a code repository structure
   1. Directories for build templates
   2. Directories for scripts
   3. Etc
3. Begin writing the Packer template and variables
4. Develop an answer file for your target OS
   1. Debian: preseed.cfg
   2. Ubuntu: autoinstall.yaml
   3. RedHat: ks.cfg — Kickstart
   4. Windows: autounattend.xml
5. Begin writing any provisioning scripts

Repository Structure

packer/
|---- .gitlab-ci.yml                                                   # GitLab pipeline configuration
|---- .gitignore                                                       # List of directories and/or files to keep out of remote repository
'---- proxmox/
      |---- builds/
      |      |---- plugins.pkr.hcl                                     # Shared plugins for all builds
      |      |---- linux/
      |      |      |---- variables.pkr.hcl                            # Shared variables for Linux builds, define expected inputs
      |      |      |---- debian/
      |      |      |      |---- debian-13/
      |      |      |      |      |---- debian-13-vm.auto.pkrvars.hcl  # Populates variables defined in variables.pkr.hcl (Infisical will populate others)
      |      |      |      |      |---- debian-13-vm.pkr.hcl           # The actual infrastructure-as-code for the VM
      |      |      |      |      '---- preseed.pkrtpl.hcl             # Template for preseed.cfg file, allowing for variable substitution (no hard-coded values)
      |      |      |      |
      |      |      |      '---- debian-12/
      |      |      |             '---- .gitkeep                       # Empty file, but ensures the directory is committed to repo
      |      |      |
      |      |      '---- ubuntu/
      |      |             '---- ubuntu-2404/
      |      |                    '---- .gitkeep                       # Empty file, but ensures the directory is committed to repo
      |      |      
      |      '---- windows/
      |             '---- 11-enterprise/
      |                     '---- .gitkeep
      |
      |---- files/                                                     # Directory to store any static files required for configuration
      |      '---- .gitkeep
      |
      '---- scripts/
             |---- powershell/
             |       '---- .gitkeep
             '---- bash/
                     |---- post_install_cleanup.sh
                     '---- post_install_dhcp_fix.sh

Create the Core Structure

cd ~/Code/IaC_Project/packer

touch .gitignore

touch .gitlab-ci.yml

mkdir -p proxmox/builds/linux/debian/{debian-12-vm,debian-13-vm}

mkdir -p proxmox/builds/linux/ubuntu/ubuntu-2404-vm

mkdir -p proxmox/builds/windows/11-enterprise-vm

mkdir -p proxmox/files

mkdir -p proxmox/scripts/{bash,powershell}

touch proxmox/builds/plugins.pkr.hcl

touch proxmox/builds/linux/variables.pkr.hcl

touch proxmox/builds/linux/debian/debian-12-vm/.gitkeep

touch proxmox/builds/linux/debian/debian-13-vm/{debian-13-vm.auto.pkrvars.hcl,debian-13-vm.pkr.hcl,preseed.pkrtpl.hcl}

touch proxmox/builds/linux/ubuntu/ubuntu-2404-vm/.gitkeep

touch proxmox/builds/windows/11-enterprise-vm/.gitkeep

touch proxmox/files/.gitkeep

touch proxmox/scripts/bash/{post_install_cleanup.sh,post_install_dhcp_fix.sh}

touch proxmox/scripts/powershell/.gitkeep

Debian 13 VM

Defining Expected Inputs

Recall that Packer will automatically process any .pkr.hcl file it finds in the working directory. You can name the file anything you'd like, but I'm naming it variable.pkr.hcl. This file serves the purpose of:

• Telling Packer what variable names to expect
• Telling Packer what kinds of input the variables should accept – e.g. string, number, etc
• Defining any default values that should be assigned to those variables in the event none are supplied

cd ~/Code/IaC_Project/packer/proxmox/builds/linux/debian/debian-13-vm

ln -s ../../variables.pkr.hcl .

# ----- Infisical Variables: Begin ----- #

/*
Infisical will inject authentication variables into environment
    - PROXMOX_USERNAME : Proxmox plugin automatically discovers this in environment variables
    - PROXMOX_TOKEN    : Proxmox plugin automatically discovers this in environment variables
        - Unprotected GitLab runner won't have access to Infisical
        - So, we'll set them as empty environment variables in the pipeline configuration
*/

# PROTECTED
variable "ssh_password" {
  # Injected by Infisical as "PKR_VAR_packer_build_password"
  # We will hash this on the fly using Packer bcrypt() function
  type        = string
  sensitive   = true
  description = "Password SSH login as template provisioning user during Packer builds"
  default     = ""
}

# UNPROTECTED
variable "ssh_username" {
  # Injected by Infisical as "PKR_VAR_ssh_username"
  # Default "placeholder" to pass validation in unprotected pipeline runner (no Infisical access)
  type        = string
  description = "Username injected into preseed.cfg for template provisioning"
  default     = "placeholder"
}

# ----- Infisical Variables: End ----- #

/* Local Variables: Begin
.-----------------------------------------------.
| Locally Sourced Variables: *.auto.pkrvars.hcl |
|     We can commit these to Git, cause         |
|     there are no protected variables          |
'-----------------------------------------------'
*/

# Node and Storage
variable "proxmox_node" {
  type        = string
  description = "The node to build the template on"
  default     = "pve"
}
variable "proxmox_node_domain" {
  type    = string
  default = "lab.home.internal"
}
variable "iso_storage_pool" {
  # Defaults to "local", which would match a default Proxmox VE installation
  type    = string
  default = "local"
}
variable "vm_disk_storage_pool" {
  # Defaults to "local-lvm", which would match a default Proxmox VE installation
  type    = string
  default = "local-lvm"
}
variable "vm_tags" {
  type        = string
  description = "Semicolon-separated list of tags to add to the VM"
  default     = ""
}
variable "resource_pool" {
  type        = string
  description = "The resource pool in Proxmox VE logical grouping of units"
  default     = ""
}

# Hardware
variable "vm_cpu" {
  # Per the documentation, the default CPU on the CLI is "kvm64", in the UI is "x86-64-v2-AES"
  # Requires a Proxmox VE node with recent chipsets, but shouldn't be a problem for most
  type        = string
  description = "The CPU type to use when creating a VM, will default to UI setting"
  default     = "x86-64-v2-aes"
}
variable "vm_id" {
  # 10000 -- 19999 for VM templates
  type        = number
  description = "The numerical ID to assign to the VM"
}
variable "vm_name" {
  type    = string
  default = "debian-13-template"
}
variable "cores" {
  type    = number
  default = 2
}
variable "memory" {
  type    = number
  default = 2048
}
variable "disk_size" {
  type    = string
  default = "20G"
}

# Network
variable "switch_name" {
  type        = string
  description = "The name of the network bridge to use"
  default     = "vmbr0"
}
variable "vlan_tag" {
  type    = number
  default = null
}
variable "dhcp_timeout_seconds" {
  type        = number
  description = "The number of seconds to wait before skipping DHCP"
  default     = 30
}

# Operating System
variable "iso_file_name" {
  type        = string
  description = "Example: debian-13-netinst.iso"
}

# Locale and Encoding
variable "system_keyboard" {
  type    = string
  default = "us"
}
variable "system_locale" {
  type    = string
  default = "en_US.UTF-8"
}
variable "system_time_zone" {
  type    = string
  default = "UTC"
}

# Packages
variable "packages_to_install" {
  type        = string
  description = "Space-separated list of packages to install"
  default     = "qemu-guest-agent cloud-init sudo curl"
}

Setting the Inputs

Recall that Packer will automatically process any .auto.pkrvars.hcl files it finds in the working directory. If discovered, it will use these key=value pairs to automatically assign values to any variables defined in configuration files (such as variables.pkr.hcl for example).

# Only define variables not exported by Infisical here
proxmox_node         = "proxmox" # Hostname of the target server
proxmox_node_domain  = "lab.home.internal"
iso_storage_pool     = "ISO"         # "local" is the default in most PVE installs
vm_disk_storage_pool = "Guest_Disks" # the default in most PVE installs
vm_id                = 10001         # 10000 -- 19999 for VM templates
vm_name              = "debian-13-packer-template"
vm_cpu               = "x86-64-v2-aes"
vm_tags              = "packer-template;vm;linux;debian;trixie"
resource_pool        = "packer-templates" # The resource pool as created in initial user / group script
cores                = 2
memory               = 2048
disk_size            = "20G"
iso_file_name        = "debian-13.4.0-amd64-netinst.iso"
system_keyboard      = "us"
system_locale        = "en_US.UTF-8"
system_time_zone     = "UTC"
switch_name          = "vmbr0"
vlan_tag             = 302 # You can delete this variable entirely if you don't use VLANs
dhcp_timeout_seconds = 30
packages_to_install  = "qemu-guest-agent cloud-init sudo curl binutils"

Build the Preseed Answer File

The .pkrtpl.hcl file extension allows the Packer templatefile() function to parse this file and do variable substitution on ${var_name} using any variables defined in variable files, environment variables, or variables defined using the -var switch at build time.

If you look to the source code below, you'll see examples of this such as:

• ${system_locale} — which comes from the .auto.pkrvars.hcl file
• ${ssh_username} — which comes from Infisical-sourced PKR_VAR_ssh_username

This keeps secrets out of the code, one less thing to worry about when we git commit.

# Disable scanning of additional installation media
# |--- 1. Force apt to ignore any CD-ROM as installation media
d-i apt-setup/disable-cdrom-entries boolean true
d-i apt-setup/cdrom/set-first       boolean false
d-i apt-setup/cdrom/set-next        boolean false
d-i apt-setup/cdrom/set-failed      boolean false
# '--- 2. partman/early_command runs at the opportune time to bring device offline
d-i partman/early_command string \
  echo offline > /sys/block/sr1/device/state || true

# Localization & Keyboard
d-i debian-installer/locale string ${system_locale}
d-i keyboard-configuration/xkb-keymap select ${system_keyboard}

# Network
d-i netcfg/choose_interface select auto
d-i netcfg/dhcp_timeout string ${dhcp_timeout_seconds}
d-i netcfg/get_hostname string unassigned-hostname
d-i netcfg/get_domain string unassigned-domain

# Mirror settings
d-i apt-setup/use_mirror boolean true
d-i mirror/country string manual
d-i mirror/http/hostname string deb.debian.org
d-i mirror/http/directory string /debian
d-i mirror/http/proxy string

# Account setup (Variables injected by Packer)
d-i passwd/root-login boolean false
d-i passwd/user-fullname string ${ssh_username}
d-i passwd/username string ${ssh_username}
d-i passwd/user-password-crypted password ${ssh_password_hash}

# Timezone and Clock Setup
d-i time/zone string ${system_time_zone}
d-i clock-setup/utc boolean true
d-i clock-setup/utc-auto boolean true

# Partitioning (Regular, no LVM, max out the disk)
d-i partman-auto/method string regular
d-i partman-auto/choose_recipe select atomic
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true

# Package selection (Standard system + SSH + Cloud-Init + Guest Agent)
tasksel tasksel/first multiselect standard, ssh-server
d-i pkgsel/include string ${packages_to_install}
popularity-contest popularity-contest/participate boolean false

# Boot loader
d-i grub-installer/only_debian boolean true
d-i grub-installer/with_other_os boolean true
d-i grub-installer/bootdev  string default

# Grant passwordless sudo to the packer user so provisioners can run
d-i preseed/late_command string \
    echo '${ssh_username} ALL=(ALL) NOPASSWD: ALL' > /target/etc/sudoers.d/${ssh_username}; \
    in-target chmod 440 /etc/sudoers.d/${ssh_username}

# Finish
d-i finish-install/reboot_in_progress note

Infrastructure-as-Code

This is the actual template file where we define the parameters that will be used to build the infrastructure — the virtual machine itself — in Proxmox VE using the proxmox-iso source.

source "proxmox-iso" "debian-13" {
  # Infisical will inject authentication variables into environment
  #     - PROXMOX_USERNAME : Proxmox plugin automatically discovers this in environment variables
  #     - PROXMOX_TOKEN    : Proxmox plugin automatically discovers this in environment variables

  # Connection  
  proxmox_url              = "https://${var.proxmox_node}.${var.proxmox_node_domain}:8006/api2/json"
  insecure_skip_tls_verify = true
  node                     = var.proxmox_node

  # VM General
  vm_id                = var.vm_id
  vm_name              = var.vm_name
  cpu_type             = var.vm_cpu
  tags                 = var.vm_tags
  pool                 = var.resource_pool
  template_description = "Debian 13 Template. Built by Packer on ${formatdate("YYYY-MM-DD hh:mm", timestamp())}"

  # OS & Boot
  # Ensure you're targeting the correct PVE node where this .iso lives
  boot_iso {
    type     = "ide"
    index    = "0"
    iso_file = "${var.iso_storage_pool}:iso/${var.iso_file_name}"
    unmount  = true
  }
  os         = "l26"
  qemu_agent = true

  # Hardware
  cores           = var.cores
  memory          = var.memory
  sockets         = 1
  scsi_controller = "virtio-scsi-pci"

  disks {
    disk_size    = var.disk_size
    format       = "raw"
    storage_pool = var.vm_disk_storage_pool
    type         = "scsi"
  }

  network_adapters {
    bridge   = var.switch_name
    model    = "virtio"
    vlan_tag = var.vlan_tag
  }

  # Cloud-Init Drive Configuration
  cloud_init              = true
  cloud_init_storage_pool = var.vm_disk_storage_pool

  # Uses the "templatefile()" function to substitute variables in the ".pkrtpl.hcl" file
  additional_iso_files {
    type             = "ide"
    index            = "2"
    cd_label         = "PRESEED"
    iso_storage_pool = var.iso_storage_pool
    unmount          = true
    cd_content = {
      "preseed.cfg" = templatefile("${path.root}/preseed.pkrtpl.hcl", {
        system_locale        = var.system_locale
        system_keyboard      = var.system_keyboard
        dhcp_timeout_seconds = var.dhcp_timeout_seconds
        ssh_username         = var.ssh_username
        ssh_password_hash    = bcrypt(var.ssh_password, 10)
        system_time_zone     = var.system_time_zone
        packages_to_install  = var.packages_to_install
      })
    }
  }

  boot_wait = "5s"
  /*
  - Debian doesn't auto-mount the second .iso file containing the preseed.cfg file
  - So, we have to do some unfortunately hacky stuff to self-mount and copy the preseed.cfg file
  - This is also combined with some configurations in the preseed.pkrtpl.hcl file
  - See the top section of the file where we use some commands
    - d-i apt-setup -- ignores the second .iso file as installation media
    - d-i partman/eary_command -- brings /dev/sr1 offline to make in unavailable for scanning
  */
  boot_command = [
    "<esc><wait>",
    "install auto=true priority=critical ",
    format("preseed/early_command=\"%s\" ", join("; ", [
      "modprobe isofs",
      "mkdir -p /tmp/preseed",
      "mount /dev/sr1 /tmp/preseed",
      "cp /tmp/preseed/preseed.cfg /tmp/preseed.cfg",
      "umount -f /tmp/preseed"
    ])),
    "preseed/url=file:///tmp/preseed.cfg ",
    "locale=${var.system_locale} ",
    "keymap=${var.system_keyboard} ",
    "<enter>"
  ]

  # Provisioning Connection
  ssh_username = var.ssh_username
  ssh_password = var.ssh_password
  ssh_timeout  = "20m"
}

build {
  # Follows the source "proxmox-iso" "debian-13" declaration above
  sources = [
    "source.proxmox-iso.debian-13"
  ]

  provisioner "shell" {
    # The relative path walks up 4 directories to the root 'proxmox' folder, 
    # then back down into 'scripts/bash'
    script          = "../../../../scripts/bash/post_install_dhcp_fix.sh"
    execute_command = "chmod +x {{ .Path }}; sudo bash -c '{{ .Vars }} {{ .Path }}'"
  }

  provisioner "shell" {
    script          = "../../../../scripts/bash/post_install_cleanup.sh"
    execute_command = "chmod +x {{ .Path }}; sudo bash -c '{{ .Vars }} {{ .Path }}'"
  }
}

Create Provisioning Scripts

For this part, we'll create a couple of bash scripts at packer/proxmox/scripts/bash/ to run some tasks as the final step.

echo 'Adding Systemd unit to restart networking service after cloud-init'

sudo tee /etc/systemd/system/dhcp-hostname-sync.service > /dev/null << EOF
[Unit]
Description=Restart DHCP discover after hostname set
After=cloud-init.service
Before=cloud-config.service

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'systemctl restart networking.service'
RemainAfterExit=yes

[Install]
WantedBy=cloud-config.service
EOF

echo 'Enabling Systemd service: dhcp-hostname-sync'

sudo systemctl enable dhcp-hostname-sync.service

#!/bin/bash
set -e

echo 'Cleaning up Cloud-Init...'
sudo cloud-init clean --logs --machine-id --seed --configs all

echo 'Cleaning up the system...'
sudo apt-get clean
sudo rm -rf /var/lib/apt/lists/*

# The machine-id must be empty so it's regenerated on clone (crucial for DHCP)
sudo truncate -s 0 /etc/machine-id
[ -f /var/lib/dbus/machine-id ] && sudo rm /var/lib/dbus/machine-id
sudo ln -s /etc/machine-id /var/lib/dbus/machine-id

# Wipe SSH keys so the template doesn't share them
sudo rm -f /etc/ssh/ssh_host_*

# Clear logs
sudo find /var/log -type f -exec truncate -s 0 {} \;

Linking the Shared Plugins

cd ~/Code/IaC_Project/packer/proxmox/builds/linux/debian/debian-13-vm/

ln -s ../../../plugins.pkr.hcl .

packer {
  required_plugins {
    proxmox = {
      version = ">= 1.2.1"
      source  = "github.com/hashicorp/proxmox"
    }
  }
}

Packer Build Testing

Segmenting Dev and Prod

Proxmox VE Setup

When we first set up Infisical, we created some resource pools, groups, service accounts, and API tokens in Proxmox VE. We then saved the API tokens in the prod environment in Infisical.

In the debian-13-vm.auto.pkrvars.hcl file above, we set the input for resource_pool to packer-templates, which will be evaluated to /pool/packer-templates. And, this is fine for packer build running in the pipeline. However, while testing, we should keep those test builds in a separate resource pool.

To accomplish this, we'll run another script in a Proxmox VE node to create:

• A new Proxmox VE group for testing
• A new Proxmox VE resource pool for testing
• A new Proxmox VE service account and assign to the test group
• A new Proxmox VE API token for this service account and store in dev inside Infisical

#!/bin/bash

REALM="pve"
GROUP="PackerTesting"
USERNAME="svc_devbox_packer@${REALM}"
TOKEN_NAME="packer-testing-token"
PACKER_TEST_POOL="packer-testing"
PACKER_TEST_POOL_PATH="/pool/${TF_TEST_POOL}"
ISO_STORAGE_POOL="local" # change according to your environment
DISK_STORAGE_POOL="local-lvm" # Change accordingly

# Create the resource pool
pveum pool add $PACKER_TEST_POOL --comment "Resource pool for testing Packer builds"

# Create the group
pveum group add $GROUP --comment "Group for any service accounts needing to test Packer builds"

# Create the service account
pveum user add $USERNAME --comment "Service account for devbox testing"

# Add the user to the group
pveum user modify $USERNAME --groups $GROUP

# ---- GROUP PERMISSIONS ----
# Full CRUD on its own pool
pveum aclmod $PACKER_TEST_POOL_PATH --group $GROUP --role PVEVMAdmin
pveum aclmod $PACKER_TEST_POOL_PATH --group $GROUP --role PVEPoolAdmin

# Packer needs to read the node state and find available VM IDs, but cannot modify anything outside its pool.
pveum aclmod /nodes --group $GROUP --role PVEAuditor

# Read-only on global storage configurations
pveum aclmod /storage --group $GROUP --role PVEAuditor

# Full permissions on required storage pools
pveum aclmod /storage/$ISO_STORAGE_POOL --group $GROUP --role PVEDatastoreAdmin
pveum aclmod /storage/$DISK_STORAGE_POOL --group $GROUP --role PVEDatastoreAdmin

# Network Permissions: Ability to attach the network interface
pveum aclmod /sdn/zones/localnetwork --group $GROUP --role PVESDNUser

# Create an API token for the user
# No privilege separation, since inheriting off the group's permissions
pveum user token add $USERNAME $TOKEN_NAME --privsep 0

Store PVE Token in Infisical

Secret 1

• Key: PROXMOX_USERNAME
• Value: svc_devbox_packer@pve!packer-testing-token
• Comment: Proxmox VE API token for testing Packer builds

Secret 2

• Key: PROXMOX_TOKEN
• Value: 2e9xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx8d9
• Comment: Proxmox VE API token for testing Packer builds

Secret 3

• Key: PKR_VAR_ssh_username
• Value: packertest
• Comment: Username Packer uses when logging into test VM for provisioning

Secret 4

• Key: PKR_VAR_ssh_password
• Value: Generate a strong, random password and store it in here
• Comment: Password Packer uses when logging into test VMs for provisioning

Secret 5

• Key: PKR_VAR_windows_admin_password
• Value: Generate a strong, random password and store it in here
• Comment: Use this password for WinRM and RDP login on test VMs

Firewall Rules

Firewalls required for testing:

If you're operating on a flat network where all of your hosts are in the same subnet, this will most likely not apply to you, unless you're using host-based firewalls — e.g. iptables.

1. Allow DevBox to reach PVE API
   1. Source: DevBox
      Source Port: any
   2. Destination: Proxmox Node(s)
      Destination Port: 8006
2. Allow Packer to provision VMs over SSH
   1. Source: Developer Box
      Source Port: any
   2. Destination: Packer-provisioned VM VLAN
      Destination Port: 22 — SSH provisioning
3. Allow DevBox to Pull from Container Registry
   1. Source: Developer Box
      Source Port: any
   2. Destination: GitLab CE Server
      Destination Port: 5050

Install Docker on DevBox (Debian)

Debian

Learn how to install Docker Engine on Debian. These instructions cover the different installation methods, how to uninstall, and next steps.

Docker IncDocker Inc

sudo usermod -a -G docker $(whoami)

Pull the Packer Image from GitLab

Set up Credential Helper

Docker Credential Help... | 0xBEN | Notes

Install Prerequisites sudo apt update && sudo apt install -y gpg pass pinentry-tty curl Setup Pass…

0xBEN | Notes

docker login gitlab-ce.lab.home.internal:5050

Pull the Packer Image

docker pull gitlab-ce.lab.home.internal:5050/infrastructure/runner-images/packer:latest

Testing the Packer Container

Now is an excellent opportunity to test out pulling packer from our Container Registry and making sure the containerized environment can build Packer templates without a hitch. It also has the infisical CLI installed, so we should be ready to go.

cd ~/Code/IaC_Project/packer/proxmox

docker run --rm -it \
-u "$(id --user):$(id --group)" \
-e "HOME=/tmp" \
-v "$PWD":/workspace \
-v /etc/passwd:/etc/passwd:ro \
-v /etc/group:/etc/group:ro \
-w /workspace \
gitlab-ce.lab.home.internal:5050/infrastructure/runner-images/packer:latest \
bash

• Mount /etc/passwd and /etc/group from the host to the container to force the container to acknowledge our UID / GID
  • These files are already read-only on the host anyway
  • We want to launch the container with our UID and GID using the to ensure that any files created from within the container do not cause ownership issues inside the repository.

Infisical Access

Add a Custom Role

Add a Machine Identity

Set Universal Auth Token

Authenticate and Import Secrets

read -e -s -p 'Enter your machine ID (input hidden): ' machine_id
export MACHINE_ID="$machine_id"
read -e -s -p 'Enter your machine secret (input hidden): ' machine_secret
export MACHINE_SECRET="$machine_secret"
read -e -p 'Enter your Infisical project ID (input shown): ' infisical_project_id
export INFISICAL_PROJECT_ID="$infisical_project_id"
export INFISICAL_SECRET_PATH="/packer/pve"

INFISICAL_ACCESS_TOKEN=$(infisical login \
--domain="https://secrets.lab.home.internal" \
--method="universal-auth" \
--client-id="${MACHINE_ID}" \
--client-secret="${MACHINE_SECRET}" \
--silent \
--plain)

eval $(infisical export \
--domain="https://secrets.lab.home.internal" \
--token="${INFISICAL_ACCESS_TOKEN}" \
--projectId="${INFISICAL_PROJECT_ID}" \
--env=dev \
--path="${INFISICAL_SECRET_PATH}" \
--format=dotenv-export \
--silent)

Building the Proxmox VM Template

cd /workspace/builds/linux/debian/debian-13-vm/

packer init .

packer fmt .

packer validate .

packer build \
-var 'resource_pool=packer-testing' \
-var 'vm_name=debian-13-test-build' \
-var 'vm_id=30000' .

Prepare to Commit the Code

.gitignore

cd ~/Code/IaC_Project/packer/

tree -a -I .git/

.gitlab-ci.yml

This is where we define the pipeline configuration for our infrastructure/packer repository. Be sure to read the comments — which go into more detail about each section.

# Source in the Infisical authentication helper
include:
  - project: 'infrastructure/ci-helpers'
    ref: main
    file: '/infisical.gitlab-ci.yml'
  - template: Security/Secret-Detection.gitlab-ci.yml
  - template: Security/SAST.gitlab-ci.yml

# Trigger pipeline in the web and on merge rquests to main (or other default)
workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "web"
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

# ----- SECRETS SCANNING START ----- #

sast:
  stage: validate

secret_detection:
  stage: validate

# Stages to break up pipeline jobs
stages:
  - validate
  - build

# ----- SECRETS SCANNING END ----- #

# Global pipeline variables
# Authentication to container registry to pull Dockerized packer
variables:
  SECRET_DETECTION_ENABLED: 'true'
  DOCKER_AUTH_CONFIG: >
    {
      "auths": {
        "$CI_REGISTRY": {
          "username": "$CI_REGISTRY_USER",
          "password": "$CI_JOB_TOKEN"
        }
      }
    }
  PACKER_IMAGE:           "$CI_REGISTRY/infrastructure/runner-images/packer:${PACKER_VERSION}" # References version variable from group CI/CD settings

# Hidden job (helper)
# Defines the helper script for running:
# 'packer fmt', 'packer init', and 'packer validate'
# PROXMOX_USERNAME and PROXMOX_TOKEN are empty placeholders
# Because we do not pull Infisical secrets in the unprotected runner
# And, these variables are required for 'packer validate' to pass
.validate-build:
  stage: validate
  image: $PACKER_IMAGE
  tags: [lint]
  variables:
    PROXMOX_USERNAME: "placeholder"
    PROXMOX_TOKEN:    "placeholder"
  script:
    - |
      set -e
      echo "[*] Checking format: ${BUILD_DIR}"
      cd "${BUILD_DIR}"
      packer fmt -check .
      
      echo "[*] Validating: ${BUILD_DIR}"
      packer init .
      
      packer validate .
      echo "[+] All validation tasks passed on: ${BUILD_DIR}."


# -- Hidden: base build job ---------------------------------------
# All PKR_VAR_* and provider-native variables are already in the
# environment before this script runs:
#   - Non-secrets: from GitLab project-level CI/CD variables
#   - Secrets: exported by .infisical-auth before_script
# Nothing to export. Just build.
.build-template:
  stage: build
  image: $PACKER_IMAGE
  tags: [packer]
  variables:
    INFISICAL_SECRET_PATH: "/packer/pve"
  extends: .infisical-auth
  script:
    - |
      set -e
      cd "${BUILD_DIR}"
      packer init .
      packer build --force .
  rules:
    # Triggers job any user creates a new pipeline in the web UI
    # Must be manually started
    - if: $CI_PIPELINE_SOURCE == "web"
      when: manual


# ----- Debian 13 VM ----- #

validate-debian-13-vm:
  extends: .validate-build
  variables:
    BUILD_DIR: "proxmox/builds/linux/debian/debian-13-vm"
  rules:
    # Triggers job when merge request created
    # But only if code changed inside the working directory
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      changes:
        - proxmox/builds/linux/debian/debian-13-vm/**/*
        - proxmox/scripts/bash/**/*
    # Triggers job any time code is commited to default branch
    # But only if code changed inside the working directory
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      changes:
        - proxmox/builds/linux/debian/debian-13-vm/**/*
        - proxmox/scripts/bash/**/*

build-debian-13-vm:
  extends: .build-template
  variables:
    BUILD_DIR: "proxmox/builds/linux/debian/debian-13-vm"
  rules:
    # Rules are evaulated from top to bottom
    # Also, 'rules' here will override rules in '.build-template'
    # To include the rules from '.build-template', we use a '!reference'
    # We put it at the top, because "changes" always evaluates to "true" in the web
    - !reference [.build-template, rules]
    # Triggers job any time code is commited to default branch
    # But only if code changed inside the working directory
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      changes:
        - proxmox/builds/linux/debian/debian-13-vm/**/*
        - proxmox/scripts/bash/**/*

Merge Request

Firewall Rules for Production

If you're operating on a flat network where all of your hosts are in the same subnet, this will most likely not apply to you, unless you're using host-based firewalls — e.g. iptables.

1. Allow Protected Runner to reach PVE API
   1. Source: Protected Runner VM
      Source Port: any
   2. Destination: Proxmox Node(s)
      Destination Port: 8006
2. Allow Packer to provision VMs over SSH
   1. Source: Protected Runner VM
      Source Port: any
   2. Destination: Packer-provisioned VM VLAN
      Destination Port: 22 — SSH provisioning

Push Files to GitLab

cd ~/Code/IaC_Project/packer/

git status

git add .

git commit -m "Adds all packer files and pipeline config."

git push -u origin add-initial-files

Create a Merge Request

Inspect Validate Pipeline Job

Trigger the Build Job

Clean up Feature Branch

git switch main

git pull --prune

git branch -d add-initial-files

Pipeline Sanity Check

Code Change

Given how the pipeline is configured, you should be able to test the change detections by making a small cosmetic change.

cd ~/Code/IaC_Project/packer

git checkout -b test-pipeline-logic

vm_tags              = "packer-template;vm;linux;debian;trixie"

vm_tags              = "packer-template;vm;linux;debian;trixie;test"

git add . && git commit -m "Adds test tag to template."

git push -u origin test-pipeline-logic -o merge_request.create

cd ~/Code/IaC_Project/terraform

git switch main && git pull --prune

git branch -d test-pipeline-logic

Closing the Packer Module

Quick Review

Lessons Learned

• We established the model of local iterative refinement
  • We don't want to troubleshoot packer validate or packer build in the pipeline
  • We do want to run those locally in our development environment while we debug issues until a build successfully completes
  • Then, packer build --force . in the pipeline will create the final template
• We identified the need to test packer build in a resource pool isolated from production
  • We created a separate Proxmox VE group, resource pool, user, and token and scoped permissions to write to the test pool
  • We set up secrets in dev inside Infisical and scoped role for our DevBox to authenticate with universal auth and pull the secrets from dev
    • The variable names match prod, so are immediately ready for testing with the Packer files
  • We identified the need to run packer build with a -var 'resource_pool=packer-testing flag during test builds

Moving Forward

• Our pipeline for debian-13-vm should now be stable enough that we can:
  • Start a new feature branch
  • Make changes to the template or variables
  • git add ., git commit, and git push and let the pipeline handle future builds
• For new VM templates...
  • You'll create your source code and repeat the local iterative refinement until you have a successful build
    • Use your packer container from the GitLab Container Registry
    • Log into Infisical and import secrets from prod
    • packer init ., packer fmt ., and packer validate .
    • Then, packer build -var 'resource_pool=packer-testing'. until a template build is successful
  • Then, you'll update .gitlab-ci.yml with new job:
    • validate-template-name
    • build-template-name
  • git add, git commit, and git push your new template and pipeline
• Updating your README.md
  • At some point, you should update your README.md file
  • In this file, you should outline the steps you (or other developers) should take when carrying out local development and testing and commits to production

Helpful Links

Proxmox Builder | Integrations | Packer | HashiCorp Developer

Explore Packer product documentation, tutorials, and examples.

Proxmox Builder | Integrations | Packer | HashiCorp Developer

bento/packer_templates at main · chef/bento

Packer templates for building minimal Vagrant baseboxes for multiple platforms - chef/bento

GitHubchef

Next Step

Infrastructure-as-Code with Proxmox: GitOps - Terraform Pipeline

In this module, we’ll establish the strategies for developing and testing Terraform plans. We’ll finish the module by creating a production plan to clone the Debian 13 Packer template and deploy to Proxmox VE via the GitLab pipeline.

0xBEN0xBEN