ℹ️

This page is part of a larger series on learning Infrastructure-as-Code (IaC) using Proxmox Virtual Environment. Click here to be taken back to the project home page.

Previous Step

Before We Start

⚠️

This is NOT going to be a high-availability setup.

In order to make this a high-availability setup, you'd ideally:

Have a distinct PostgreSQL server
Have a distinct Redis server
Have a distinct Infisical server

Having all three components as separate hosts allows for replication across cluster nodes in the event that a service or node goes down.

ℹ️

However, since this is a lab environment, we'll just be creating an all-in-one instance.

Prerequisites

Linux Container

💡

We created a Debian 13 template Linux Container in a previous step. We can clone this and modify the resources and options as needed.

Add some additional tags to cloned LXC for automation purposes

ℹ️

I'll be setting swap to 0 in order to adhere to the production hardening recommendations here: https://infisical.com/docs/self-hosting/guides/production-hardening#linux-binary-deployment

PostgreSQL

Add APT Repository

mkdir -p /usr/share/postgresql-common/pgdg

Create a directory to store the repository public key signature

curl -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc --fail https://www.postgresql.org/media/keys/ACCC4CF8.asc

Save the public key

echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" | tee /etc/apt/sources.list.d/pgdg.list

Add the APT source using the public key file to verify

Install PostgreSQL

apt update && apt install -y postgresql-18 postgresql-contrib

Configure Database

DB_PASS=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 32) && echo "$DB_PASS"

Generate a random password for the DB user and save it somewhere secure

su postgres

psql

CREATE USER infisical_user WITH PASSWORD 'ENTER_YOUR_PASSWORD_HERE';

Add the user account that will manage the application database

CREATE DATABASE infisical OWNER infisical_user;

Create the application database to be managed by the user

quit

exit

Return to "root" shell

Redis Server

Install Redis

apt install -y redis-server

systemctl stop redis-server

REDIS_PASS=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 32) && echo "$REDIS_PASS"

Generate a random password for Redis authentication and save somewhere secure

grep -n '^# requirepass' /etc/redis/redis.conf

Note the line number where the password is defined

nano -l /etc/redis/redis.conf

Start nano with line numbering and replace the password on the target line

systemctl enable --now redis-server

Start Redis server

Infisical Server

Add APT Repository

curl -1sLf 'https://artifacts-infisical-core.infisical.com/setup.deb.sh' | bash

Adds the apt repository and any dependencies

⚠️

ZFS-backed storage users...

First, let me say that the issue below can be totally avoided by using a VM instead of LXC.

When installing the infisical-core omnibus package, it makes hundreds of fsync() syscalls while dpkg untars the installation files to the file system. This will cause a massive bottleneck with ZFS. To work around this, I'll install with eatmydata.

apt install -y eatmydata

alias apt='/bin/eatmydata /bin/apt'

echo "alias apt='/bin/eatmydata /bin/apt'" >> /etc/bash.bashrc

apt install -y infisical-core

Consult man eatmydata and you'll note that this overrides fsync() and arbitrarily returns a 0 exit code when this function is called.

Security Hardening

echo "* hard core 0" | tee -a /etc/security/limits.conf

ulimit -c 0

mkdir /etc/infisical

touch /etc/infisical/infisical.rb

chmod 640 /etc/infisical/infisical.rb

Create Configuration Files

⚠️

Update the postgres:// and redis connection details with the correct username and password.

cat << EOF > /etc/infisical/infisical.rb
infisical_core['DB_CONNECTION_URI'] = "postgres://infisical_user:${DB_PASS}@localhost:5432/infisical"
infisical_core['REDIS_URL'] = "redis://:${REDIS_PASS}@localhost:6379"
infisical_core['ENCRYPTION_KEY'] = '$(openssl rand -hex 16)'
infisical_core['AUTH_SECRET'] = '$(openssl rand -base64 32)'
infisical_core['LISTEN_ADDR'] = '127.0.0.1'
infisical_core['PORT'] = 8080
EOF

infisical-ctl reconfigure

ℹ️

You can use infisical-ctl tail to debug any issues during configuration.

Reverse Proxy for TLS

Dynamic DNS Workflow

As has been demonstrated in the previous step of the lab during testing, this host is going to be assisted by DHCP and Dynamic DNS.

Hostname — secrets
VLAN: 301 (10.30.30.0/24), DHCP
DNS domain — lab.home.internal
DNS server — 10.30.30.1

I'll be creating a DHCP reservation in pfSense for this host, so that it's reliably at the same IP address.

DHCP reservation in pfSense

ℹ️

I've configured pfSense DHCP Dynamic DNS to use the hostname in the static mapping. Therefore, even if my LXC name were different, it will always use the FQDN secrets.lab.home.internal.

⚠️

Also, ensure you allow tcp/443 to the Intermediate CA and ensure the Intermediate CA can reach tcp/80 of any ACME clients.

ACME clients request an ACME certificate from the CA server at tcp/443 and the CA server verifies a nonce at tcp/80 on the client.

Request ACME Certificate

apt install -y nginx certbot python3-certbot-nginx

curl -k https://sub-ca.pki.home.internal/roots.pem -o /usr/local/share/ca-certificates/home-lab-root.crt

Save the Root CA locally

update-ca-certificates

Add to certificate store

certbot certonly \
  --nginx \
  --server https://sub-ca.pki.home.internal/acme/acme@lab.home.internal/directory \
  --domain secrets.lab.home.internal \
  --email admin@lab.home.internal \
  --agree-tos \
  --non-interactive

Using the FQDN, "test123.lab.home.internal" as the domain

Install and Configure Nginx

nano /etc/nginx/sites-available/infisical.conf

⚠️

Update the server_name directive according to your environment.

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 80;
    listen [::]:80;
    server_name secrets.lab.home.internal;
    
    return 301 https://$host$request_uri;
}

server {
    http2 on;
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name secrets.lab.home.internal;

    # Use certbot certificates
    ssl_certificate /etc/letsencrypt/live/secrets.lab.home.internal/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/secrets.lab.home.internal/privkey.pem;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384';
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 1d;

    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; connect-src 'self' wss:;" always;

    access_log /var/log/nginx/infisical_access.log;
    error_log /var/log/nginx/infisical_error.log;

    location / {
        proxy_pass http://127.0.0.1:8080;
        
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;

        proxy_buffers 16 16k;
        proxy_buffer_size 32k;
    }
}

🚨

As a best practice, always edit files in /etc/nginx/sites-available/ as the symbolic link source.

ln -s /etc/nginx/sites-available/infisical.conf /etc/nginx/sites-enabled/infisical.conf

nginx -t && systemctl restart nginx

Initial Setup

✅

I've imported the Root CA and Intermediate CA certificates to my Windows host, so the certificate is trusted.

Fill out the form, set a strong password, and save it somewhere secure (e.g. password manager)

⚠️

The first user to register at this screen becomes the server owner, so complete this step before allowing any other logins.

Click "Server Console" > Disable signups since I'll be the only one using it.

Organization Setup

Recommended reading to understand how data is compartmentalized

Create a Project

Select "Secrets Management" and click "Create Project"

Create Directory Structure

Select "Production" environment > "Add Folder"

Add a folder for Packer > Click "Create"

Desired end-state (All Environments view)

Testing Secrets Access

Create a Secret in Staging

Choose "Staging" and click "Add a New Secret"

Create a "test" secret inside the "test folder" and save in "Staging"

"Staging" environment, "/test" path, name is "test"

Create a Test Machine Account

⚠️

We want to use a least privilege model when creating these machine accounts. I'll set the role to No Access, so that the machine account cannot read any details about the project.

Set role to "No Access" and click "Create"

✅

You'll be routed to the machine account overview when you create the account.

Add Privileges

Set a Privilege Name: test-view-secrets (only lowercase letters, numbers, hyphens)
Click + Add Policies
- Choose Secrets
  - Click Add Policies
Click + Add Condition
- Environment Slug equals staging
- Secret Path in /test
Click Save

Add Universal Auth Secret

Since this is a test account, we won't bother with expirations

🚨

Copy the key, as it will only be shown once

Testing Authentication and Secret Retrieval

ℹ️

This test is going to demonstrate using the REST API to retrieve our test secret. There are multiple ways to retrieve secrets beyond the API.

Created a quick test LXC in Proxmox to act as API client

Generate a Bearer Token

read -s -e -p "Enter client ID (input hidden): " CLIENT_ID

Enter the UUID4 as shown in the "Universal Auth" panel

read -s -e -p "Enter client secret (input hidden): " CLIENT_SECRET

Enter the secret that was generated just above

curl -s -k --location --request POST 'https://secrets.lab.home.internal/api/v1/auth/universal-auth/login' \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode "clientId=${CLIENT_ID}" \
  --data-urlencode "clientSecret=${CLIENT_SECRET}"

read -s -e -p "Enter Bearer token (input hidden): " TOKEN

Enter the access token from the JSON

Use Bearer Token to Fetch Secret

read -e -p "Enter project ID: " PROJECT_ID

curl -s -k -G --request GET \
  --url 'https://secrets.lab.home.internal/api/v3/secrets/raw/test' \
  --data-urlencode "workspaceId=${PROJECT_ID}" \
  --data-urlencode 'environment=staging' \
  --data-urlencode 'secretPath=/test/' \
  --header "Authorization: Bearer ${TOKEN}"

✅

Testing complete. You may now delete the machine identity and secret from the staging environment.

Adding Service Accounts for Tools

Generate Pools, Groups, Users, and Tokens in Proxmox VE

Now that we've created a project under the Infisical organization, we'll want to create service accounts, so that Packer, Terraform, and Ansible can authenticate to the Proxmox VE API. Then, we'll store those API tokens in Infisical.

nano generate-service-accounts.sh

#!/bin/bash

REALM="pve"
ISO_STORAGE_POOL="local" # change according to your environment
DISK_STORAGE_POOL="local-lvm" # change according to your environment

# 1. CREATE RESOURCE POOLS
echo "Creating resource pools..."
pveum pool add packer-templates --comment "Packer-built VM templates"
pveum pool add terraform-managed --comment "Terraform-managed VMs"

# 2. CREATE GROUPS
echo "Creating automation groups..."
pveum group add Packer --comment "Packer service accounts"
pveum group add Terraform --comment "Terraform service accounts"
pveum group add Ansible --comment "Ansible service accounts"

# 3. CREATE USERS
echo "Creating service users..."
pveum user add svc_packer@$REALM --comment "Packer service account"
pveum user add svc_terraform@$REALM --comment "Terraform service account"
pveum user add svc_ansible@$REALM --comment "Ansible service account"

# 4. ASSIGN USERS TO GROUPS
echo "Assigning users to groups..."
pveum user modify svc_packer@$REALM --groups Packer
pveum user modify svc_terraform@$REALM --groups Terraform
pveum user modify svc_ansible@$REALM --groups Ansible

# 5. ASSIGN GRANULAR PERMISSIONS VIA ACLs
echo "Assigning pool and system permissions..."

# ---- PACKER PERMISSIONS ----

# Full CRUD on its own pool
pveum aclmod /pool/packer-templates --group Packer --role PVEVMAdmin
pveum aclmod /pool/packer-templates --group Packer --role PVEPoolAdmin

# Packer needs to read the node state and find available VM IDs, but cannot modify anything outside its pool
pveum aclmod /nodes --group Packer --role PVEAuditor

# Read-only on global storage configurations
pveum aclmod /storage --group Packer --role PVEAuditor
# Full permissions on required storage pools
pveum aclmod /storage/$ISO_STORAGE_POOL --group Packer --role PVEDatastoreAdmin
pveum aclmod /storage/$DISK_STORAGE_POOL --group Packer --role PVEDatastoreAdmin

# Ability to attach the network interface
pveum aclmod /sdn/zones/localnetwork --group Packer --role PVESDNUser

# ---- TERRAFORM PERMISSIONS ----

# Full CRUD on its own pool
pveum aclmod /pool/terraform-managed --group Terraform --role PVEVMAdmin
pveum aclmod /pool/terraform-managed --group Terraform --role PVEPoolAdmin

# Read-only access to Packer pool (Required to clone templates)
pveum aclmod /pool/packer-templates --group Terraform --role PVEPoolUser
pveum aclmod /pool/packer-templates --group Terraform --role PVETemplateUser

# Terraform needs to read the node state and find available VM IDs, but cannot modify anything outside its pool
pveum aclmod /nodes --group Terraform --role PVEAuditor

# Read-only on global storage configurations
pveum aclmod /storage --group Terraform --role PVEAuditor
# Full permissions on required storage pools
pveum aclmod /storage/$ISO_STORAGE_POOL --group Terraform --role PVEDatastoreAdmin
pveum aclmod /storage/$DISK_STORAGE_POOL --group Terraform --role PVEDatastoreAdmin

# Ability to attach the network interface
pveum aclmod /sdn/zones/localnetwork --group Terraform --role PVESDNUser

# ---- ANSIBLE PERMISSIONS ----

# Read-only on Terraform pool (For inventory and state checking)
pveum aclmod /pool/terraform-managed --group Ansible --role PVEAuditor
# Read-only on nodes and VMs for managing VMs outside the pool as well
pveum aclmod /nodes --group Ansible --role PVEAuditor
# Give Ansible read-only to query other VMs to manage as well
pveum aclmod /vms --group Ansible --role PVEAuditor

# 6. GENERATE API TOKENS
echo "Generating API tokens..."
pveum user token add svc_packer@$REALM packer-token --privsep 0
pveum user token add svc_terraform@$REALM terraform-token --privsep 0
pveum user token add svc_ansible@$REALM ansible-token --privsep 0

ℹ️

Run on any PVE node in the cluster.

A Few Clarifications:

Pools
- /pool/packer-templates —
  - svc_packer has full permissions on this pool
    - Later, we'll create the Packer templates, so that they are written here
    - This makes it so that svc_packer may only write VMs to this pool
  - svc_terraform has read-only permissions to this pool
    - Terraform needs to be able to see VMs inside this pool and be able to clone off of them
- /pool/terraform-managed
  - svc_terraform has full permissions on this pool
    - Later, Terraform managed VMs will be written to this pool
    - This makes it so that Terraform may only manage VMs in this pool
  - svc_ansible has read-only permissions to this pool
    - Ansible needs to be able to list VMs in this pool and query their IP addresses using dynamic inventory
--privsep 0 is used because we don't want an API token with unique privileges
- We want the API token to inherit the permissions of the user
- The permissions of the user are controlled at the group level

bash generate-service-accounts.sh

We need to save the yellow highlighted fields in Infisical

⚠️

In Proxmox VE, there's no API key rotation mechanism. You must delete the API key and create a new one. In the event you do, ensure you update it in Infisical.

Save in Infisical Project

Recall we added the folders in Production before

Generate a SSH Key Pair

You can run the below command on any Linux box. Save the resulting automation_key and automation_key.pub files for reference later.

ssh-keygen -t ed25519 -a 100 -C "" -N "" -f automation_key

Packer

Fille out accordingly and click "Create"

💡

The Keys are deliberately set to UPPERCASE to reflect the environment variables that the tools (packer, terraform, and Ansible) will be expecting. We'll fetch secrets from something like /packer/pve and this will inject all of the secrets shown below as environment variables.

Secrets for Packer:

See documentation on any environment variables that will be read by the proxmox provider

Secret 1
- Key: PROXMOX_USERNAME — injected as environment variable, read by plugin
- Value: svc_packer@pve!packer-token
  - When packer runs with the Proxmox provider, it will automatically discover the variable
Secret 2
- Key: PROXMOX_TOKEN — injected as environment variable, read by plugin
- Value: 3xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx9
  - When packer runs with the Proxmox provider, it will automatically discover the variable
Secret 5
- Key: PKR_VAR_ssh_username
- Value: packer
- Comment: Username Packer uses when logging into VM for provisioning
Secret 4
- Key: PKR_VAR_ssh_password
- Value: Password Packer uses when logging into VM for provisioning
Secret 5
- Key: PKR_VAR_windows_admin_password — for use with WinRM
- Value: ENTER_YOUR_SECURE_PASSWORD — generate a secure password
- Comment: Use this password for WinRM and RDP login to provision Windows VMs

✅

Now, we repeat the process for Terraform and Ansible.

Terraform

Terraform Registry

Production folder in Infisical
- terraform parent directory
  - pve subdirectory

Secret 1
- Key: PROXMOX_VE_API_TOKEN
- Value: svc_terraform@pve!terraform-token=3xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxe

Secret 2
- cat ./automation_key.pub — Run command and copy output

Key: TF_VAR_ssh_public_key
Value: ssh-ed25519 AAAAC3...[SNIP]...KVDWli — automation_key.pub output from before
Comment: Terraform adds this SSH key for the user defined in TF_VAR_ssh_username

Secret 3
- cat ./automation_key — Run command and copy output
- Key: TERRAFORM_SSH_PRIVATE_KEY
- Value: --—BEGIN OPENSSH PRIVATE KEY----- — paste all of the output into this field
- Comment: SSH private key to verify successful provisioning of VMs
- Enable Multiline Encoding: ✅

Secret 4
- Key: TF_VAR_ssh_username
- Value: ansible
- Comment: Terraform adds this SSH user, and will login with private key matching TF_VAR_ssh_public_key

Ansible

💡

It may not seem clear now, but in later modules, we can keep things DRY by pulling the TERRAFORM_SSH_PRIVATE_KEY and TF_VAR_ssh_username secrets from /terrafrom/pve so that we only need to update these secrets in one place.

Production folder in Infisical
- ansible parent directory
  - pve subdirectory
    - Secret 1
      - Key: PROXMOX_URL
      - Value: https://proxmox.lab.home.internal:8006
    - Secret 2
      - Key: PROXMOX_USER
      - Value: svc_ansible@pve
    - Secret 3
      - Key: PROXMOX_TOKEN_ID
      - Value: ansible-token
    - Secret 4
      - Key: PROXMOX_TOKEN_SECRET
      - Value: 9xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxf

Secret 5
- Key: WINRM_USER
- Value: Administrator
Secret 6
- Key: WINRM_PASSWORD
- Value: Copy the same password used above in the Packer secrets
Secret 7
- Key: ANSIBLE_HOST_KEY_CHECKING
- Value: False
- Comment: Disable Ansible SSH host key checking, as we don't have PKI-signed host keys

Next Step

Infrastructure-as-Code with Proxmox: GitOps - Scaffolding

In this module, we will self-host a GitLab CE server and set up an OIDC trust between GitLab and Infisical. We will then scaffold our group and projects within the group. Finally, we’ll set up two GitLab docker runners for various pipeline jobs.

0xBEN0xBEN

Infrastructure-as-Code with Proxmox: Infisical Secrets Management

Previous Step

Before We Start

Prerequisites

Linux Container

PostgreSQL

Add APT Repository

Install PostgreSQL

Configure Database

Redis Server

Install Redis

Infisical Server

Add APT Repository

Security Hardening

Create Configuration Files

Reverse Proxy for TLS

Dynamic DNS Workflow

Request ACME Certificate

Install and Configure Nginx

Initial Setup

First Login

Organization Setup

Create a Project

Create Directory Structure

Testing Secrets Access

Create a Secret in Staging

Create a Test Machine Account

Add Privileges

Add Universal Auth Secret

Testing Authentication and Secret Retrieval

Generate a Bearer Token

Use Bearer Token to Fetch Secret

Adding Service Accounts for Tools

Generate Pools, Groups, Users, and Tokens in Proxmox VE

Save in Infisical Project

Generate a SSH Key Pair

Packer

Terraform

Ansible

Next Step

0xBEN

Infrastructure-as-Code with Proxmox

Infrastructure-as-Code with Proxmox: Manual Templating

Infrastructure-as-Code with Proxmox: Dynamic DNS (DDNS)