Proxmox: Running OpenCTI

In this post, I walk you through steps of running an OpenCTI server to aggregate threat intelligence into a single interface.

10 months ago   •   19 min read

By 0xBEN
Table of contents

OpenCTI Project Links





System Requirements

ℹ️
Note that these are the recommended system specifications:
CPU RAM Disk Space
6 Cores 16GiB >32GB

I'll be running my VM with:

  • 8 Cores
  • 16 GiB RAM
  • 128 GB disk





Creating the VM

Debian ISO

Installing Debian via the Internet

I'll be using the amd64 small ISO from the official Debian page. Create your VM using the hardware specifications above, or your own hardware specifications. Attach the .iso file and boot the VM.



VM Hardware

Adjust your bridge and any VLANs as necessary



Installing the Operating System

Next, choose your language, location, and keyboard mapping.

My local domain is home.lab, adjust accordingly. If you don't know leave it blank.

Next, set a password for the root user.

Set a display name for the non-root user
Set a username for the non-root user

Set a password for your non-root user.

Set your timezone.

Select the target disk
Click 'Continue'
Click 'Continue'
Click 'Continue'

Choose your country for the proper Debian package mirror. Then, choose your mirror. deb.debian.org should work in most cases.

If not using a proxy, leave this blank and click 'Continue'
Choose your preferred telemetry option and click 'Continue'
I won't be installing a desktop environment, so adjust accordingly
Click 'Continue'
Choose the GRUB target and press 'Continue'
Congrats! Install complete. Press 'Continue'.
Log in using 'root' or your non-root user after rebooting

ℹ️
If you log in as your non-root user and the sudo binary is not installed...
  1. Run the command su root and enter the 'root' user's password
  2. Install sudo by running:
# Install operating system upgrades and install sudo
apt clean && apt update && apt upgrade -y && apt install -y sudo
# Add your non-root user to the sudo group
/usr/sbin/usermod -a -G sudo <username>
# Add the /usr/sbin directory to root's path at login
echo 'export PATH=/usr/sbin:$PATH' >> ~/.bashrc
# Exit out of your root session
exit

# Now running as your non-root user again
# Add the /usr/sbin directory to your non-root user's path at login
echo 'export PATH=/usr/sbin:$PATH' >> ~/.bashrc

  1. Logoff from your non-root user's session and log back in
  2. Now, verify your sudo privileges





Installing OpenCTI

The official documentation states that Docker is the recommended method to run the OpenCTI components, so that's what we'll be doing in this write-up.

Increase Virtual Memory for ElasticSearch

sudo sysctl -w vm.max_map_count=1048575
echo 'vm.max_map_count=1048575' | sudo tee --append /etc/sysctl.conf





Install Docker Engine

Install Docker Engine on Debian
Instructions for installing Docker Engine on Debian
sudo apt install -y ca-certificates curl gnupg lsb-release
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin docker-compose
sudo systemctl enable --now docker
sudo usermod -a -G docker $(whoami)

Once finished, log off and log back in as the non-root user.





Install and Configure Portainer

Install Portainer

Portainer is a web frontend that allows us to manage Docker environments through a GUI.

sudo mkdir /opt/portainer
sudo chown $(whoami):$(whoami) /opt/portainer
cd /opt/portainer
curl -sL https://downloads.portainer.io/portainer-agent-stack.yml -o portainer-agent-stack.yml

Edit a few Portainer variables to prevent any clashes with TCP port bindings.

nano ./portainer-agent-stack.yml
      - "9000:9000"
      - "8000:8000"
Before
      - "9900:9000"
      - "8800:8000"
After



Deploy Portainer

sudo docker swarm init
sudo docker stack deploy --compose-file=portainer-agent-stack.yml portainer

Connect to your Portainer instance at http://fqdn-or-ip:9900

Go ahead and set up a new password for the admin user and save it in your password vault. Once your password is set, you should be redirected to the Portainer dashboard.



Add a Docker Stack for OpenCTI

Click on 'primary'



Add the Docker-Compose Configuration

We are going to paste the official docker-compose.yml configuration here
docker/docker-compose.yml at master · OpenCTI-Platform/docker
OpenCTI Docker deployment helpers. Contribute to OpenCTI-Platform/docker development by creating an account on GitHub.
Copy the contents of the docker-compose.yml file in the official GitHub repo

Remove all comments from the Elasticsearch configuration:

before
after

I prefer to serve the application on TCP port 80, as opposed to 8080. This port mapping states that we open 80 externally and map internally to 8080 on the OpenCTI container.

Change the following values:

    ports:
      - "8080:8080"
Before
    ports:
      - "80:8080"
After



Configure Environment Variables

Generate some environment variables by running this command:

cat << EOF
CONNECTOR_EXPORT_FILE_CSV_ID=$(cat /proc/sys/kernel/random/uuid)
CONNECTOR_EXPORT_FILE_STIX_ID=$(cat /proc/sys/kernel/random/uuid)
CONNECTOR_HISTORY_ID=$(cat /proc/sys/kernel/random/uuid)
CONNECTOR_IMPORT_FILE_STIX_ID=$(cat /proc/sys/kernel/random/uuid)
CONNECTOR_IMPORT_REPORT_ID=$(cat /proc/sys/kernel/random/uuid)
ELASTIC_MEMORY_SIZE=8G
MINIO_ROOT_PASSWORD=$(cat /proc/sys/kernel/random/uuid)
MINIO_ROOT_USER=$(cat /proc/sys/kernel/random/uuid)
OPENCTI_ADMIN_EMAIL=admin@opencti.io
OPENCTI_ADMIN_PASSWORD=CHANGEMEPLEASE
OPENCTI_ADMIN_TOKEN=$(cat /proc/sys/kernel/random/uuid)
OPENCTI_BASE_URL=http://localhost
RABBITMQ_DEFAULT_PASS=guest
RABBITMQ_DEFAULT_USER=guest
SMTP_HOSTNAME=$(hostname)
EOF

You should see some environment variables print to the terminal screen:

Copy the output to your clipboard.

Click 'Advanced mode'

Paste your environment variables in the input box:

Some variables are just placeholders and need to be changed. Please update the following variables:
  • OPENCTI_ADMIN_EMAIL
    • Used to log into OpenCTI
    • Also, used for system and other alerts
  • OPENCTI_ADMIN_PASSWORD
    • Password to log into OpenCTI
    • Change this and store in a password vault
  • OPENCTI_BASE_URL
    • Change localhost to your FQDN or IP address
    • Example: http://10.148.148.3 or http://opencti.home.lab
  • RABBITMQ_DEFAULT_USER
    • Change this and store in a password vault
  • RABBITMQ_DEFAULT_PASS
    • Change this and store in a password vault

Configuration is complete. We can now deploy the Docker configuration.

ℹ️
At your initial deployment of the stack -- and with any subsequent updates to the stack -- you may need to wait a several minutes for all of the Docker components to initialize.

If the OpenCTI interface is not immediately available, be patient.





Logging into OpenCTI

In your web browser, navigate to http://ipaddress or http://fqdn. Remember, we configured the application to be served over TCP port 80, whereas most documentation references TCP port 8080.

The username and password were set by you in the docker-compose.yml configuration.

Upon first glance, the empty canvas on the OpenCTI UI can be a little underwhelming. OpenCTI does not ship with any data, so you either have to import it manually or use connectors to programmatically pull data.





Configuring OpenCTI Connectors

A list of officially supported connectors (and their configurations) is available on their GitHub repo.

GitHub - OpenCTI-Platform/connectors: OpenCTI Connectors
OpenCTI Connectors. Contribute to OpenCTI-Platform/connectors development by creating an account on GitHub.

Connectors are divided into several categories:

  • External Import
  • Internal Enrichment
  • Internal Export File
  • Internal Import File
  • Stream



Configure an External Import Connector

AlienVault OTX

For the sake of this walkthrough, let's configure the AlienVault OTX external connector, since this is a very powerful and widely used threat intel exchange.



Connector Roles and Permissions

According to the official documentation, the proper way to add connectors to OpenCTI is:

  • Create a specific role for connectors
  • Create an account per connector and attach the role to it
Click 'Settings' in the bottom-left
Click 'Accesses'



Verify the Connector Role Permissions

Default 'Connector' role

At the time of this writing, the OpenCTI Docker install provides a Connector role with the capabilities configured to work out of the box. I had previously written about an issue where this role was not permissive, but this seems to have been fixed in the latest version — 5.7.3 at the time of this writing.



Create the Connector Account

We're still working in the Accesses tab from above.

Click the 'Users' tab
Click the 'Create a user' button
Fill out the details of the user

Specify a password for the connector account and click Create.



Update the Account Roles

Click on the account name
Click the menu icon and click 'Update'
Change the roles to the 'Connector' role only
While you're here, make a note of the API token

Click anywhere outside of the pane to return to the main view.



Update the Docker-Compose Configuration

We are going to use the docker-compose.yml template for the AlienVault connector directly from the project GitHub. We'll be borrowing a portion of this file and injecting into the main docker-compose.yml in Portainer.

connectors/docker-compose.yml at master · OpenCTI-Platform/connectors
OpenCTI Connectors. Contribute to OpenCTI-Platform/connectors development by creating an account on GitHub.

ℹ️
Indentation is very important for YAML, so please be sure to maintain the indentation when pasting into your Portainer stack.

We need to change some values in the YAML below:

      - OPENCTI_URL=http://localhost
Before
      - OPENCTI_URL=http://opencti:8080
Affter
      - OPENCTI_TOKEN=ChangeMe
Before
      - OPENCTI_TOKEN=connector-api-token-here
After: paste your API token for the AlienVault user here
      - CONNECTOR_ID=ChangeMe
Before

Every connector must have a unique CONNECTOR_ID. There are UUID4 random generators on the web that you can use. Or, you can generate them in PowerShell or bash.

Bash: echo $(cat /proc/sys/kernel/random/uuid)

PowerShell: [Guid]::NewGuid().Guid

Then paste the output into the CONNECTOR_ID variable.

      - CONNECTOR_ID=random-uuid-v4-here
After
      - ALIENVAULT_API_KEY=ChangeMe
Before
      - ALIENVAULT_API_KEY=alienvault-otx-api-key
After: log into your AlienVault account and retrieve this
  connector-alienvault:
    image: opencti/connector-alienvault:5.5.2
    environment:
      - OPENCTI_URL=http://localhost
      - OPENCTI_TOKEN=ChangeMe
      - CONNECTOR_ID=ChangeMe
      - CONNECTOR_TYPE=EXTERNAL_IMPORT
      - CONNECTOR_NAME=AlienVault
      - CONNECTOR_SCOPE=alienvault
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_UPDATE_EXISTING_DATA=false
      - CONNECTOR_LOG_LEVEL=info
      - ALIENVAULT_BASE_URL=https://otx.alienvault.com
      - ALIENVAULT_API_KEY=ChangeMe
      - ALIENVAULT_TLP=White
      - ALIENVAULT_CREATE_OBSERVABLES=true
      - ALIENVAULT_CREATE_INDICATORS=true
      - ALIENVAULT_PULSE_START_TIMESTAMP=2020-05-01T00:00:00                  # BEWARE! Could be a lot of pulses!
      - ALIENVAULT_REPORT_TYPE=threat-report
      - ALIENVAULT_REPORT_STATUS=New
      - ALIENVAULT_GUESS_MALWARE=false                                        # Use tags to guess malware.
      - ALIENVAULT_GUESS_CVE=false                                            # Use tags to guess CVE.
      - ALIENVAULT_EXCLUDED_PULSE_INDICATOR_TYPES=FileHash-MD5,FileHash-SHA1  # Excluded Pulse indicator types.
      - ALIENVAULT_ENABLE_RELATIONSHIPS=true                                  # Enable/Disable relationship creation between SDOs.
      - ALIENVAULT_ENABLE_ATTACK_PATTERNS_INDICATES=true                      # Enable/Disable "indicates" relationships between indicators and attack patterns
      - ALIENVAULT_INTERVAL_SEC=1800
    restart: always
    depends_on:
      - opencti

ℹ️
I like to put the connector configurations at the top, since it's easier to find them there.
⚠️
Be sure to maintain the indentation, as Portainer will throw an error if the indentation is not precise.

You can paste the configuration into the Portainer stack right at the top, just below services:. Don't paste it in the middle of another service's configuration. The good thing is that the editor in Portainer will throw an error if you make a mistake in your YAML syntax.

When you've finished editing the Portainer stack, scroll down and click Update the stack.

⚠️
It can take a few minutes for everything to fully initialize after stack updates. Please be patient before panicking, because your OpenCTI web interface isn't coming up.



Check the Status of the Connector

Log into Portainer and open your opencti stack.

Click on the page icon to view container logs
It's still making its first run and fetching pulses

Log into OpenCTI to view the connector.

Click 'Data'
Click 'AlienVault'
Note the 'In Progress Works'

Also, note the interval in the connector YAML configuration from above:

      - ALIENVAULT_INTERVAL_SEC=1800
Every 1800 seconds (30 minutes), check for new pulses





Exploring the Platform

In this tutorial, we got the platform installed and configured with a single external connector, the AlienVault connector. That is only scratching the surface in terms of available connectors and information you can import to the platform. Some other recommended external connectors to check out:

  • abuse-ssl
  • abuseipdb-ipblacklist
  • cve
  • mitre
  • opencti (yes, that is the name of the connector)
  • urlhaus

That is just to name a few. Remember that we take the docker-compose.yml of the connector and plug it into Portainer in the same way we did for AlienWare. Just be sure to carefully inspect the configuration files for any values that need to be corrected.

Please note that some connectors may require additional information such as API keys.
ℹ️
The walkthrough video below shows an OpenCTI environment that has been configured with information from various sources. So, please don't be surprised if your environment doesn't compare, as we have just gotten started.





Upgrading OpenCTI

Check for Breaking Changes

You can view the latest release on their GitHub repository. Please double check the release notes and the official documentation for any advisories, as there could be significant changes between versions.

Releases · OpenCTI-Platform/opencti
Open Cyber Threat Intelligence Platform. Contribute to OpenCTI-Platform/opencti development by creating an account on GitHub.





Make a Snapshot of Your VM

Log into Proxmox and click on your OpenCTI VM. Click on Snapshots > Take Snapshot. I'm going to create my snapshot accordingly:

You should see TASK OK in the output when the snapshot has been successfully created.





Update the OpenCTI Stack in Portainer

Log into Portainer

I'm serving my Portainer instance at https://10.148.148.14:9443, log into your Portainer instance according to your environment.

Log into Portainer
Click on Primary
Click on Stacks
Click on 'opencti'
Clickon 'Stop this stack'
Click on 'Editor'

In this example, I'll be upgrading from 5.5.2 to 5.7.3. We'll be approaching the upgrade in two parts:

  1. Core Docker images
  2. Connector Docker images

Core Docker Images

You can find the latest docker-compose.yml for the the OpenCTI Dockerized deployment here:

docker/docker-compose.yml at master · OpenCTI-Platform/docker
OpenCTI Docker deployment helpers. Contribute to OpenCTI-Platform/docker development by creating an account on GitHub.

We're looking through the docker-compose.yml file for any line that starts with image: and updating those lines accordingly in our Portainer stack. We can simplify the process by running this command:

curl -s https://raw.githubusercontent.com/OpenCTI-Platform/docker/master/docker-compose.yml | grep 'image:'
Show the images used in the latest Dockerized OpenCTI stack

Using redis as an example, you can see that I am using Docker image version 7.0.6 in my environment and will be updating to 7.0.9.

Before
After
ℹ️
Repeat this process for all remaining docker images shown in the curl output.





Connector Docker Images

You can simplify the process of finding your current OpenCTI connectors by pressing CTRL + F to use the search function in Portainer.

Search for the keyword: opencti/connector and use the next / previous buttons to find connectors in your compose file.

Using the AlienVault connector as an example, the current Docker image version is 5.5.2 and I need to update to 5.7.3. From my experience, the connector versions track the OpenCTI version, but you can always check the connectors' docker-compose.yml file at their GitHub repo here.

Before
After





Push the Stack Changes into Production

Click 'Update the stack'
Enable 'Re-pull image...' and click 'Update'
⚠️
Be patient! It's going to take a while for the upgrade to complete and for OpenCTI to come back online. Don't be surprised if the upgrade takes up to 10 minutes or possibly more.

If you encounter any issues during your upgrade, some things you can try:

  • Inspect the containers in the opencti stack and check logs
  • Restart the server
  • Roll back your snapshot and double-check your steps
  • Consult the official documentation and GitHub releases page for any major change documentation





Addendum

Upgrade Portainer

Upon logging into Portainer, you'll probably notice an advertised upgrade to a newer version. It's important to keep your technology stack up to date. When installing Portainer (and OpenCTI), we used Docker Swarm to deploy the Docker components. The official documentation for upgrading Portainer using swarm can be found here:

Upgrading on Docker Swarm - Portainer Documentation
docker pull portainer/portainer-ce:latest
docker service update --image portainer/portainer-ce:latest --force portainer_portainer

docker pull portainer/agent:latest
docker service update --image portainer/agent:latest --force portainer_agent 
Run these commands to update your Portainer components





Serving Portainer over TLS

Log into Portainer and navigate to the Settings menu.

From there, scroll down to the SSL certificate section.

Here, you'll need to provide the X.509 certificate file and the associated key for that file. If you don't provide a certificate and key file, then Portainer will use a default certificate when serving HTTPS.

If you select the Force HTTPs only option, this is going to disable HTTP authentication – which is served on TCP/9900 from when we first set up Portainer.

If you've applied the changes, navigate to your Portainer instance's IP address or FQDN on TCP/9443 – for example, https://opencti.home.lab:9443 and use your Portainer admin credentials to sign in.





Serving OpenCTI over TLS

Stop the OpenCTI Stack

Click 'Stacks'
Click 'opencti'
Stop this stack



Create a Local Volume to Store TLS Cert and Key

Click on 'Volumes'
Add volume

We are going to create a local volume using Portainer and then use a bind mapping on our opencti container, so that the container (guest) can read the files on the host's file system.

Fill out the fields and click 'Create the volume'
ℹ️
Once you create the volume, this will create the following path on your Docker host (assuming you named your volume, opencti_https): /var/lib/docker/volumes/opencti_https/_data/. Once created, place your.crt and .key TLS files in this directory.
⚠️
If your certificate expires and you need to replace the files, don't forget to stop and start your OpenCTI stack again.



Update the OpenCTI Stack

ℹ️
This is just an example TLS configuration. Use the correct domain name for your environment.

Let's say you got a TLS certificate for your OpenCTI instance with the hostname, opencti.domain.tld. This is the hostname we'll be referencing throughout this configuration example.

Update the Base URL Environment Variable

Click 'Stacks'
Under 'Environment variables'...
Update your 'OPENCTI_BASE_URL' with your HTTPS URL



Add the Volume to the Docker Compose Configuration

Click 'Editor'

Make the opencti_https volume that we created before available to the OpenCTI stack.

Before
After



Update the OpenCTI Container Configuration

Scroll down until you find the opencti container declaration.

Before
After

Here are the changes we're making in the After picture:

  • Port changes:
    • Change the port binding from 80 to 443 for the default HTTPS TCP port binding. 443 is the external facing port.
  • Add the volume we created:
    • In a previous step we created a volume called opencti_https
    • We copied our .crt and .key files to the /var/lib/docker/volumes/opencti_https/_data/ directory
    • Now, we're saying the opencti container should map that directory from the host to the /certs mountpoint internally to the container
  • Define the paths to the TLS files:
    • APP__HTTPS_CERT__KEY
    • APP__HTTPS_CERT__KEY
    • Both of these point to the internal /certs mapping, so the container can read the files internally (even though they're stored on the host)
    • Make sure to change the my-tls-cert.key and my-tls-cert.crt names to whatever the names of your files are in /var/lib/docker/volumes/opencti_https/_data/



Update the OpenCTI URL Variable on All Workers

Every connector and worker in the Docker Compose configuration has an environment variable pointing to the OpenCTI API.

    environment:
      - OPENCTI_URL=http://opencti:8080
Before: This is the default configuration
You need to update EVERY INSTANCE of the OPENCTI_URL variable. This should match the hostname that was used for your TLS certificate. Up until now, we've been using the hostname opencti.domain.tld, so update yours accordingly.

    environment:
      - OPENCTI_URL=https://opencti.domain.tld
After: This is the new configuration



Update the Stack

Once you've made all of the designated changes to the stack, it's time to update the stack with your changes. Pressing this button will redploy the entire stack, so be patient while all of the changes are loaded and your containers are deployed.



Verify HTTPS Functionality

In your browser, try navigating to your new URL, https://opencti.domain.tld. You should find that your site loads normally and that your browser trusts the connection as evidenced by the padlock icon in the URL bar.

Also, make sure your containers are not reporting any issues with connecting to the API.

Click 'Containers'
Click the 'Logs' icon to view container logs

Inspect your worker and connector logs and ensure there are no error logs regarding API connectivity issues.





Conclusion

If you've followed along up to this point, I want to thank you for reading. I hope that this blog has helped you successfully get your OpenCTI instance running and configured. OpenCTI is a powerful cyber threat intelligence engine that can aggregate and enhance data from multiple sources. Check out their documentation for more ways to enhance your OpenCTI instance.

Spread the word

Keep reading