Background

I am writing this primarily as a follow up for anyone who may have followed my initial installation guide here:

Adding a Comprehensive Wazuh SIEM and Network Intrusion Detection System (NIDS) to the Lab

In this module, we will take a look at the process setting up a comprehensive Wazuh SIEM, including a NIDS and some HIDS agents.

0xBEN0xBEN

I had already installed Wazuh, customized my environment, and had data that I didn't want to lose if I had chosen to install a new instance of Wazuh instead of upgrading in place.

My Environment

It's important to state first that my environment may likely be different from yours. Therefore, your upgrade may look different from mine. Ultimately, you're responsible for your data and your upgrades. I am not responsible for any data loss as a result of following this guide.

Here's what I have in my environment:

• 1 Wazuh Manager server (Linux Container)
• 3 OpenDistro for Elasticsearch servers (Linux Containers)
• 1 OpenDistro Kibana server (Linux Container)

Breaking Down the Upgrade Process

The procedure will go in this order:

1. Stop sharding indices on Elasticsearch
   • Single API call to the _cluster endpoint
2. Stop indexing and flush to complete writes
   • Single API call to the _flush endpoint
3. Stop Filebeat on Wazuh Manager
4. Stop Elasticsearch
   • If multiple servers, stop the service on each node
5. Install the Wazuh Indexer on every Elasticsearch node
6. Copy existing X.509 certificates from Elasticsearch configuration to Wazuh Indexer configuration
   • This is how nodes verify authenticity
7. Modify /etc/wazuh-indexer/opensearch.yml so that you port settings from Elasticsearch
8. Port the jvm.options to the Wazuh Indexer
9. Stop Kibana
10. Upgrade Wazuh Manager server to the latest version (4.3.x)
    • Also, install the latest version of the Wazuh Filebeat module
11. Install Wazuh Dashboard on the Kibana server
12. Copy existing X.509 certificates from Kibana configuration to Wazuh Dashboard configuration
13. Port settings from Kibana to Wazuh Dashboard
14. Update the password for the service account
15. Upgrade completed

Let's Get to Work

Migrating to Wazuh Indexer

Referencing the official documentation here:

Migrating to the Wazuh indexer - Migration guide · Wazuh documentation

User manual, installation and configuration guides. Learn how to get the most out of the Wazuh platform.

WazuhWazuh

I have three Elasticsearch nodes:

• wazuh-elastic-1 is at 10.148.148.6
• wazuh-elastic-2 is at 10.148.148.15 (added later when I expanded to multi-node cluster)
• wazuh-elastic-3 is at 10.148.148.16 (added later when I expanded to multi-node cluster)

Issue an API Call to Stop Sharding Indices

We want to ensure no more data is being written to Elasticsearch, as we want to ensure there is no data loss. Ensure you replace <username> and <password> with your Elasticsearch administrator credentials.

You can issue these API calls from any device where curl is installed as long as it can route to the host.

curl -X PUT "https://10.148.148.6:9200/_cluster/settings" -u <username>:<password> -k -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}
'

We also want to make sure we add any final data to any indices.

curl -X POST "https://10.148.148.6:9200/_flush/synced" -u <username>:<password> -k

Stop Filebeat

My Wazuh Manager is at 10.148.148.8 and is running as a single node. SSH into your Wazuh Manager and issue this command:

systemctl stop filebeat

Stop Elasticsearch

SSH into all of your Elasticsearch nodes and issue this command on each:

systemctl disable --now elasticsearch

Install the Wazuh Indexer

Install the Wazuh Indexer on all of your Elasticsearch nodes.

apt update
apt -y install wazuh-indexer

Copy all of the X.509 certificates from the Elasticsearch configuration directory to the Wazuh Indexer configuration directory.

NOTE: the admin.pem and admin-key.pem certificates are not present on all of your Elasticsearch nodes, so if you get any errors about those files missing, safely ignore. Those certificates will be present on the Elasticsearch node that started your cluster from Day 1.

mkdir /etc/wazuh-indexer/certs
cp /etc/elasticsearch/certs/elasticsearch-key.pem /etc/wazuh-indexer/certs/indexer-key.pem
cp /etc/elasticsearch/certs/elasticsearch.pem /etc/wazuh-indexer/certs/indexer.pem
cp /etc/elasticsearch/certs/admin.pem /etc/wazuh-indexer/certs/admin.pem
cp /etc/elasticsearch/certs/admin-key.pem /etc/wazuh-indexer/certs/admin-key.pem
cp /etc/elasticsearch/certs/root-ca.pem /etc/wazuh-indexer/certs/root-ca.pem
chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs/
chmod 500 /etc/wazuh-indexer/certs/
chmod 400 /etc/wazuh-indexer/certs/*

Copy your Elasticsearch data to Wazuh Indexer. Be careful these are your Elasticsearch indices and logs.

rm -rf /var/lib/wazuh-indexer/ /var/log/wazuh-indexer/
mv /var/lib/elasticsearch/ /var/lib/wazuh-indexer/
mv /var/log/elasticsearch/ /var/log/wazuh-indexer/
chown wazuh-indexer:wazuh-indexer -R /var/log/wazuh-indexer/
chown wazuh-indexer:wazuh-indexer -R /var/lib/wazuh-indexer/

Port Your Elasticsearch Settings to Wazuh Indexer

I am only going to point out the lines that I had to overwrite when I ported from /etc/elasticsearch/elasticsearch.yml to /etc/wazuh-indexer/opensearch.yml.

Wazuh-Elastic-1 (10.148.148.6)

Edit /etc/wazuh-indexer/opensearch.yml

This...

network.host: "0.0.0.0"
node.name: "node-1"
cluster.initial_master_nodes:
- "node-1"
#- "node-2"
#- "node-3"
cluster.name: "wazuh-cluster"
#discovery.seed_hosts:
#  - "node-1-ip"
#  - "node-2-ip"
#  - "node-3-ip"

plugins.security.authcz.admin_dn:
- "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"

plugins.security.nodes_dn:
- "CN=node-1,OU=Wazuh,O=Wazuh,L=California,C=US"
#- "CN=node-2,OU=Wazuh,O=Wazuh,L=California,C=US"
#- "CN=node-3,OU=Wazuh,O=Wazuh,L=California,C=US"

Becomes this...

network.host: 10.148.148.6
node.name: wazuh-elastic-1
cluster.name: home-lab
cluster.initial_master_nodes:
  - wazuh-elastic-1
  - wazuh-elastic-2
  - wazuh-elastic-3
discovery.seed_hosts:
  - 10.148.148.6
  - 10.148.148.15
  - 10.148.148.16

network.host is the node's IP address

network.name is the node's hostname

cluster.name this is the name for my three-node cluster

cluster.initial_master_nodes my cluster nodes

discovery.seed_hosts my cluster nodes

plugins.security.authcz.admin_dn:
- "CN=admin,OU=Docu,O=Wazuh,L=California,C=US"

plugins.security.nodes_dn:
- CN=wazuh-elastic-1,OU=Docu,O=Wazuh,L=California,C=US
- CN=wazuh-elastic-2,OU=Docu,O=Wazuh,L=California,C=US
- CN=wazuh-elastic-3,OU=Docu,O=Wazuh,L=California,C=US

Wazuh-Elastic-2 (10.148.148.15) & Wazuh-Elastic-3 (10.148.148.16)

network.host: 10.148.148.15
node.name: wazuh-elastic-2
...
...
...

network.host: 10.148.148.16
node.name: wazuh-elastic-3
...
...
...

Port the jvm.options to Wazuh Indexer

Make a note of the -Xms and -Xmx settings in /etc/elasticsearch/jvm.options and port them to /etc/wazuh-indexer/jvm.options (along with any other customizations you may have made).

Enable and Start the Wazuh Indexer Service

systemctl daemon-reload
systemctl enable --now wazuh-indexer

Restart Filebeat

SSH back into Wazuh Manager and restart Filebeat.

systemctl restart filebeat

Now, test Filebeat connectivity with your Wazuh Indexers.

filebeat test output

If all goes accordingly, you should see output akin to:

elasticsearch: https://127.0.0.1:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.10.2

Double-Check Your Cluster Health

NOTE: Again, referencing my IP addresses here.

curl -X GET "https://10.148.148.6:9200/_cluster/health?pretty" -u <username>:<password> -k

As long as things aren't red, you should be good to resume shard allocation:

curl -X PUT "https://10.148.148.6:9200/_cluster/settings" -u <username>:<password> -k -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}
'

Cleanup Old Elasticsearch Data

SSH back into all of your Elasticsearch nodes (now Wazuh Indexer nodes) and run these commands.

apt-get remove --auto-remove opendistroforelasticsearch

rm -rf /etc/elasticsearch /var/lib/elasticsearch

Migrating to Wazuh Dashboard

Upgrade Wazuh Manager to Latest Version

If you're reading this as someone who followed my previous installation guide, you're likely still on Wazuh Manager 4.2.x. We want to upgrade to 4.3.x.

SSH into the Wazuh Manager Server

apt update
apt install -y wazuh-manager

Update the Wazuh Filebeat module.

curl -s https://packages.wazuh.com/4.x/filebeat/wazuh-filebeat-0.2.tar.gz | sudo tar -xvz -C /usr/share/filebeat/module

Download the alerts template.

curl -so /etc/filebeat/wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/v4.3.3/extensions/elasticsearch/7.x/wazuh-template.json
chmod go+r /etc/filebeat/wazuh-template.json

Restart Filebeat.

systemctl daemon-reload
systemctl enable filebeat
systemctl start filebeat

Update Filebeat with the new Wazuh template.

filebeat setup --index-management -E output.logstash.enabled=false

Install Wazuh Dashboard

Stop Kibana

SSH into your Kibana server and stop the service.

systemctl stop kibana

Install the Wazuh Dashboard Service

Install the Wazuh Dashboard service. This will replace the existing Kibana service.

apt update
apt install -y wazuh-dashboard

Copy any existing X.509 certificates to the Wazuh Dashboard configuration folder.

mkdir /etc/wazuh-dashboard/certs
cp /etc/kibana/certs/kibana.pem /etc/wazuh-dashboard/certs/dashboard.pem
cp /etc/kibana/certs/kibana-key.pem /etc/wazuh-dashboard/certs/dashboard-key.pem
cp /etc/kibana/certs/root-ca.pem /etc/wazuh-dashboard/certs/root-ca.pem
chmod 500 /etc/wazuh-dashboard/certs
chmod 400 /etc/wazuh-dashboard/certs/*
chown -R wazuh-dashboard:wazuh-dashboard /etc/wazuh-dashboard/certs

Configure Wazuh Dashboard

I am only going to point out the lines that I had to overwrite when I ported from /etc/kibana/kibana.yml to /etc/wazuh-dashboard/opensearch_dashboards.yml.

Edit /etc/wazuh-dashboard/opensearch_dashboards.yml.

This...

server.host: 0.0.0.0
server.port: 443
opensearch.hosts: https://localhost:9200

Becomes this...

server.host: "10.148.148.7"
server.port: 443
opensearch.hosts: ["https://10.148.148.6:9200", "https://10.148.148.15:9200", "https://10.148.148.16:9200"]

server.host is the server's IP address

opensearch.hosts any Wazuh Indexer nodes (formerly Elasticsearch nodes)

I ultimately deleted these lines, cause they are unnecessary:

#opensearch.username:
#opensearch.password:

Keep in mind that if you installed any custom TLS certificates on your Kibana server, you'll want to point to them here. Mine are the default and didn't change in /etc/kibana/kibana.yml.

server.ssl.key: "/etc/wazuh-dashboard/certs/dashboard-key.pem"
server.ssl.certificate: "/etc/wazuh-dashboard/certs/dashboard.pem"

Update the Password for the Service Account

Inspect /etc/kibana/kibana.yml and find the password for the kibanaserver user. Update the username and/or password.

/usr/share/wazuh-dashboard/bin/opensearch-dashboards-keystore --allow-root add opensearch.password

/usr/share/wazuh-dashboard/bin/opensearch-dashboards-keystore --allow-root add opensearch.username

Enable and Start the Wazuh Dashboard Service

systemctl daemon-reload
systemctl enable --now wazuh-dashboard

Update the Wazuh API Plugin for Wazuh Dashboard

I am only going to point out the lines that I had to overwrite when I ported from /usr/share/kibana/data/wazuh/config/wazuh.yml to /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml.

Edit /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml.

This...

hosts:
  - default:
     url: https://localhost
     port: 55000
     username: wazuh-wui
     password: wazuh-wui
     run_as: false

Becomes this...

hosts:
  - default:
     url: https://10.148.148.8
     port: 55000
     username: wazuh-wui
     password: <redacted>
     run_as: false

Copy Any Exported Data from Kibana

Verify if there is any data in /usr/share/kibana/data/wazuh/downloads/ that you want to preserve.

Verify Ability to Log In

Go to https://10.148.148.7 (my IP address) and log in. This is the new Wazuh Dashboard service replacing Kibana. Make sure you can access your Elasticsearch data.

Uninstall Kibana and Remove Old Data

apt-get remove --purge opendistroforelasticsearch-kibana

rm -rf /etc/kibana /var/lib/kibana

Migration Complete

If all has gone accordingly, the following should be true:

• You successfully migrated your indices to Wazuh Indexer
• You successfully upgraded Wazuh Manager to 4.3.x
• You successfully migrated to Wazuh Dashboard.