Wazuh servers have the powerful native ability to push Wazuh agent upgrades to most endpoints on request. This foundational feature can be built upon to establish:
- Full automation
- An audit trail of upgrade attempts
- Dashboard presentation of agent upgrade attempts and version compliance statistics in context
Would you like to stay keenly aware of how up-to-date your fleet of Wazuh agents is, and to notice when certain agents are recurrently failing to upgrade? In this blog I will present the Blue Wolf Ninja Wazuh Agent Upgrade Manager solution which I authored with these goals in mind.
Wazuh agent WPK upgrades
For optimal functionality, it is desirable that Wazuh agents match, but not exceed, the Wazuh version running on the Wazuh server to which they maintain a connection. If they are older, newer Wazuh features will not work with those agents. If they are running a Wazuh version newer than the one running on the Wazuh server, the server may refuse to allow those agents to connect at all. Knowing that not all of us have a general software management solution spanning all systems where we need to maintain Wazuh agents, Wazuh developed very early on in their history, an ingenious system in which Wazuh servers are able to push Wazuh agent upgrade packages known as WPK (Wazuh signed package) files, down to connected outdated agents by using the existing network connections already established by those agents to the Wazuh server. This is a robust and secure facility, using code signing, agent backup, and automatic rollback functionality. Having used it extensively, I assert this feature is now quite mature and reliable. The Wazuh SIEM natively supports manually initiating agent upgrades and checking upgrade results via the Wazuh Dashboard or directly via the Wazuh API. However, neither automation of pushing WPK upgrades, nor high level visibility of the results of upgrade attempts, are available out of the box.
Blue Wolf Ninja Wazuh Agent Upgrade Manager
For years I used a small script I wrote to be run by cron several times per day, that simply instructed the Wazuh API to initiate agent upgrades on all connected out-of-date Wazuh agents. This worked fairly well, but did not give me visibility into how well the upgrade attempts were actually performing. In my Wazuh-based MSSP context, I did not have an adequate sense of how well the fleets of agents of each of my clients’ Wazuh SIEM stacks were faring with respect to running the target version of Wazuh. It was still too easy for problematic agents to “fall through the cracks,” evading my attention for too long. Recently I refactored this solution in the form of a python script called bwn-waupg-mgr.py. The new script does the following:
- Enumerates connected outdated agents that are candidates for a WPK upgrade
- Initiates upgrade tasks for a configurable maximum number of those agents
- Periodically checks the status of each agent upgrade in progress, writing JSON log output about each upgrade attempt and its final result, until no upgrades remain in progress
- Refreshes stateful index bwn-states-agents with the latest info reported by the Wazuh API about the Wazuh version and other stateful details of each agent.
- Refreshes stateful index bwn-states-agent-version-stats with a single high level statistics document representing the version status of all Wazuh agents
- Sends a textual operational log trail to stdout accounting for the above process, separate from the upgrade results logging to the json log file
Deploying the solution
I recommend you perform the following command line steps throughout this article on your standalone Wazuh manager server, or if you have a Wazuh manager cluster, then on your master node Wazuh manager server.
Download and install the agent upgrader script
Install required Python modules, download the agent upgrader script to /usr/local/bin/, and make it executable.
pip3 install jq
pip3 install opensearch-py
pip3 install python-dateutil
curl https://raw.githubusercontent.com/BlueWolfNinja/wazuh-resources/refs/heads/main/bwn-waupg-mgr.py > /usr/local/bin/bwn-waupg-mgr.py
chmod 744 /usr/local/bin/bwn-waupg-mgr.pyThe above assumes you already have Python 3 and pip for Python 3 installed. It also takes the most common but potentially conflict-causing path of globally installing pip modules.
Please Note: Some modern distros like Ubuntu 24.04 may even refuse to globally install pip modules unless you use the –break-system-packages flag in your pip3 calls. While outside the scope of this article, it is worthwhile to consider skipping the pip3 install lines above, and instead setting up a proper Python virtual environment into which these dependencies are installed, and then updating the shebang line of the above script to reference the Python 3 interpreter in your virtual environment. This eliminates the risk of causing conflicts with other Python projects on your system that may require different versions of the same modules we are installing above. For more information, see here.
Edit the config file to reflect Wazuh Manager and Wazuh Indexer API credentials.
/etc/bwn-waupg-mgr.conf:
WAPIHOST=(IP or DNS name of the standalone or master Wazuh manager server)
WAPIUSER=(Wazuh Manager API user - usually wazuh-wui unless you create a dedicated account for this)
WAPIPASS=(Wazuh Manager API user's password)
WIHOST=(IP or DNS name of the/a Wazuh Indexer node)
WIPORT=(usually 9200)
WIUSER=(Wazuh Indexer user)
WIPASS=(Wazuh Indexer user's password)The account specified by WAPIUSER needs Wazuh API administrative permissions to initiate and check on agent upgrades as well as to read agent information in general. The wazuh-wui API account already has this.
The account specified by WIUSER must have the all_access permission to the bwn-states-* index pattern.
Configure Wazuh manager to collect the JSON upgrade accounting records produced by the script
Append the following to /var/ossec/etc/ossec.conf:
<ossec_config>
<localfile>
<log_format>json</log_format>
<location>/var/log/bwn-waupg-mgr.json</location>
</localfile>
</ossec_config>Add a custom Wazuh rule family
Create a new rule family file /var/ossec/etc/rules/bwn-waupg-mgr.xml
<group name="bwn-waupg-mgr,">
<rule id="119900" level="4">
<location>/var/log/bwn-waupg-mgr.json</location>
<description>BWN Wazuh Agent Upgrade Manager - upgrade failure</description>
</rule>
<rule id="119910" level="3">
<if_sid>119900</if_sid>
<status>^Updated$</status>
<description>BWN Wazuh Agent Upgrade Manager - upgrade success</description>
</rule>
</group>Please confirm you are not already using Wazuh rule sids 119900 or 119910. If you are, then renumber the above rules to resolve the sid conflict before proceeding.
Set the needed ownership of the new rule file.
sudo chown wazuh:wazuh /var/ossec/etc/rules/bwn-waupg-mgr.xmlFinally, create an empty version of the new json log file and restart the Wazuh manager service.
sudo touch /var/log/bwn-waupg-mgr.json
sudo systemctl restart wazuh-managerAdd a new ingest node pipeline to the alerts tripped by the above rules
While logged into the Wazuh Dashboard with admin rights, navigate to ☰ ➜ Indexer management ➜ Dev Tools, paste the following API command section into the console panel on the left, and then click on the green triangle to the right of the topmost line of this API call. This will install the dedicated bwn-waupg ingest node pipeline into your Wazuh manager cluster.
PUT _ingest/pipeline/bwn-waupg-mgr
{
"description" : "Blue Wolf Ninja Wazuh Agent Upgrade Manager Ingest Node Pipeline",
"processors" : [
{
"script": {
"description": "move data. to waupg",
"lang": "painless",
"source": "ctx.waupg = ctx.remove('data');"}
},
{
"set" : {
"description" : "Set agent.id as value of waupg.id",
"field" : "agent.id",
"value" : "{{waupg.id}}",
"ignore_failure" : true,
"tag" : "set_agent.id"
}
},
{
"set" : {
"description" : "Set agent.name as value of waupg.name",
"field" : "agent.name",
"value" : "{{waupg.name}}",
"ignore_failure" : true,
"tag" : "set_agent.name"
}
},
{
"rename" : {
"description": "Rename waupg.version to waupg.starting_version",
"field" : "waupg.version",
"target_field" : "waupg.starting_version",
"ignore_failure" : true,
"if" : "ctx.waupg?.version != null",
"tag": "waupg.version>waupg.starting_version"
}
},
{
"date" : {
"field" : "waupg.timestamp",
"target_field" : "timestamp",
"formats" : [
"yyyy-MM-dd'T'HH:mm:ss'Z'"
],
"output_format": "yyyy-MM-dd'T'HH:mm:ss.SSSZZZ",
"timezone" : "UTC",
"if" : "ctx.waupg?.timestamp != null",
"ignore_failure" : true
}
},
{
"date" : {
"field" : "waupg.update_time",
"target_field" : "timestamp",
"formats" : [
"yyyy-MM-dd'T'HH:mm:ss'Z'"
],
"output_format": "yyyy-MM-dd'T'HH:mm:ss.SSSZZZ",
"timezone" : "local",
"if" : "ctx.waupg?.update_time != null",
"ignore_failure" : true
}
},
{
"set" : {
"description" : "Set waupg.result to default value of failure",
"field" : "waupg.result",
"value" : "failure",
"ignore_failure" : true,
"tag" : "set_failure_waupg.result"
}
},
{
"set" : {
"description" : "Set waupg.result to success if success reported",
"field" : "waupg.result",
"value" : "success",
"ignore_failure" : true,
"if" : "ctx.waupg?.status == 'Updated'",
"tag" : "set_success_waupg.result"
}
},
{
"remove" : {
"field" : "waupg.id",
"ignore_missing" : true,
"ignore_failure" : true
}
},
{
"remove" : {
"field" : "waupg.name",
"ignore_missing" : true,
"ignore_failure" : true
}
},
{
"remove" : {
"field" : "waupg.timestamp",
"ignore_missing" : true,
"ignore_failure" : true
}
}
],
"on_failure" : [
{
"append": {
"field": "error.message",
"value": "{{ _ingest.on_failure_message }}",
"tag": "bwn-waupg"
}
}
]
}Add to the top level Wazuh ingest node pipeline, a conditional call to the above pipeline
On your Wazuh manager command line, edit the Wazuh-provided top level pipeline file /usr/share/filebeat/module/wazuh/alerts/ingest/pipeline.json.
Back up this file. Then near the end of this file, replace the following line
{ "remove": { "field": "service", "ignore_missing": true, "ignore_failure": true } }with the following block
{ "remove": { "field": "service", "ignore_missing": true, "ignore_failure": true } },
{
"pipeline" : {
"name" : "bwn-waupg-mgr",
"if" : "ctx.location == '/var/log/bwn-waupg-mgr.json'"
}
}and then make Filebeat push the updated top level Wazuh pipeline to the indexer cluster
sudo filebeat setup --pipelines --modules wazuhSet up a cron job to periodically run the agent upgrader script
I suggest creating a file /etc/cron.d/bwn-waupg-mgr with content like this, which will invoke the upgrader script every fourth hour and append details about what it is doing to an operational log called /var/log/bwn-waupg-mgr.log.
0 */4 * * * root PYTHONUNBUFFERED=1 /usr/local/bin/bwn-waupg-mgr.py &>> /var/log/bwn-waupg-mgr.logThen restart your cron daemon to activate the new cron job. This is normally done with either “sudo systemctl restart cron” or “sudo systemctl restart crond”.
Do an initial manual run
/usr/local/bin/bwn-waupg-mgr.pyExample operational log output
Written to the screen if run manually, or to /var/log/bwn-waupg-mgr.log if invoked by the cron.d job file entry:
Agents to attempt to upgrade:
{
"os": {
"name": "Microsoft Windows 10 Pro",
"platform": "windows",
"version": "10.0.19045.6332"
},
"id": "9582",
"name": "CLINIC-SEC",
"version": "4.11.2"
}
{
"os": {
"name": "Microsoft Windows 10 Pro",
"platform": "windows",
"version": "10.0.19045.6332"
},
"id": "9583",
"name": "HGR-RECEPTION",
"version": "4.11.2"
}
{
"os": {
"name": "Microsoft Windows 10 Pro",
"platform": "windows",
"version": "10.0.19045"
},
"id": "9700",
"name": "PAS-RECEIVING",
"version": "4.3.9"
}
Initiating upgrade tasks for agents: https://192.168.100.5:55000/agents/upgrade?agents_list=9320,9582,9583,9680,9700,9704,9705
Pausing for 300 seconds before checking on agent upgrade progress...
Checking upgrade task status for Agent 9582 (CLINIC-SEC)
Agent is updating now.
Checking upgrade task status for Agent 9583 (HGR-RECEPTION)
Agent is updating now.
Checking upgrade task status for Agent 9700 (PAS-RECEIVING)
Agent is updating now.
Pausing for 300 seconds before checking on agent upgrade progress...
Checking upgrade task status for Agent 9582 (CLINIC-SEC)
Final result found - merging and logging details. Status: "Updated".
Checking upgrade task status for Agent 9583 (HGR-RECEPTION)
Final result found - merging and logging details. Status: "Failed".
Checking upgrade task status for Agent 9700 (PAS-RECEIVING)
Final result found - merging and logging details. Status: "Updated".
Agent upgrade attempts complete
Deleting index bwn-states-agents if present...
Template creation response: {'acknowledged': True}
Created index bwn-states-agents {'acknowledged': True, 'shards_acknowledged': True, 'index': 'bwn-states-agents'}
Refresh of bwn-states-agents index and template complete.
Refreshing bwn-states-agent-version-stats...
Inserted global statistics into bwn-states-agent-version-stats.Do a one-time refresh of the wazuh-alerts-* index pattern to align it with this tool
Navigate to ☰ ➜ Dashboard management -> Dashboards Management -> Index patterns -> wazuh-alerts-*
Then click the ↻ button to reload the index pattern so it reflects the new alert fields introduced by this tool. Without this step, multiple panels in the new dashboards will not render correctly.
Wazuh Dashboard web user RBAC permissions
Wazuh web interface user accounts that are already able to use Wazuh Dashboard normally, may need to be granted the read permission for the bwn-states-* index pattern before using the following dashboards.
Download and import the dashboard suite bundle
Download with your web browser:
https://raw.githubusercontent.com/BlueWolfNinja/wazuh-resources/refs/heads/main/bwn-waupg-mgr.ndjson
In Wazuh Dashboard, navigate to ☰ ➜ Dashboard management ➜ Dashboards Management ➜ Saved objects ➜ Import (Select file) ➜ Check for existing objects (Automatically overwrite conflicts) ➜ Import (button)
The Blue Wolf Ninja Wazuh Agent Upgrade Manager Dashboard Suite
In Wazuh Dashboard, navigate to ☰ ➜ Explore ➜ Dashboards and pick either of the BWN_waupg_dash_attempts or the BWN_waupg_dash_compliance dashboard. Examples of both are below.
Upgrade Attempts Dashboard
This dashboard accounts for the history of initiating automated agent upgrade attempts and the results of those attempts. Especially keep an eye on the Attempt count by agent panel as agents with higher counts are ones for which upgrades have been repeatedly been attempted, presumably without success.
Version Compliance Dashboard
This dashboard gives high level statistics over increasingly large time windows showing how many agents that were actively connected within the last period of time were at the target Wazuh agent version, and what % that represents of those active agents. It also profiles the agents by current Wazuh agent version, OS and architecture.
Known issues
No Wazuh ARM support for WPK upgrades
In order to upgrade an outdated Wazuh agent on ARM systems, i.e., MacOS systems and many Surface Pros, you must run the desired version of the Wazuh installer package directly on the system. Push upgrades are not supported.
This script filters out agents with architecture types “arm64” or “aarch64” from the upgrade candidates list, but there may be other ARM platforms that report their architecture differently. Online Wazuh agents on such platforms will be unsuccessfully targeted for an upgrade every time the script is run. If you encounter such cases, please share details with me so that I can refine the filtering out of ARM architecture types.
I have been told by Wazuh that they plan to provide WPKs for ARM systems starting with the release of Wazuh 5.0.
Windows systems are sometimes temporarily unreceptive to WPK upgrades
This usually is caused by too many pending Windows updates piling up, or a pending Windows update related reboot being delayed for too long. In such cases, normally a Windows reboot will clear up the issue.
Much older versions of Wazuh agent may not successfully upgrade via WPK at all
If a Wazuh agent’s version is much older than that of the Wazuh manager, i.e., 4.3 vs 4.8, its WPK upgrade attempts may persistently fail. In such a case, manual uninstallation and fresh redeployment of Wazuh agent at the desired version should lead to future WPK upgrade successes for the host.
Agent architecture not consistently reported by the Wazuh manager API
In Wazuh’s GET /agents API endpoint responses, the architecture information is provided for Linux and Mac platforms, but not for Windows. In Wazuh 4.13+ environments, my script compensates for this by looking up the missing information in the new stateful wazuh-states-inventory-system-* index. If you are not using Wazuh 4.13 yet or some of your agents have not yet had a system inventory run under 4.13, you will likely see the architecture type “unknown” on your dashboards.
In rare cases, Wazuh erroneously reports certain agents as “Updated” when they did not actually update successfully.
I plan to add additional logic to this script to individually check the actual reported version of “Updated” agents before assuming they are truly updated.
Would you like a hand with this?
Feel free to comment on this blog with general questions or to inquire of me by email about commercial assistance related to deploying and/or customizing this solution for use in your own environment.