Information Lifecycle Management (ILM) in the Wazuh SIEM is usually achieved by setting up Index State Management (ISM) policies and applying them to Wazuh indexes. An ISM policy consists of one or more states through which a “policy managed” index will pass according to its age. For example, a policy could be applied to wazuh-alerts-* indexes, that transitions them from a hot to a warm state at 30 days old by moving them from hot-tier to warm-tier indexer nodes, and then at 180 days old, transitions them to a deleted state by removing them entirely.

That is the plan, but I have repeatedly encountered cases where past indexes, though managed by the intended ISM policy, were not transitioned out of their original ISM state even though they had reached and exceeded the age criteria in the policy. Re-indexing turns out to be the unexpected cause, and the effect can be an indexer cluster running out of shards or disk space due to older indexes not being deleted when intended. The solution requires a different approach to Wazuh ILM.

Re-indexing throws ISM-based ILM a fatal curveball

At times it is necessary to re-index some or many of your Wazuh indexes to perform actions like:

Resolve critical mapping conflicts
Change the shard count of indexes
Retroactively transform field data

This re-indexing process creates a new index from an old one, setting the new index’s immutable creation_date attribute to the time when the new index is created, not the time when the original index was created. The Wazuh ISM facility uses an index’s creation_date setting to determine its age, which means a year-old index that was recently re-indexed will look to an ISM policy like it has a young age, greatly skewing the intended timing of the ISM policy applied to it.

This is a fundamental shortcoming that Wazuh inherits from the OpenSearch ISM facility, and there is as yet no way to make an ISM policy accurately recognize the true age of re-indexed indexes. Fortunately, ISM is not the only way to achieve ILM of Wazuh indexes.

Curator-based ILM to the rescue

Curator is a powerful tool for managing Elasticsearch indexes, especially via automation. While not directly compatible with Wazuh or OpenSearch, the curator-opensearch project forked Elasticsearch’s Curator (version 5.8.4) and made it compatible with OpenSearch and thus also with Wazuh. Hereafter, I will mostly refer to curator-opensearch as simply Curator.

Curator empowers us via yaml job files, to implement most of the same kind of age-based actions taken by ISM policies, but unlike ISM, curator jobs can be configured to calculate index age based on the date suffix in the index name rather than by the creation_date attribute. Thus, Curator-based ILM can factor in the original age of re-indexed indexes, allowing us to fire off our intended ILM actions at the right time, even for re-indexed indexes.

Implementing Curator

Curator is fairly easy to implement on Wazuh SIEMs as an alternative to ISM, though this is accomplished at the CLI level instead of in Wazuh Dashboard. It is generally best to install it on the Wazuh indexer node of your choice, though this is not required. Python 3 is required, and a pip install of curator-opensearch will pull in all the necessary dependencies. In this blog, I provide an example of installing curator-opensearch in a Python virtual environment on an Ubuntu 24.04 server, with a recurring Curator job to delete indexes more than 365 days old. Fairly small variations on the following steps should make the same process work on different flavors and versions of Linux.

Unapply ISM policies and disable ISM plugin

To avoid any conflicts between ISM and Curator, bring all wazuh-alerts-* indexes out from under ISM management, and persistently disable the ISM plugin itself. It is best not to mix ISM and Curator functionality on the same Wazuh SIEM.

Navigate to ☰ ➜ Indexer management ➜ Dev Tools and execute these two API calls:

POST _plugins/_ism/remove/wazuh-alerts-*

PUT /_cluster/settings
{
  "persistent": {
    "plugins.index_state_management.enabled": "false"
  }
}'

Safely install curator-opensearch

This approach allows the many dependencies of curator-opensearch to be met without risk of creating conflicts with other Python-based programs installed on the same system that require different versions of those same dependencies. Run the following as root.

apt install python3-virtualenv
mkdir /venv
cd /venv
virtualenv curator
source curator/bin/activate
pip3 install curator-opensearch
deactivate
sed -i 1d /venv/curator/bin/curator /venv/curator/bin/curator_cli
sed -i '1i#!/venv/curator/bin/python3' /venv/curator/bin/curator /venv/curator/bin/curator_cli
ln -fs /venv/curator/bin/curator /usr/local/bin/curator
ln -fs /venv/curator/bin/curator_cli /usr/local/bin/curator_cli

Connect Curator to Wazuh Indexer

Create the file /root/.curator/curator.yml with this content:

client:
  hosts:
    - >>>INDEXER NAME<<<
  port: 9200
  url_prefix:
  use_ssl: True
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: True
  http_auth: >>>USERNAME<<<:>>>PASSWORD<<<<
  timeout: 30
  master_only: False
logging:
  loglevel: INFO
  logfile: /var/log/curator.log
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

Replace the following in the above file

>>>INDEXER NAME<<< with the name or IP of your Wazuh indexer for Curator to manage
>>>USERNAME<<< with an admin-privileged Wazuh indexer username like “admin”
>>>PASSWORD<<<< with the password for the above username

Save your changes and exit the file.

Test Curator

Confirm that Curator is working by requesting an interactive listing of all Wazuh indexes.

/usr/local/bin/curator_cli show-indices

If the output consists of a clean index list, then continue. Otherwise revisit the curator.yml file, revise it, and retest until you get things working.

Configure a Curator job

To make Curator delete year-old indexes, create a file /root/.curator/prune.yml containing the following:

---
actions:
  1:
    action: delete_indices
    description: "Delete old alerts indices"
    options:
      ignore_empty_list: True
      disable_action: False
      timeout_override: 300
      continue_if_exception: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: wazuh-alerts-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 365
      exclude:

Test the Curator job

Make sure you really do want to delete all wazuh-alerts-* indexes that are more than a year old, adjusting the job if you want something different. Also make sure you have a recent snapshot of all wazuh-alerts-* indexes. Only after attending to these things, should you manually run the Curator job. This may delete indexes!

curator /root/.curator/prune.yml

Watch the output of the above job execution, and examine further log output in /var/log/curator.log. If it appears to have run as intended, then you are ready to automate it.

Set up cron to run this job daily

Create /etc/cron.d/prune-indices to run this job daily at 5 minutes after midnight.

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# Once per day, prune old indices from Wazuh-Indexer
5 0 * * * root curator /root/.curator/prune.yml

Remember to restart cron

systemctl restart cron

For further consideration

Curator is capable of much more than just deleting old indexes at the right time. It has a feature set similar to that of the ISM facility. For example:

alias             Add/Remove index(es) to/from an alias
allocation        Apply shard allocation filtering rules to indexes
close             Close index(es)
cluster_routing   Apply routing rules to the entire cluster
create_index      Create an index
delete_indices    Delete index(es)
delete_snapshots  Delete snapshot(s)
forcemerge        Force-merge index(es) to reduce segment count
freeze            Freeze index(es)
index_settings    Change settings for index(es)
open              Open index(es)
reindex           Re-index index(es)
replicas          Change replica count of index(es)
restore           Restore index(es) from a snapshot
rollover          Rollover index associated with alias
shrink            Shrink index(es) to have fewer shards
snapshot          Snapshot index(es) to a repository
unfreeze          Unfreeze index(es)

Read here for details about the broader set of actions Curator can take.

Thanks for reading!

When OpenSearch forked Elasticsearch, the original Elasticsearch ILM facility was excluded due to licensing restrictions. The OpenSearch team ultimately developed their own ISM solution, which has matured quite a bit in the last few years, even exceeding in flexibility and functionality Elasticsearch’s own ILM in some areas. One final area of immaturity in OpenSearch ISM which long ago was fixed in Elasticsearch ILM, is the inability to perform ISM actions based on the original age of re-indexed indexes. I’m sure eventually OpenSearch will fix this gap in ISM, making it the superior choice for automated Wazuh ILM in most cases. In the meantime, we can be thankful that the curator-opensearch project has given us a tool that can meet our needs in this area right now. Having performed extensive re-indexing on most of my clients’ Wazuh indexer clusters, this solution has met my own ILM needs quite well over the last year or so. I hope it proves useful to you as well.

As always, you are welcome to comment with general questions or to inquire of me by email about possible direct private assistance related to this or other Wazuh issues. I’d love to hear from you!