Backup and restore

es-tools copies a broker's data directory into one archive and restores it, on the same machine or another. It works directly on the files; no broker needs to be running.

Back up

es-tools dump --data-dir ./data --output backup.tar.gz
es-tools dump --data-dir ./data --output orders.tar.gz --topics orders,billing
es-tools dump --data-dir ./data --output full.tar.gz --include-keys --verify

--data-dir

The broker's data directory. Default ./data.

--output

Archive to write. Default backup.tar.gz. Created readable only by you.

--topics

Comma-separated topics to include. Default: all.

--include-keys

Include api_keys.json (key hashes and grants). bootstrap.key, a plaintext secret, is never included.

--verify

Check every record's CRC while copying. Slower; stops at the first corrupt record.

Restore

es-tools restore --input backup.tar.gz --data-dir ./new-data
es-tools restore --input backup.tar.gz --data-dir ./data --force

--input

Archive to restore. Default backup.tar.gz.

--data-dir

Where to restore. Default ./data.

--force

Replace a directory that already holds data. Its contents are moved aside, not deleted.

Nothing in the target is touched until the archive has been checked:

1. Extract to a staging directory next to the target, writing every file readable only by you, whatever mode the archive records.

2. Check the archive. manifest.json must come first and be a version this tool reads. Only paths a broker would create are extracted; links, .. and anything else, such as a planted bootstrap.key or an api_keys.json the manifest did not declare, are skipped. The data may not exceed what the manifest declares.

3. Move the old directory aside to <dir>.pre-restore-<timestamp> (with --force) and the restored one into its place. Delete the old copy once you have checked the restore.

A truncated archive or a mistyped --input leaves the existing data exactly as it was. When the broker next starts, its recovery checks every segment.

What is in an archive

A standard gzip-compressed tar; tar tzf backup.tar.gz lists it.

manifest.json                         # version, time, topics, file count, total size
topics/orders/topic.json
topics/orders/0/00000000000000000000.log
topics/orders/0/00000000000000000000.index
groups/analytics.json
schemas.json
producers.json, producers.journal
api_keys.json                         # only with --include-keys

Segment files are copied byte for byte. Raft state is not included: restore a cluster member from a backup only into a new cluster, or let it rejoin and catch up from the leader instead.

Moving a broker to another machine

# On the old machine: stop the broker, then back up and verify
systemctl stop es-broker
es-tools dump --data-dir /var/lib/es --output es.tar.gz --include-keys --verify

# On the new machine
es-tools restore --input es.tar.gz --data-dir /var/lib/es
es-broker --data-dir /var/lib/es --bind 10.0.0.9:9000 --auth required

# Check
ES_BROKER=http://10.0.0.9:9000 es --auth-file ./admin.key topic describe --name orders