CLI and broker flags

es is a command-line client for the HTTP API. es-broker is the server. This page covers both.

Connecting

The CLI reads the broker address and the key from flags or the environment:

--broker, ES_BROKER: Broker URL. Default http://127.0.0.1:9000.
--auth, ES_AUTH: API key, when the broker runs with --auth required.
--auth-file: Read the key from a file instead, such as data/bootstrap.key. Keeps it out of shell history.

shell

export ES_BROKER=https://broker.internal:9000
es --auth-file ./secrets/es.key topic list

A session

shell

es topic create --name orders --partitions 3
es produce --topic orders --key alice --value '{"action":"checkout"}'
partition=0 offset=0
es produce --topic orders --key bob --value '{"action":"signup"}'
partition=1 offset=0

# One record per line of a JSON-lines file
es produce --topic orders --from-file events.jsonl

es consume --topic orders --partition 0 --offset 0
offset=0 key=alice value={"action":"checkout"}

# Read as a group, then record how far you got
es consume --topic orders --partition 0 --group analytics
es commit --group analytics --topic orders --partition 0 --offset 2
es group show --name analytics

Topics

shell

es topic create --name orders --partitions 3 \
  --retention-ms 86400000 --retention-bytes 1073741824 \
  --cleanup-policy compact,delete --segment-bytes 67108864 \
  --tombstone-retention-ms 86400000
es topic alter --name orders --retention-ms 3600000   # omitted flags keep their value
es topic list
es topic describe --name orders                        # partitions, offsets, sizes
es topic show-config --name orders
es topic delete --name orders

--retention-ms: Delete sealed segments whose newest record is older than this.
--retention-bytes: Delete the oldest sealed segments until the partition is under this size.
--cleanup-policy: delete, compact or compact,delete.
--segment-bytes: Segment size for this topic; applies from the next roll.
--tombstone-retention-ms: How long compaction keeps a key's empty-value tombstone.

Produce and consume

shell

es produce --topic orders --key k1 --value hi             # routed by key
es produce --topic orders --key k1 --value hi --partition 0
es produce --topic orders --from-file events.jsonl       # instead of --value

es consume --topic orders --partition 0 --offset 0 --max 100
es consume --topic orders --partition 0 --group analytics  # from the group's offset

Consume prints one line per record and a summary with next_offset and high_watermark. Reading through a group does not commit; run es commit.

Idempotent producers

shell

es produce --topic orders --partition 0 --value '…' \
  --producer-id checkout-svc-1 --sequence 42
partition=0 offset=57
# Running it again changes nothing
partition=0 offset=57 (duplicate)

# From a file, sequences count up from --sequence
es produce --topic orders --partition 0 --from-file events.jsonl \
  --producer-id checkout-svc-1 --sequence 100

es producer list
es producer revoke --id checkout-svc-1

Consumer groups

shell

es group join --name analytics --topic orders --topic billing
es group heartbeat --name analytics --member-id m_abc… --generation 3
es group assignment --name analytics --member-id m_abc…
es group leave --name analytics --member-id m_abc…
es group show --name analytics
es reset-offsets --topic orders     # every group back to the first offset

rebalance_required from a heartbeat means the assignment changed: fetch it again. unknown_member means the member timed out: join again.

Keys

shell

es key create --name orders-writer \
  --acl write:orders. --acl read:billing \
  --produce-bytes-per-sec 1048576
es key list
es key revoke --id key_a1b2c3

Schemas and cold storage

shell

es schema register --subject orders-value --type json_schema --schema ./schemas/order.json
es schema list
es schema get --id 1
es schema latest --subject orders-value

es tiered list --topic orders --partition 0     # base offsets of offloaded segments

Broker flags

shell

es-broker --data-dir /var/lib/es --bind 10.0.0.5:9000 --auth required \
  --tls-cert /etc/es/cert.pem --tls-key /etc/es/key.pem

Network and security

--bind: HTTP address. Default 127.0.0.1:9000; port 0 picks a free one.
--bind-binary: Also listen for the binary protocol on this address. Off by default.
--auth: disabled (default) or required. Disabled is refused on a non-loopback address.
--allow-remote-unauthenticated: Allow --auth disabled on a non-loopback address. Every caller is then an admin.
--allowed-host: Extra host name accepted while auth is disabled (repeatable).
--tls-cert, --tls-key: PEM files. TLS for the HTTP and binary listeners; give both.
--cors-origin: Origin allowed to call the API from a browser (repeatable). None by default.

Storage

--data-dir: Where everything is stored. Default ./data.
--segment-bytes: Default segment size. Default 64 MiB.
--flush-every-records: 1 (default): acknowledge only what is fsynced. Above 1: fsync once that many records are waiting.
--flush-interval: With the above over 1, fsync at least this often. Default 1s.
--retention-check-interval: How often retention runs. Default 30s.
--compaction-check-interval: How often compaction runs. Default 60s.
--segment-delete-grace: Delay before deleting a segment retention dropped. Default 60s.
--default-tombstone-retention-ms: Tombstone retention for topics that do not set one. Default one day.
--cold-storage-dir: Move segments retention drops here instead of deleting them.

Limits

--max-record-bytes: Largest record (key plus value) on either protocol. Default 8 MiB.
--max-fetch-bytes: Most one consume returns. Default 8 MiB.
--max-fetch-records: Most records one consume returns. Default 10,000.
--max-request-body-bytes: Largest HTTP body. Default 10 MiB.
--binary-max-connections-per-ip: Binary connections from one address. Default 256.
--binary-idle-timeout: Close an idle binary connection after this. Default 10m.
--shutdown-timeout: How long a shutdown waits for connections to drain. Default 30s.

Clustering

--raft-node-id: This broker's id in the cluster. Setting it replicates every topic.
--raft-bind: Address for traffic from the other members.
--raft-peer: <id>=<host:port>, once per other member.
--raft-shared-secret, ES_RAFT_SHARED_SECRET: Secret the members prove to each other. Required unless the Raft address is loopback.

Logs go to stderr; set RUST_LOG=info (the default) or debug. The broker shuts down gracefully on SIGINT and SIGTERM. See Clustering with Raft for a full setup.