es v0.2.0

Quickstart

Build es from source, start a broker on your machine, and write and read records with the CLI. Or let one command do all of it, including a restart, retention and compaction.

Before you start

  • A Rust toolchain. The repository pins the version in rust-toolchain.toml; rustup installs it on first build.
  • A free port on localhost. The examples use 9000; the demo script uses 9876 (change it with ES_PORT).

Run the demo

shell
make demo

The script runs these steps and stops at the first failure. It ends with DEMO OK.

  1. Reset. Deletes ./data so every run starts the same way.

  2. Build. Compiles the workspace in release mode.

  3. Start the broker with --segment-bytes 1024, a deliberately tiny segment size, and waits for /healthz.

  4. Produce 200 records into orders, two partitions. The small segments make several rolls per partition visible in es topic describe.

  5. Consume through a group. Group analytics reads five records and commits offset 5.

  6. Restart. Stops the broker, starts it again, and checks the group resumes at offset 5.

  7. Retention and compaction. A topic with a 1.5-second retention loses its old segments; a compacted topic keeps only the latest value per key.

Run it by hand

Start a broker in one terminal:

terminal 1
cargo build --release
./target/release/es-broker \
  --data-dir ./data \
  --bind 127.0.0.1:9000

Then create a topic, write a record and read it back in another:

terminal 2
export ES_BROKER=http://127.0.0.1:9000
./target/release/es topic create --name orders --partitions 2
./target/release/es produce --topic orders --key k1 --value '{"id":1}'
./target/release/es consume --topic orders --partition 0 --offset 0

The record lands on the partition its key hashes to, so with two partitions it may be in partition 1. es topic describe --name orders shows each partition's end offset.

Call the HTTP API directly

Everything the CLI does is a plain HTTP call:

shell
curl -s -X POST localhost:9000/topics/orders/produce \
  -H 'content-type: application/json' \
  -d '{"records":[{"key":"k2","value":"hello"}]}'

curl -s 'localhost:9000/topics/orders/consume?partition=0&offset=0'

Run the tests

The suite boots real brokers on ephemeral ports and drives them over HTTP, the binary protocol and the Raft transport: durability and recovery, retention and compaction, auth and quotas, consumer groups, replication and snapshot catch-up.

shell
cargo test --workspace