Events on disk, in order.

es is an event-streaming broker in one Rust binary: topics, partitions, consumer groups and Raft replication. A produce call returns its offsets only once the records are fsynced, and it still moves millions of records a second.

Run the quickstart Read the docs

Topic orders, 3 partitions

orders/0 1204

orders/1 983

orders/2 1117

Group billing has committed 1190 on orders/0. Each square is one record; the number is the next offset.

What ships in the binary

No ZooKeeper, no JVM, no client library to install. One process per node, a CLI, and an HTTP API you can call with curl.

Storage

Durable by default. One write and one fsync per request, shared between concurrent requests.
CRC-checked segment files with a sparse index.
Retention by age or size, and key compaction with tombstones.
Crash recovery that trims a torn tail and never touches newer segments.
Tiered storage: expired segments move to a cold directory instead of being deleted.

Replication

Raft per partition: leader election and batched log replication.
Late or rebuilt replicas catch up by streaming the leader's records.
The leader assigns offsets and timestamps, so replicas hold identical records.
Peers prove a shared secret without sending it; every frame is authenticated.

Access control

API keys with read, write and admin grants by topic prefix.
Produce and consume byte-rate quotas per key.
Consumer groups and producer ids belong to the key that created them.
TLS on both the HTTP and the binary listener.

Protocols

HTTP and JSON for everything, with base64 for binary values.
A binary protocol with pipelining and gzip for the hot path.
Idempotent producers: retried records are written once.
A schema registry with backward-compatibility checks.

Operations

Consumer groups with sticky rebalancing and heartbeats.
Prometheus metrics, scoped to what the caller's key can read.
Offline dump and restore that validates an archive before replacing anything.
Graceful shutdown that drains connections and flushes state.

Throughput with fsync on

Every number below is measured with the default durability: nothing is acknowledged before it is on disk.

Binary produce8 connections, batches of 500

4.6M rec/s

HTTP and JSON produce8 clients, batches of 500

3.7M rec/s

Binary consume1 connection, 4 MiB fetches

3.0M rec/s

Raft partition4 connections, batches of 100

1.0M rec/s

Small pipelined writes64 requests in flight, 10 records each

118k rec/s

256-byte records over loopback, release build, on a 32-core server with NVMe storage; the Raft row is a single-node group. Batch size matters most: small requests pay for their own fsync. The bench is crates/es-broker/tests/bench.rs; read the method and run it on your hardware.

Running in three commands

Build the workspace, start a broker on localhost, then write and read a few records with the CLI.

Build

Needs a Rust toolchain. Produces es-broker, es and es-tools.
shell
```
cargo build --release
```
Start a broker

Data goes to ./data. Auth is off, which the broker only allows on loopback.
shell
```
./target/release/es-broker \
  --data-dir ./data \
  --bind 127.0.0.1:9000
```

Produce and consume

Create a topic, write a record, read it back from offset 0.

shell

export PATH="$PWD/target/release:$PATH"
export ES_BROKER=http://127.0.0.1:9000
es topic create --name orders --partitions 2
es produce --topic orders --key k1 --value hello
es consume --topic orders --partition 0 --offset 0

Walk through the full quickstart