Overview

es is an event-streaming broker written in Rust. It stores records in append-only, partitioned topics, hands each one an offset, and lets consumers read from any offset or resume through a consumer group. It runs as a single binary, alone or as a Raft cluster.

The model

If you have used Kafka, you already know it. A topic is split into partitions. Each partition is an ordered log: a record appended to it gets the next offset, and the order never changes afterwards. Producers choose a partition explicitly or let the broker pick one from the record's key, so records with the same key stay in order. Consumers read a partition from any offset; a consumer group stores how far it has read, so a restarted consumer resumes where it stopped.

What it does

Area	What you get
Storage	CRC-checked segment files with a sparse index. Nothing is acknowledged before it is fsynced, one fsync per request shared between concurrent requests. Crash recovery trims a torn tail and leaves every other segment alone.
Cleanup	Retention by age or size, key compaction with tombstones, and tiered storage that moves expired segments to a cold directory instead of deleting them.
Clients	An HTTP and JSON API for everything, a binary protocol with pipelining and gzip for the hot path, and the `es` CLI.
Delivery	Consumer groups with explicit commits, a coordinator with sticky partition assignment, and idempotent producers whose retries are written once.
Replication	Raft per partition: leader election, batched replication, snapshot transfer to late replicas, and a mutually authenticated transport between brokers.
Security	TLS, API keys with grants by topic prefix, per-key byte-rate quotas, and groups and producer ids owned by the key that created them.
Operations	Health and readiness endpoints, Prometheus metrics, a schema registry, offline dump and restore, and graceful shutdown.

What it does not do

These Kafka features are deliberately left out:

Transactions across partitions. They need a coordinator, epochs and fencing.
Linearizable reads. A replica serves what it has applied; delivery is at least once.
Record headers. A record carries a key, a value and a timestamp.
Compression on disk. Segments are stored as written; gzip is available on the binary protocol.
Object-store tiering. The cold-storage interface is pluggable, but only the local-directory backend ships.

The workspace

Repository layout

es/
├── Cargo.toml             # workspace
├── Makefile               # build, run-broker, demo, test, lint, audit
├── scripts/demo.sh        # end-to-end run, ends with DEMO OK
├── crates/
│   ├── es-broker          # the broker: storage, HTTP, binary protocol, Raft
│   ├── es-protocol        # request and response types, binary wire format
│   ├── es-cli             # the es command
│   └── es-tools           # offline dump and restore
└── data/                  # created at runtime
    ├── topics/<topic>/<partition>/*.log, *.index
    ├── groups/<group>.json
    ├── raft/<topic>/      # only on cluster members
    └── api_keys.json      # only with --auth required

Where to go next

Quickstart takes you from source to a running broker and a consumer group in a few minutes.
HTTP and binary API lists every endpoint, the binary frames, auth and limits.
CLI and broker flags covers the es command and every es-broker option.
How storage works explains segments, durability, recovery and the benchmark.
Clustering with Raft sets up a three-node cluster and explains replication.