Event Driven

event-driven architecture inverts the direction of knowledge: instead of the caller knowing who must react, the reactor knows what it cares about. producers announce facts; consumers subscribe. the whole style is the observer pattern with a broker in the middle and a network underneath — which is exactly where the interesting failure modes come from.

events, commands, queries#

three message species, constantly confused:

• event: a fact about the past, named in past tense — OrderPlaced, PaymentFailed. immutable, owned by the producer, zero expectation about who (if anyone) reacts. broadcasting is safe because nothing is being asked.
• command: a request for the future — PlaceOrder. imperative mood, addressed to exactly one handler, can be rejected. commands express intent; events record outcome. compare the command pattern, which reifies exactly this.
• query: a question, side-effect free, wants an answer now.

the classic smell is the command in event’s clothing: SendWelcomeEmailRequested published as an “event” that exactly one consumer must process, or the producer silently depends on a specific reaction — you have built rpc with extra steps and none of rpc’s error handling. 𐃏

pub/sub and brokers#

two semantics, one word#

“message broker” covers two genuinely different machines, and most design errors come from expecting one to behave like the other:

• queue (rabbitmq-style): the broker actively pushes work to competing consumers; an unacked message is redelivered (possibly to another consumer); an acked message is destroyed. the broker tracks per-message state.
• log (kafka-style): the broker is a dumb append-only file per partition. consumers pull, and the only per-consumer state is an integer — the offset, “i have processed everything before position \(n\)”. committing an offset is a consumer-side act, wholly decoupled from message lifetime — which is what makes replay, late-joining consumers and reprocessing-after-a-bugfix possible. 𐃏

consumer groups and partitions#

• a topic is split into \(p\) partitions; records with the same key hash to the same partition, so ordering is guaranteed per key, never per topic.
• a consumer group divides partitions among its members: each partition is owned by exactly one consumer in the group at a time. two groups reading the same topic each see everything — pub/sub and work-sharing from one primitive.
• consequence: parallelism is capped at \(p\). a topic with 6 partitions feeds at most 6 active consumers per group; the 7th idles. choose keys to spread load, and expect hot partitions when one key (one big customer) dominates.

delivery guarantees, honestly#

the guarantee is a property of the loop (receive, process, acknowledge), not of the broker box on the diagram:

• at-most-once: ack (or commit the offset) before processing. crash mid-process and the message is lost. acceptable for metrics ticks, disastrous for money.
• at-least-once: ack after processing. crash between processing and ack, and the message is redelivered — every consumer will eventually see duplicates. this is the default posture of every serious system.
• exactly-once: as a transport guarantee across arbitrary systems, this does not exist — the ack itself can be lost, and the broker cannot know whether your side effect happened. 𐃏 what you can build is exactly-once processing: at-least-once delivery plus deduplication at the consumer.

idempotency keys#

• producer stamps each logical operation with a unique key (uuid, or a hash of the business operation).
• consumer records processed keys — ideally in the same transaction as the side effect — and drops any message whose key it has seen.
• the effect: \(f(f(x)) = f(x)\); retries become free. naturally idempotent operations (set balance = 120, upserts) need no table; non-idempotent ones (increment balance, send email) always do.
• the producer-side twin is the outbox pattern from microservices: local transaction + relay gives at-least-once publication; idempotency keys give at-most-once effect. compose both and the whole pipeline is exactly-once in effect.

ordering and partitioning#

• global order across a distributed topic is a fiction you pay for with throughput (one partition = one writer’s ordering = one consumer’s parallelism). the honest contract: order per key, concurrency across keys.
• design events so per-key order is all you need: all events for account 42 share key 42, and no invariant spans accounts 42 and 43 within the stream.
• consumers must still tolerate interleaving anomalies: a rebalance can hand a partition to a new consumer that re-reads uncommitted records (duplicates again), and cross-partition joins see arrival order, not event-time order. carry timestamps in the payload and reason in event time when it matters.

event sourcing#

state = fold(events)#

instead of storing current state and losing history, store the history and derive the state. if \(e_1, e_2, \dots, e_n\) is the append-only log and \(a\) the apply function,

\begin{equation} s_n = a(s_{n-1}, e_n), \qquad s_0 = \text{empty}, \qquad \text{state} = \operatorname{fold}(a,\ s_0,\ [e_1, \dots, e_n]). \end{equation}

the database’s own recovery machinery works exactly this way — replaying a write-ahead log to reconstruct state (Silberschatz, Abraham and Korth, Henry F. and Sudarshan, S., 2019) — event sourcing just makes the log the system of record and the “current state” a cache.

• you gain: a perfect audit trail (the ledger is the data), time travel (fold a prefix), retroactive bug fixes (fix the fold, replay), new read models for free (new fold over old history).
• you pay: schema evolution forever (old events never die), snapshotting machinery once logs get long (fold from the last snapshot, not from genesis), and the up-front cost that queries against “current state” all need a projection.

a worked bank account#

the whole idea in fifty lines: commands validate against folded state, then append facts; state is always derived, never stored.

from dataclasses import dataclass

# --- events are facts: past tense, immutable ---
@dataclass(frozen=True)
class Opened:      owner: str
@dataclass(frozen=True)
class Deposited:   amount: int
@dataclass(frozen=True)
class Withdrawn:   amount: int

# --- state = fold(apply, events, empty) ---
@dataclass(frozen=True)
class Account:
    owner: str = ""
    balance: int = 0

def apply(state, event):                     # the fold step: (state, event) -> state
    match event:
        case Opened(owner):     return Account(owner, 0)
        case Deposited(amount): return Account(state.owner, state.balance + amount)
        case Withdrawn(amount): return Account(state.owner, state.balance - amount)

def replay(events):
    state = Account()
    for e in events:
        state = apply(state, e)
    return state

# --- the command side: validate against current state, append new facts ---
class EventStore:
    def __init__(self):
        self.log = []                        # the only mutable thing in the system

    def state(self, upto=None):
        return replay(self.log[:upto])

    def execute(self, command, arg):
        s = self.state()
        if command == "withdraw" and arg > s.balance:
            raise ValueError(f"insufficient funds: balance {s.balance}, asked {arg}")
        event = {"open": Opened, "deposit": Deposited, "withdraw": Withdrawn}[command](arg)
        self.log.append(event)
        return event

store = EventStore()
for cmd, arg in [("open", "aayush"), ("deposit", 100), ("deposit", 50), ("withdraw", 30)]:
    e = store.execute(cmd, arg)
    print(f"appended {e!r:<22} -> state {store.state()}")

try:
    store.execute("withdraw", 1000)
except ValueError as err:
    print(f"command rejected: {err}")

print("\ntime travel — state after each prefix of the log:")
for i in range(len(store.log) + 1):
    print(f"  events[:{i}] -> {store.state(upto=i)}")

appended Opened(owner='aayush') -> state Account(owner='aayush', balance=0)
appended Deposited(amount=100)  -> state Account(owner='aayush', balance=100)
appended Deposited(amount=50)   -> state Account(owner='aayush', balance=150)
appended Withdrawn(amount=30)   -> state Account(owner='aayush', balance=120)
command rejected: insufficient funds: balance 120, asked 1000

time travel — state after each prefix of the log:
  events[:0] -> Account(owner='', balance=0)
  events[:1] -> Account(owner='aayush', balance=0)
  events[:2] -> Account(owner='aayush', balance=100)
  events[:3] -> Account(owner='aayush', balance=150)
  events[:4] -> Account(owner='aayush', balance=120)

note what fell out for free: the rejected overdraft never touched the log (commands are validated, events are facts), and “time travel” is just folding a prefix — this is the memento pattern where the mementos happen to be the data itself. a production store adds snapshots (persist \(s_k\) every \(k\) events, replay from there) and optimistic concurrency (append fails if the log grew since you read it).

cqrs#

command query responsibility segregation: split the write model from the read model(s).

• the write side accepts commands, enforces invariants, emits events — normalised, consistent, small.
• the read side subscribes to those events and maintains denormalised projections: one table shaped exactly like each screen or query. no joins at read time; the join happened at projection time.
• the price is eventual consistency between the sides: a user can write and then read a stale projection. mitigations — read-your-own-writes from the write side, or return the new state in the command response.
• cqrs does not require event sourcing (a plain db plus change-events works) and event sourcing does not require cqrs — but they compose naturally: the fold above is a projection, and every projection is just another fold with a different apply function.
• adopt asymmetrically: most systems need cqrs on two or three hot aggregates, not everywhere. full-system cqrs + event sourcing is an architecture astronautics smell — see code smells for the in-process analogues.

schema evolution#

events outlive the code that wrote them; a five-year-old OrderPlaced v1 must still fold correctly today.

• compatibility direction: backward compatibility = new code reads old events (mandatory for event sourcing — history is forever); forward compatibility = old code reads new events (needed during rolling deploys, when producers upgrade first).
• safe changes: add optional fields with defaults; never rename, never retype, never reuse a field number/name for a different meaning.
• mechanics: schema’d formats (avro/protobuf/json-schema) plus a schema registry that rejects incompatible producer schemas at publish time — turning a 3 a.m. consumer crash into a ci failure.
• when a change is genuinely breaking: upcasters (translate v1 to v2 at read time, chainable) or publish to a new versioned topic and migrate consumers deliberately.

backpressure#

producers and consumers are decoupled in time, not in arithmetic: if arrival rate exceeds service rate for long enough, the queue grows without bound — the broker just decides where the pile of unprocessed work sits.

• bounded queues everywhere: an unbounded buffer converts overload into an out-of-memory crash at maximum distance from the cause. bound the queue and choose a policy — block the producer (backpressure proper), shed load (drop with a metric), or spill to disk (a log broker’s native move).
• pull beats push under load: pull-based consumers (kafka-style) take work at their own pace, so pressure surfaces as consumer lag — one number (log head minus committed offset) that tells you precisely how far behind reality each group is. alert on lag, not on cpu.
• slow-consumer handling: scale consumers up to the partition cap, then scale partitions; make handlers faster (batch the db writes); or admit the stream has two speeds and split it (hot path folds counters, cold path does the heavy enrichment).
• reactive-streams credit protocols (consumer grants \(n\) message credits upstream) are the same idea formalised in-process.

see also#

• microservices — sagas and the outbox pattern, the transactional glue for all of this
• api design — the synchronous counterpart: request/response contracts and webhooks
• design patterns — observer, command and memento, before the network got involved
• databases — write-ahead logs, transactions and the machinery event stores borrow
• lan messenger — a from-scratch pub/sub-ish messaging system over raw sockets

References#

Silberschatz, Abraham and Korth, Henry F. and Sudarshan, S. (2019). Database System Concepts, McGraw-Hill Education.