Event-Driven Systems: Stop Calling, Start Announcing

A payment gets captured. Five things need to happen because of it: the ledger books it, the warehouse reserves the stock, the customer gets a receipt, analytics counts it, and later someone adds a loyalty points service. The question this post is about is simple: whose job is it to make all of that happen?

Checkout calls everyone, and waits on everyone

The obvious answer is the checkout service's, since it's the one that knows the payment happened:

@itsnas — TypeScript

Hide line numbersCopy code

// POST /payments
async function capturePayment(req: Request) {
  const payment = await payments.capture(req.body)
 
  await ledger.book(payment.id, payment.amount)
  await warehouse.reserve(payment.orderId)
  await email.sendReceipt(payment)
  await analytics.track('payment_captured', payment)
 
  return json(201, payment)
}

main

Nas (@itsnas)

It reads well and it works. Now look at what it costs.

Latency is the sum. The customer waits for every call, in a row, including a receipt email they'll read later:

@itsnas — time-the-customer-waits-at-checkout.svg

Availability is the product. If each of those four services is up 99.9% of the time, checkout only succeeds when all four are: 0.999⁴ ≈ 99.6%. That's the difference between about 9 hours of downtime a year and about 35. Analytics being down now means customers can't pay.

Every new consumer is a change to checkout. The loyalty team can't ship until the payments team adds a fifth call, reviews it, and deploys it. Checkout has become the place where every other team's reaction to a payment lives.

The root of all three is the same: the service that did the thing is also responsible for telling everyone about it, one call at a time.

Announce what happened

Flip the responsibility. Checkout's job is to capture the payment and say so. Whoever cares listens:

@itsnas — TypeScript

Hide line numbersCopy code

async function capturePayment(req: Request) {
  const payment = await payments.capture(req.body)
 
  await events.publish('payment.captured', {
    paymentId: payment.id,
    orderId: payment.orderId,
    amount: payment.amount,
  })
 
  return json(201, payment)
}

main

Nas (@itsnas)

Each consumer subscribes on its own:

@itsnas — TypeScript

Hide line numbersCopy code

events.subscribe('payment.captured', async event => {
  await ledger.book(event.paymentId, event.amount)
})

main

Nas (@itsnas)

The message goes through a broker, something like #kafka or #rabbitmq, which stores it so each subscriber can read it on its own schedule. (RabbitMQ keeps a message until it's taken; Kafka keeps everything for a retention window, days by default, whether anyone has read it or not.)

Checkout

Broker

Ledger

Warehouse

Email

Analytics

1. Checkout to Broker: payment.captured
2. Broker to Ledger
3. Broker to Warehouse
4. Broker to Email
5. Broker to Analytics

Re-run the three costs. Checkout now waits on one write to the broker, not four services, and depends on one thing being up (the broker) instead of four. Analytics going down means analytics falls behind, and catches up from the broker when it comes back, as long as it's back within the retention window; payments keep working. And the loyalty team subscribes to payment.captured and ships without asking the payments team for anything.

Events are not commands

The word "message" covers two different things, and mixing them up is the most common way an event-driven design goes wrong.

The tell for a command wearing an event's name is an event named after what should happen next: SendReceiptEmail. That's checkout telling the email service what to do, with the coupling still there, just routed through a broker. Name the fact (PaymentCaptured) and let the email service decide that a receipt is its reaction.

Queues and topics

Brokers deliver in two shapes, and you usually need both.

A queue hands each message to exactly one of the workers reading it. Run three copies of the email service on one queue and each receipt goes to one copy, whichever is free, not to all three. That's how a consumer scales.

A topic hands each message to every subscriber. The ledger, the warehouse and the email service all get their own copy of payment.captured. That's how producers stay unaware of consumers.

Most brokers combine them. In Kafka, a topic delivers to every consumer group, and within a group each message goes to one member. So each service is a group: every service sees every payment, and within a service the work is split across its instances.

payment.captured

Ledger

Email service

instance 1

instance 2

instance 3

1. payment.captured to Ledger: a copy
2. payment.captured to Email service: a copy
3. Email service to instance 2: one instance

The bill

Nothing above is free. Here's what you sign up for:

Choreography or orchestration

Fan-out works when the reactions are independent. The ledger doesn't care whether the email went out. But some flows are a sequence with consequences. A refund has to reverse the ledger entry, then return the money through the provider, then restock, and if the provider refuses, the ledger reversal has to be undone.

Two ways to build it, both forms of a saga (local steps, each with a compensating step that undoes it):

The rule of thumb: independent reactions to a fact are choreography. A multi-step business process that can fail halfway belongs to an orchestrator, which still talks to the rest of the system through events and commands.

When a plain call is still right

One call was left out of the handler on purpose: the fraud check. It runs before the capture, and it stays a direct call:

@itsnas — TypeScript

Hide line numbersCopy code

const verdict = await fraud.check(req.body)
if (verdict.block) return json(402, { error: 'Payment declined' })

main

Nas (@itsnas)

Checkout can't answer the customer until it knows the answer, so an event doesn't help. There's nothing to do while waiting. The split that decides most cases:

That last row matters. If checkout, the ledger and the email sender are one app with one database, owned by one team, a broker adds latency, infrastructure and every item on the bill above, to decouple code that a function call already separates well enough. Events earn their place when separate services, owned by separate teams, need to react to the same thing without waiting on each other.

Back to the payment. Checkout captures it, says so, and answers the customer in the time it takes to write one message. The ledger, the warehouse, the receipt and the loyalty points all still happen, each in its own service, on its own schedule, and none of them can take payments down.

---