What Breaks From 1k to 1M Requests Per Second

"Scale" gets talked about as one thing, a bigger number on a graph. It isn't. Each order of magnitude breaks a different assumption, and the fix for one tier is often irrelevant, or actively wrong, at the next.

This post follows one endpoint, a product page read (GET /products/:id), through four traffic tiers: 1k, 10k, 100k, and 1M requests per second. At each tier, the question is the same: given the exact same code, what's the first thing that actually gives. Then, why that thing and not something else.

The tool doing most of the explaining is Little's law: the average number of requests in flight at any instant equals the arrival rate times how long each one takes to finish (L = λW).

It's the reason a system can look completely fine, then fail the moment throughput crosses a specific number, with nothing else about the code having changed.

1k req/s: almost anything survives

Here's the naive version of the endpoint: one instance, one query, straight to the primary.

@itsnas — TypeScript

Hide line numbersCopy code

// GET /products/:id
async function getProduct(req: Request) {
  const [product] = await db`
    SELECT id, name, price, description
    FROM products
    WHERE id = ${req.params.id}
  `
  if (!product) return json(404, { error: 'Not found' })
 
  return json(200, { product })
}

main

Nas (@itsnas)

Say this query takes 8ms on a warm index. By Little's law, 1k req/s x 0.008s means about 8 of these queries are in flight at any given instant. A default connection pool of 20, against #postgres's own default max_connections of 100, swallows that without noticing.

This is the uncomfortable part of the 1k tier: it isn't validation. It's cover. An N+1 query, a rate limiter that keeps its counter in a plain in-memory Map, a cache that lives in module-level process memory: all of it works at this volume, on a single instance.

None of it says anything about whether the design is sound. It only says the design hasn't been tested yet.

10k req/s: the database and the single instance give first

Run the same math at 10k req/s: 10k x 0.008s = 80 connections needed in flight, for this one query, on one endpoint. That's already most of Postgres's default 100-connection ceiling, before any other route in the service has asked for a single connection of its own.

The second problem at this tier isn't the database at all. A single Node instance, even doing nothing but this query, cannot physically push 10k req/s through one event loop once there's any real work attached to each request (serialization, validation, logging).

Getting to 10k means running multiple instances behind a load balancer, which quietly breaks anything that assumed "one process" as a scaling unit. A per-instance rate limiter's real-world limit is now (configured limit x instance count). A per-instance in-memory cache's effective hit rate is divided by however many instances are running, because each one is caching its own separate copy.

The fix is the same shape for both problems: stop treating per-instance state as if it were global state, and stop asking Postgres to hold more connections than it can actually serve.

@itsnas — TypeScript

Hide line numbersCopy code

async function getProduct(req: Request) {
  const cacheKey = `product:${req.params.id}`
  const cached = await redis.get(cacheKey)
  if (cached) return json(200, { product: JSON.parse(cached) })
 
  const [product] = await db`
    SELECT id, name, price, description
    FROM products
    WHERE id = ${req.params.id}
  `
  if (!product) return json(404, { error: 'Not found' })
 
  await redis.set(cacheKey, JSON.stringify(product), 'EX', 60)
  return json(200, { product })
}

main

Nas (@itsnas)

#redis sits in front of the primary as a shared cache every instance reads from: the same fix the connection ceiling needs, a resource with one real capacity, checked by whoever's asking, instead of N private copies of it.

On the Postgres side, a pooler like PgBouncer sits between the instances and the primary. Ten instances opening pools of 20 each doesn't mean 200 real connections reaching Postgres; it means whatever PgBouncer is configured to actually hold open.

App instance

App instance

App instance

Redis cache

PgBouncer

Postgres primary

1. App instance to Redis cache: cache-aside read
2. App instance to Redis cache
3. App instance to Redis cache
4. Redis cache to PgBouncer: cache miss
5. PgBouncer to Postgres primary: fixed real connections

100k req/s: the cache and the load balancer become the systems under test

The cache from the last tier absorbed the database problem. At 100k req/s, it becomes the next problem.

Little's law doesn't stop applying just because the resource changed: at roughly 1ms per cache read, 100k x 0.001s = 100 requests in flight against Redis at any instant. Redis handles that fine; holding open connections isn't what limits it.

What does limit a single Redis node is throughput, not concurrency: commands run one at a time on a single core. A simple GET on typical hardware tops out somewhere in the neighborhood of 100k-200k ops/sec, a ballpark that moves with payload size and pipelining, not a fixed number.

A single endpoint pushing 100k req/s at one cache node is already close enough to that ceiling that anything else sharing the same node (session lookups, other endpoints' caches) starts to compete with it directly.

A second failure mode shows up at this tier that didn't exist at 10K, because it needs a specific unlucky coincidence in timing that only gets likely at high volume:

The fix is request coalescing: the first request that misses the cache takes a short-lived lock and repopulates it. Every other request for the same key, arriving while that lock is held, waits on the same in-flight fetch instead of starting its own.

Cloudflare's write-up on lock-free probabilistic caching covers the same problem, and an alternative to a lock: revalidate a key probabilistically before it expires, so it rarely hits a hard, all-at-once expiry in the first place.

@itsnas — TypeScript

Hide line numbersCopy code

async function getProduct(req: Request) {
  const cacheKey = `product:${req.params.id}`
  const cached = await redis.get(cacheKey)
  if (cached) return json(200, { product: JSON.parse(cached) })
 
  const lockKey = `lock:${cacheKey}`
  const gotLock = await redis.set(lockKey, '1', 'NX', 'EX', 5)
 
  if (!gotLock) {
    // Someone else is already repopulating this key. Wait briefly and
    // re-check the cache instead of also querying Postgres.
    await sleep(50)
    return getProduct(req)
  }
 
  const [product] = await db`
    SELECT id, name, price, description
    FROM products
    WHERE id = ${req.params.id}
  `
  if (!product) return json(404, { error: 'Not found' })
 
  await redis.set(cacheKey, JSON.stringify(product), 'EX', 60)
  await redis.del(lockKey)
  return json(200, { product })
}

Show all 26 lines

main

Nas (@itsnas)

The load balancer sitting in front of all of this is also, quietly, no longer free capacity. Terminating TLS is CPU work per connection, and a single load-balancer node doing that for 100K new connections a second is a real bottleneck of its own, not just a pass-through.

Two things reduce it. HTTP keep-alive lets a client reuse one TLS connection for many requests instead of paying the handshake cost per request. And running the load balancer itself as more than one node is the exact same "don't let one thing hold all the capacity" pattern that already showed up for Postgres connections and cache throughput.

Tier	What actually breaks first	What fixes it
1k req/s	Nothing yet, that's the trap	Nothing, but don't mistake this for validation
10k req/s	Postgres connection ceiling; single instance's own throughput	Shared cache, a pooler, and per-instance state made shared
100k req/s	Cache node throughput; cache stampede on hot keys; LB TLS termination	Request coalescing, more cache/LB nodes, keep-alive
1M req/s	Correlated retries from clients you don't control	Circuit breakers, load shedding, cell-based isolation

1M req/s: the failure mode itself changes

Every fix so far has had the same shape: a resource had one real ceiling, so put that resource behind something shared, and add more of it. That shape stops working at 1M req/s, not because the numbers get bigger, but because the thing most likely to take the system down is no longer a resource ceiling at all.

It's the correlated behavior of clients reacting to a blip that would otherwise have been minor.

Backoff with jitter, covered in Retry with Backoff & Jitter, is necessary here but not sufficient on its own at this volume. Spreading a million retries over a few seconds instead of one instant still means a few seconds of a million clients hammering a service that told them, repeatedly, that it couldn't keep up. Two more pieces close the gap.

A circuit breaker stops a client from calling a dependency that has already shown it's failing, instead of retrying into it and adding to the exact load that's causing the failures:

@itsnas — TypeScript

Hide line numbersCopy code

type State = 'closed' | 'open' | 'half-open'
 
class CircuitBreaker {
  private state: State = 'closed'
  private failures = 0
  private openedAt = 0
 
  constructor(
    private readonly threshold = 5,
    private readonly resetAfterMs = 30_000,
  ) {}
 
  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === 'open') {
      if (Date.now() - this.openedAt < this.resetAfterMs) {
        throw new Error('Circuit open, not calling dependency')
      }
      this.state = 'half-open'
    }
 
    try {
      const result = await fn()
      this.failures = 0
      this.state = 'closed'
      return result
    } catch (err) {
      this.failures++
      if (this.failures >= this.threshold) {
        this.state = 'open'
        this.openedAt = Date.now()
      }
      throw err
    }
  }
}

Show all 35 lines

main

Nas (@itsnas)

This is the same pattern Microsoft's architecture guidance describes as the Circuit Breaker pattern: once enough calls fail, the breaker opens and every subsequent call fails immediately, locally, with no request ever reaching the struggling dependency.

That's what actually breaks the retry storm's feedback loop, not the jitter alone. Jitter smooths the timing; the breaker removes the calls.

The other half is deciding what to do with the requests the system genuinely cannot serve right now. Load shedding means rejecting excess work early and cheaply, before it consumes the resources that the requests you can serve need:

@itsnas — TypeScript

Hide line numbersCopy code

async function handleRequest(req: Request) {
  if (currentLoad() > MAX_LOAD) {
    return json(
      503,
      { error: 'At capacity, retry later' },
      { 'Retry-After': '5' },
    )
  }
 
  return getProduct(req)
}

main

Nas (@itsnas)

Amazon's own write-up on this, Using load shedding to avoid overload, makes the same point the 404 decision made in the multi-tenant pooling case. An honest, cheap failure that lets a well-behaved client back off is better than a slow failure that looked like it was trying to succeed.

Edge / load balancer

Cell 1: app + cache + DB shard

Cell 2: app + cache + DB shard

Cell 3: app + cache + DB shard

1. Edge / load balancer to Cell 1: app + cache + DB shard: shard key A-H
2. Edge / load balancer to Cell 2: app + cache + DB shard: shard key I-P
3. Edge / load balancer to Cell 3: app + cache + DB shard: shard key Q-Z

The last piece is containing blast radius rather than avoiding failure entirely, because at this volume something is always degraded somewhere. Splitting traffic into independent cells, each with its own instances, cache, and database shard, means a bad deploy or a hot key in one cell costs that cell's slice of traffic, not the whole million.

AWS describes the general technique as shuffle-sharding: assign each customer or shard key to a small, overlapping subset of cells. Even a fully broken cell then only ever intersects a small fraction of any given customer's traffic.

None of this replaces the fixes from the earlier tiers: a shared cache, a connection pooler, coalesced reads still all matter at 1M req/s. What changes is that they're no longer sufficient by themselves. The thing most likely to take the system down is not one resource running out, it's every client reacting to a small problem in exactly the same way at exactly the same time.

1. 1

   Reading a system's stability at 1K req/s as evidence of good design

   Low volume hides per-instance state, N+1 queries, and connection math that hasn't been tested. It proves the code runs, not that the design holds up.

2. 2

   Sizing a connection pool or cache without accounting for instance count

   A pool or an in-memory cache that looks fine on one instance becomes (limit x instance count) real connections, or (hit rate / instance count) effective cache coverage, the moment there's more than one instance running.

3. 3

   Treating a cache miss storm as a database problem

   A cache stampede looks like the database suddenly can't handle load it handled fine yesterday. The fix is coalescing requests behind the cache, not scaling the database to survive a burst the cache should have absorbed.

4. 4

   Assuming backoff and jitter alone survive a downstream blip at high volume

   Jitter spreads retries out in time, it doesn't reduce how many clients are retrying. At high enough volume, that's still enough correlated load to keep a recovering dependency down. Pair it with a circuit breaker that stops the calls outright.

5. 5

   Letting an overloaded system accept work it can't finish

   A slow failure under load still spends the CPU, the connection, and the client's patience, then fails anyway. Reject early with a cheap, honest 503 and a Retry-After instead.

The pattern across all four tiers is the same one, repeated at different resources: something has exactly one real capacity, more than one caller assumes it has more than that, and the gap only shows up once traffic is high enough to hit it.

At 1k req/s that gap is invisible. At 1M req/s, the gap includes the other million clients' reaction to the last failure. That's why the fix stops being "add more capacity" and starts being "contain what one failure can reach."

---