What Breaks From 1k to 1M Requests Per Second

"Scale" gets talked about as one bigger number on a graph. It isn't. Traffic finds whichever layer has the smallest ceiling, and each order of magnitude hits a different one. Follow one endpoint, GET /products/:id, from 1k to 1M requests per second and ask the same question at each step: with the same code, what gives first?

The one formula

Little's law: requests in flight = arrival rate × time per request. An 8ms query at 1k req/s means 8 queries in flight at any instant. At 10k, it's 80.

@itsnas database-connections-needed-as-traffic-grows.svg

Nothing about the code changes. Traffic alone walks the line into the ceiling, for one query on one endpoint.

That's why a system can look completely fine and then fall over at one specific number. Here's the map; the rest of the post walks it.

Tier	What actually breaks first	What fixes it
1k req/s	Nothing yet, that's the trap	Nothing, but don't mistake this for validation
10k req/s	Postgres connection ceiling; single instance's own throughput	Shared cache, a pooler, and per-instance state made shared
100k req/s	Cache node throughput; cache stampede on hot keys; LB TLS termination	Request coalescing, more cache/LB nodes, keep-alive
1M req/s	Correlated retries from clients you don't control	Circuit breakers, load shedding, cell-based isolation

1k: everything works, which is the trap

@itsnas TypeScript

// GET /products/:id
async function getProduct(req: Request) {
  const [product] = await db`
    SELECT id, name, price, description
    FROM products
    WHERE id = ${req.params.id}
  `
  if (!product) return json(404, { error: 'Not found' })
 
  return json(200, { product })
}

main

Nas (@itsnas)

No cache, no pool tuning, no cross-instance concerns yet

Eight queries in flight against a pool of 20 and a #postgres limit of 100. Nothing notices. Neither would an N+1 query, a rate limiter counting in a plain Map, or a cache in module memory. At 1k req/s on one instance, all of it works. That isn't validation; it's cover.

10k: the database and the single instance

Eighty connections for one query on one endpoint, and the service has twelve endpoints, each with its own pool of 20 sized in isolation. Under real traffic they're busy together: 240 possible connections against 100. Requests queue for a connection, and every endpoint slows at once for a reason none of them caused.

The other wall: one Node event loop can't push 10k real requests a second. So you run ten instances behind a load balancer, and anything that assumed one process quietly breaks. A per-instance rate limit is now ten times too generous; a per-instance cache has a tenth of the hit rate. Same fix for both problems: stop treating per-instance state as global, and stop asking Postgres for more connections than it has.

@itsnas TypeScript

async function getProduct(req: Request) {
  const cacheKey = `product:${req.params.id}`
  const cached = await redis.get(cacheKey)
  if (cached) return json(200, { product: JSON.parse(cached) })
 
  const [product] = await db`
    SELECT id, name, price, description
    FROM products
    WHERE id = ${req.params.id}
  `
  if (!product) return json(404, { error: 'Not found' })
 
  await redis.set(cacheKey, JSON.stringify(product), 'EX', 60)
  return json(200, { product })
}

main

Nas (@itsnas)

A shared cache in front of the query, checked by every instance

Ten app instances no longer talk to Postgres directly. PgBouncer caps the real connections; Redis absorbs most reads before they reach Postgres at all.

#redis is one shared cache for every instance; PgBouncer caps the real connections no matter how many instances open pools. (The full pool-sizing story is in Designing Multi-Tenant APIs That Scale.)

100k: the cache and the load balancer

The fix from the last tier is now the system under test. Redis runs commands one at a time on one core, and a plain GET tops out around 100k to 200k ops a second. One hot endpoint is already near that, competing with everything else on the node.

Volume also makes a rare coincidence routine. A popular product's cache entry expires, and in the next few milliseconds thousands of requests for it all miss and all query Postgres for the same row: a cache stampede. The database didn't get slower; a TTL expired at a busy moment. The fix is coalescing: the first miss takes a short lock and refills the key, everyone else waits for that one fetch.

@itsnas TypeScript

async function getProduct(req: Request) {
  const cacheKey = `product:${req.params.id}`
  const cached = await redis.get(cacheKey)
  if (cached) return json(200, { product: JSON.parse(cached) })
 
  const lockKey = `lock:${cacheKey}`
  const gotLock = await redis.set(lockKey, '1', 'NX', 'EX', 5)
 
  if (!gotLock) {
    // Someone else is already repopulating this key. Wait briefly and
    // re-check the cache instead of also querying Postgres.
    await sleep(50)
    return getProduct(req)
  }
 
  const [product] = await db`
    SELECT id, name, price, description
    FROM products
    WHERE id = ${req.params.id}
  `
  if (!product) return json(404, { error: 'Not found' })
 
  await redis.set(cacheKey, JSON.stringify(product), 'EX', 60)
  await redis.del(lockKey)
  return json(200, { product })
}

main

Nas (@itsnas)

One fetch per expired key, not one per waiting request

(Cloudflare's probabilistic early revalidation avoids the hard expiry altogether.) The load balancer stops being free too: terminating TLS is CPU per connection. Keep-alive amortises the handshake, and more than one balancer node removes the single ceiling, the same move as before.

1M: the failure mode changes

Every fix so far had one shape: a resource has one real ceiling, so share it and add more of it. At 1M req/s the most likely outage isn't a ceiling. It's a million clients all reacting to the same small blip in the same way.

The loop that keeps a dependency down after the original spike has gone. Breaking any arrow breaks the loop.

Backoff with jitter spreads the retries out, but it doesn't reduce how many there are: a million clients hammering a struggling service over five seconds instead of one. Two more pieces break the loop. A circuit breaker removes the calls: after enough failures, every call fails locally and instantly, and nothing reaches the dependency.

@itsnas TypeScript

type State = 'closed' | 'open' | 'half-open'
 
class CircuitBreaker {
  private state: State = 'closed'
  private failures = 0
  private openedAt = 0
 
  constructor(
    private readonly threshold = 5,
    private readonly resetAfterMs = 30_000,
  ) {}
 
  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === 'open') {
      if (Date.now() - this.openedAt < this.resetAfterMs) {
        throw new Error('Circuit open, not calling dependency')
      }
      this.state = 'half-open'
    }
 
    try {
      const result = await fn()
      this.failures = 0
      this.state = 'closed'
      return result
    } catch (err) {
      this.failures++
      if (this.failures >= this.threshold) {
        this.state = 'open'
        this.openedAt = Date.now()
      }
      throw err
    }
  }
}

main

Nas (@itsnas)

Stop calling a dependency that's already failing

Load shedding handles what you can't serve: reject it early and cheaply, before it spends the capacity the requests you can serve need. An honest 503 with Retry-After beats a slow failure that looked like it was trying.

@itsnas TypeScript

async function handleRequest(req: Request) {
  if (currentLoad() > MAX_LOAD) {
    return json(
      503,
      { error: 'At capacity, retry later' },
      { 'Retry-After': '5' },
    )
  }
 
  return getProduct(req)
}

main

Nas (@itsnas)

Reject early and cheaply, before spending real capacity

And at this volume something is always broken somewhere, so the last move is containing it. Split traffic into independent cells, each with its own instances, cache and database shard:

Traffic split into isolated cells by a shard key. A cell that degrades takes its own slice of traffic down, not the other two.

A bad deploy or a hot key then costs one cell's slice, not the whole million. Shuffle-sharding takes it further by giving each customer an overlapping subset of cells, so even a dead cell touches a small fraction of anyone's traffic.

Same story, different resource

Every tier is the same story with a different resource: something has exactly one real capacity, more than one caller assumes it has more, and the gap only shows once traffic is high enough. At 1k the gap is invisible. At 1M it includes everyone else's reaction to the last failure, which is why the fix stops being "add capacity" and becomes "limit what one failure can reach".

If this is happening in production right now, the redesign can wait: start with the first 15 minutes of a prod incident.

Pairs with

Designing Multi-Tenant APIs That Scale Retry with Backoff & Jitter First 15 Minutes of a Prod Incident

What Breaks From 1k to 1M Requests Per Second

The one formula

1k: everything works, which is the trap

10k: the database and the single instance

100k: the cache and the load balancer

1M: the failure mode changes

Same story, different resource

When the Settlement Report Disagrees With Your Ledger

Why Does Google Obsess Over Its Slowest 1% of Requests?

You Don't Need to Store 4 Billion Users to Count Them

How Does Facebook's Distributed Like Counter Work?