Latency Numbers Every Engineer Should Know

UpdatedAugust 1, 2026

•5 min read•View as Markdown

Latency Numbers Every Engineer Should Know

Rohan Tiwari

Part of seriesSystem Design From First Principles

Comments (3)

Join the discussion

Duko tools4h ago

The "scale everything up so L1 is 1 human second" trick is the one that actually makes this stick long-term, more than the raw nanosecond numbers ever do. Nanoseconds don't have intuition behind them, but "waiting years for a letter to come back" is something you can genuinely feel, which is probably why the metaphor survives in memory long after the exact numbers fade.

The tail-latency section is the sharper point for anyone past the junior-engineer stage, though. The math on 8 dependent calls each with their own p99 compounding to a ~7.7% chance of hitting at least one slow link is the kind of thing that's easy to nod along to in the abstract but genuinely surprising the first time you actually compute it against your own service's dependency chain. "Every individual service looks healthy on its own dashboard" while the aggregate p99 balloons is exactly the failure mode that makes tail latency so hard to debug organizationally, no single team's graphs show a problem, so the instinct is to assume there isn't one.

Worth citing Dean and Barroso directly too, since "The Tail at Scale" gets referenced constantly but read in full far less, the framing that tail latency, not average, becomes the dominant constraint at fleet scale is the actual thesis, not just a footnote on percentiles.

Puneet Khandelwal1d ago

LLove this perspective on prioritizing well-being

Amit Joshi1d ago

very informative, thanks for sharing.

System Design From First Principles

Part 1 of 1

More from this blog

Understanding HTTP MIME Types

The internet hosts many kinds of resources, so each resource needs a label to tell clients how to handle it. MIME (Multipurpose Internet Mail Extensions) was originally developed to solve problems mov

Mar 22, 20262 min read4

Centralized Cache Key Management In Redis

In modern web applications, efficient data access is essential for performance and user experience. Redis, a blazing-fast in-memory store used for caching, messaging, and short-lived persistence, depe

Mar 20, 20266 min read5

Postgres Multi Version Concurrency Control - MVCC

In 1986 database researcher named Michael Stonebraker was working on a problem that has plauged databases since their inception. how do you let many people read and write to the data simultaneously wi

Feb 19, 20266 min read41

Postgres Multi Version Concurrency Control - MVCC

PostgreSQL TOAST Storage Models

The main problem TOAST solves is fundamental : postgres pages are of 8kb, and tuple must fit within that page. so what happens if you try to insert 1MB of text field ? Without toast you would get an error: "Row too big". With toast PostgreSQL handles...

Feb 19, 20264 min read12

Tyranny Blogs

9 posts

Latency is simply the time between asking for something and getting it back. Every system you will ever touch is built out of layers, and each layer has its own latency because each layer is physically farther away, or requires more steps, than the one before it. Think of it like gathering information for a research paper:

Look at the sticky note on your monitor (fastest, it is right in front of you)
Check your desk drawer (a little farther)
Walk to the shared filing cabinet down the hall (slower still)
Drive to the library across town (much slower)
Mail a request to a library in another country and wait for a reply (slowest by far)

Computers have the exact same hierarchy. The sticky note is the CPU's L1 cache. The desk drawer is L2 cache. The filing cabinet down the hall is main memory (RAM). The library across town is an SSD or disk. The library in another country is a network call to another server. Nobody designed it to be this way out of preference,it is forced by physics: the closer something is physically, and the simpler the retrieval mechanism, the faster you can get it.

Now let's put real numbers on it

A little farther' is not good enough for an engineer:

L1 cache: about 1 nanosecond. Tiny, extremely fast memory built directly into the CPU core.
L2 cache: about 4 nanoseconds. Still on-chip, but farther from the core, sometimes shared with neighboring cores.
Main memory (RAM): about 100 nanoseconds. This means going through a whole different chip (the memory controller) and often physically activating a row of memory cells before reading them.
NVMe SSD random read: about 50,000-100,000 nanoseconds (50-100 microseconds). Now you are waiting on an entirely separate piece of hardware plus a small protocol negotiation.
Spinning hard disk: 2,000,000-10,000,000 nanoseconds (2-10 milliseconds). A physical arm has to move to the right spot on a spinning platter, genuinely mechanical, like a record-player needle moving.
Same-datacenter network round trip: about 500,000 nanoseconds (0.5ms).
Cross-country network round trip: 30-80 milliseconds.
Cross-ocean network round trip: 130-280 milliseconds.

Here is the trick every senior engineer uses to make this stick: scale everything up so L1 cache access takes 1 human second. On that scale:

L2 cache: about 4 seconds
Main memory: about 1.7 minutes
SSD read: about 14-28 hours
Same-datacenter round trip: about 6 days
Cross-country round trip: about 8-9 months
Cross-ocean round trip: about 3-4.5 years

Sit with that. If touching CPU cache is like glancing at a sticky note, one network call across an ocean is like waiting years for a letter to come back. This is not a metaphor for flavor, it is the actual ratio. It is why 'just add a network call, it's fine' is one of the most dangerous sentences in software engineering.

Why this is physics, not opinion

Light in fiber-optic cable travels at roughly two-thirds its vacuum speed (the glass slows it down). New York to London is about 5,585 km. Even with a perfect, empty, uncongested connection, the theoretical minimum round trip is about 56 milliseconds, purely from distance. You cannot engineer around that. You can only choose how many times, in your architecture, you pay that tax. This constraint is exactly why content delivery networks like Cloudflare exist: to move data physically closer to users so you pay the light-speed tax fewer times, over shorter distances.

The math, worked through

Suppose a request touches 1,000 L1 hits, 100 L2 hits, 10 main-memory hits, and one same-datacenter network call (0.5ms): 1,000 x 1ns + 100 x 4ns + 10 x 100ns + 1 x 500,000ns = about 502,400ns, or 0.5ms. The single network call is 99.7% of the total. You could make your CPU-side code infinitely fast and barely move the needle. This is the most common mistake junior engineers make: spending a week shaving 2 microseconds off a loop while an unnecessary network round trip sitting three lines away costs half a millisecond.

How this changes as you scale up

At small scale (an internal tool doing 10,000 requests a day), an accidental cross-region hop costing an extra 60ms is basically invisible. At medium scale (1,000,000 requests a day, roughly 12 requests/sec average but bursting much higher), if a service makes 5 sequential network calls per request, p50 creeps toward 200-300ms and users start to feel it. At huge scale (1,000,000,000 requests a day, over 11,000 requests/sec sustained), a subtlety appears that catches almost everyone off guard: if you have 8 dependent calls in a row, each with its own p99, the chance at least one of them lands in its own bad 1% goes up fast, roughly 1-(0.99)^8, about 7.7%. Around 1 in 13 requests hits at least one slow call somewhere in the chain, and your overall p99 balloons even though every individual service looks healthy on its own dashboard. This is exactly the phenomenon Google researchers Jeffrey Dean and Luiz Andre Barroso described in 'The Tail at Scale' (Communications of the ACM, 2013): at large fleet scale, tail latency, not average latency, becomes the dominant design constraint.

Command Palette