# Latency Numbers Every Engineer Should Know

Latency is simply the time between asking for something and getting it back. Every system you will ever touch is built out of layers, and each layer has its own latency because each layer is physically farther away, or requires more steps, than the one before it. Think of it like gathering information for a research paper:

*   Look at the sticky note on your monitor (fastest, it is right in front of you)
    
*   Check your desk drawer (a little farther)
    
*   Walk to the shared filing cabinet down the hall (slower still)
    
*   Drive to the library across town (much slower)
    
*   Mail a request to a library in another country and wait for a reply (slowest by far)
    

Computers have the exact same hierarchy. The sticky note is the CPU's L1 cache. The desk drawer is L2 cache. The filing cabinet down the hall is main memory (RAM). The library across town is an SSD or disk. The library in another country is a network call to another server. Nobody designed it to be this way out of preference,it is forced by physics: the closer something is physically, and the simpler the retrieval mechanism, the faster you can get it.

## Now let's put real numbers on it

A little farther' is not good enough for an engineer:

*   L1 cache: about 1 nanosecond. Tiny, extremely fast memory built directly into the CPU core.
    
*   L2 cache: about 4 nanoseconds. Still on-chip, but farther from the core, sometimes shared with neighboring cores.
    
*   Main memory (RAM): about 100 nanoseconds. This means going through a whole different chip (the memory controller) and often physically activating a row of memory cells before reading them.
    
*   NVMe SSD random read: about 50,000-100,000 nanoseconds (50-100 microseconds). Now you are waiting on an entirely separate piece of hardware plus a small protocol negotiation.
    
*   Spinning hard disk: 2,000,000-10,000,000 nanoseconds (2-10 milliseconds). A physical arm has to move to the right spot on a spinning platter, genuinely mechanical, like a record-player needle moving.
    
*   Same-datacenter network round trip: about 500,000 nanoseconds (0.5ms).
    
*   Cross-country network round trip: 30-80 milliseconds.
    
*   Cross-ocean network round trip: 130-280 milliseconds.
    

Here is the trick every senior engineer uses to make this stick: scale everything up so L1 cache access takes 1 human second. On that scale:

*   L2 cache: about 4 seconds
    
*   Main memory: about 1.7 minutes
    
*   SSD read: about 14-28 hours
    
*   Same-datacenter round trip: about 6 days
    
*   Cross-country round trip: about 8-9 months
    
*   Cross-ocean round trip: about 3-4.5 years
    

Sit with that. If touching CPU cache is like glancing at a sticky note, one network call across an ocean is like waiting years for a letter to come back. This is not a metaphor for flavor, it is the actual ratio. It is why 'just add a network call, it's fine' is one of the most dangerous sentences in software engineering.

## Why this is physics, not opinion

Light in fiber-optic cable travels at roughly two-thirds its vacuum speed (the glass slows it down). New York to London is about 5,585 km. Even with a perfect, empty, uncongested connection, the theoretical minimum round trip is about 56 milliseconds, purely from distance. You cannot engineer around that. You can only choose how many times, in your architecture, you pay that tax. This constraint is exactly why content delivery networks like Cloudflare exist: to move data physically closer to users so you pay the light-speed tax fewer times, over shorter distances.

## The math, worked through

Suppose a request touches 1,000 L1 hits, 100 L2 hits, 10 main-memory hits, and one same-datacenter network call (0.5ms): 1,000 x 1ns + 100 x 4ns + 10 x 100ns + 1 x 500,000ns = about 502,400ns, or 0.5ms. The single network call is 99.7% of the total. You could make your CPU-side code infinitely fast and barely move the needle. This is the most common mistake junior engineers make: spending a week shaving 2 microseconds off a loop while an unnecessary network round trip sitting three lines away costs half a millisecond.

## How this changes as you scale up

At small scale (an internal tool doing 10,000 requests a day), an accidental cross-region hop costing an extra 60ms is basically invisible. At medium scale (1,000,000 requests a day, roughly 12 requests/sec average but bursting much higher), if a service makes 5 sequential network calls per request, p50 creeps toward 200-300ms and users start to feel it. At huge scale (1,000,000,000 requests a day, over 11,000 requests/sec sustained), a subtlety appears that catches almost everyone off guard: if you have 8 dependent calls in a row, each with its own p99, the chance at least one of them lands in its own bad 1% goes up fast, roughly 1-(0.99)^8, about 7.7%. Around 1 in 13 requests hits at least one slow call somewhere in the chain, and your overall p99 balloons even though every individual service looks healthy on its own dashboard. This is exactly the phenomenon Google researchers Jeffrey Dean and Luiz Andre Barroso described in 'The Tail at Scale' (Communications of the ACM, 2013): at large fleet scale, tail latency, not average latency, becomes the dominant design constraint.