NorthReducing latency in distributed systems
Latency is best treated as a budget. A user-facing request has some acceptable ceiling, and every network hop, lookup and serialization step spends against it. Teams get into trouble when they spend that budget by accident and then try to claw it back with heroics. It is cheaper to spend deliberately from the start.
Measure the distribution, not the average
Averages hide the requests that hurt. A service can have a healthy mean and still page you at the 99th percentile, where retries, cold caches and contention live. Track percentiles, and make the tail latency a first-class number on your dashboards.
Count the round trips
Most surprising latency is not slow code — it is one more network round trip than you thought. A request that fans out to five services sequentially pays five times the round-trip cost. Look for opportunities to parallelise independent calls, batch chatty ones, and move data closer to where it is needed.
Cache with intent
A cache is a promise about staleness. Decide explicitly how stale each piece of data may be, and let that decision drive the cache, rather than sprinkling caches until the graphs look better. An unbounded cache with no eviction policy is a future incident, not a performance win.
Fail fast, retry carefully
- Set timeouts that reflect the latency budget, not arbitrary round numbers.
- Use retries with backoff and jitter, and cap them — uncapped retries turn a blip into an outage.
- Shed load deliberately under pressure instead of letting queues grow without bound.
Spend the budget where users feel it
The goal is not the lowest possible latency everywhere — that is expensive and often invisible. It is to spend the budget where users actually notice, and to know, from data, exactly where it is going. A system whose latency is understood is one you can improve on purpose.