The real cost of a slow API
Latency is usually filed as an engineering concern. It is a revenue concern that happens to be measured in milliseconds.
"It is a bit slow" is one of the least useful phrases in software. It gets a ticket, it sits in the backlog, and it stays there because it never competes with a feature request. Reframing it usually gets it fixed.
Latency compounds through the stack
An endpoint that takes 400ms instead of 80ms does not just cost 320ms. It:
- Holds a connection open five times longer, so you need more of them
- Occupies a worker that cannot serve anyone else
- Pushes retries and timeouts closer to firing upstream
- Multiplies, if the page makes several calls
The infrastructure bill is partly a latency bill. Right-sizing a fleet after fixing the two slowest endpoints is a very common way to cut cloud spend meaningfully.
P95 is the number, not the average
Averages hide the problem. If the mean is 120ms and the P95 is 3 seconds, one request in twenty is a bad experience - and those are disproportionately your heaviest, most engaged users, because they have the most data.
Alert on P95 and P99. The average will look fine right up until churn tells you it was not.
The cheapest wins are boring
In order of how often they turn out to be the answer:
- A missing index
- N+1 queries introduced by an ORM upgrade
- Serialising something enormous that the client discards
- Doing work in the request that belongs in a background job
- No caching on something that changes twice a day
None are architectural. All are findable in an afternoon with tracing in place - which is the argument for having it before you need it.