Instrument before you optimize
Most performance work we are called in for starts with a team that already knows what the problem is. They are wrong about half the time, and the fix costs ten times more than the measurement would have.
Every engagement that starts with "we think it's the database" is a coin flip. Sometimes it is. Often it is a serialization step nobody thought to look at, or a health check hammering an endpoint that was never meant to be called every second.
The instinct to skip measurement is understandable. Instrumentation feels like overhead - it does not fix anything, and there is a deadline. But the alternative is spending weeks optimizing something that was never on the critical path.
What measuring first actually costs
Two weeks, usually. Tracing on the hot paths, a dashboard that shows where wall-clock time goes, and enough load in a realistic shape to make the graph mean something.
That is genuinely expensive when a system is on fire. It is trivially cheap compared to a rewrite that does not help.
What it buys
- An argument you can settle with data instead of seniority
- A baseline, so you can prove the fix worked rather than asserting it
- Regression detection, so the same problem does not come back next quarter
- A much shorter list of things to actually change
The last one is the real return. On one recent engagement, tracing turned a proposed six-month re-architecture into two endpoint fixes and three indexes. Nine weeks total, and the product never changed.
The uncomfortable part
Sometimes the measurement says the thing you already shipped was the problem. That is the point. A team that cannot be wrong in public will keep optimizing whatever is politically safe to optimize, which is rarely what is slow.
Measure first is not caution. It is the cheapest way to avoid being confidently wrong for a quarter.