How many servers do you actually need?

The throughput question: cores are spent, not guessed

Every request consumes some CPU: serialising, computing, encrypting, waiting on locks. Add up that consumption at your request rate and you have the number of cores your traffic keeps busy — requests per second multiplied by CPU seconds per request. A service taking 500 requests a second at 25 milliseconds of CPU each is burning twelve and a half cores continuously, whether it runs on two servers or twenty. This arithmetic is unforgiving, which is its virtue: the core count is a measurement problem, not an opinion. The measurement that matters is CPU service time, and the commonest planning failure is inventing it. Vendor pages quote requests per second for a hello-world handler; your handler renders, validates, logs and calls two databases. Measure your own: total CPU seconds consumed over your busiest hour, divided by requests served in that hour. That single honest number does more for a capacity plan than any benchmark ever published. The capacity estimator runs exactly this arithmetic — cores from rate and service time, whole servers from cores, redundancy applied after — and shows a growth table because the follow-up question is always "and at ten times the traffic?"

The headroom decision: why 70% is a policy, not a fudge

Nobody sane provisions exactly the cores the formula returns, because queues behave badly near saturation. A processor 95% busy on average has no slack for the minute traffic doubles, a deploy restarts half the fleet, or a dependency slows and retries multiply the load — and queueing delay grows towards infinity as utilisation approaches 100%, long before anything is technically "full". So the plan divides by a target utilisation: at 70%, each provisioned core counts as 0.7 of a core, and the other 0.3 is insurance bought deliberately. The right target is a risk decision. Latency-sensitive, spiky, user-facing traffic deserves 50–70%. Steady internal workers that tolerate delay can run at 85% and save real money. What is not defensible is 100% — a plan with no headroom is a plan for the average minute, and services die in the exceptional ones. Notice what this does to arguments: "we need twelve servers" becomes "we need nine cores of work covered at a 70% ceiling", a claim a colleague can check, challenge and re-derive — which is all a capacity plan ever needed to be.

The concurrency question: Little's law, the limit you hit first

Cores are only half the answer, and usually not the half that pages you. Little's law states that the number of requests in flight equals the request rate times the average response time: 500 requests a second at 180 milliseconds means 90 requests inside your system at every instant. Those 90 are not burning CPU most of the time — they are waiting: on databases, on downstream APIs, on locks. But each one holds a thread or an async slot, a connection from a finite pool, buffers, memory. Let the response time degrade to two seconds under load and the same traffic now holds a thousand requests in flight; pools exhaust, queues form, timeouts cascade, and the service falls over with its CPUs at 30% — the classic outage where every dashboard says the machines are fine. Size concurrency separately: pools and thread counts from Little's law at your p95-ish response time, not the median, and treat downstream latency as a capacity input, because a slow dependency is indistinguishable from a traffic surge. The estimator prints in-flight requests beside the core count for exactly this reason: the two numbers protect against different failures, and you need both.

Why benchmarks lie, and what to do instead

The load test said one server handled 2,000 requests a second, so why does the arithmetic demand four? Because the test measured a different service: warm caches, one request shape, payloads from a generous client on a quiet network, no TLS handshakes from real browsers, no retries, no deploys, no noisy neighbours, no long tail. Benchmarks measure ceilings under laboratory conditions; capacity planning needs floors under production ones. The reconciliation is not to distrust measurement but to measure the right thing: your service time, your traffic shape, your busiest hour. Then round like an operator. Servers come whole, so round up. Redundancy comes after the rounding — N+1 means the fleet that meets the plan plus one spare, so a single failure leaves the plan intact rather than eating the headroom. And revisit the numbers when the service changes: a feature that adds 10ms of CPU per request quietly raised your fleet requirement by 40%, and no alert will ever fire to tell you. Capacity is not a purchase; it is a number you keep re-deriving as the inputs move.

Run your own figures through the capacity estimator: your measured rate, your measured CPU per request, a utilisation ceiling you can defend in the post-incident review. If the answer surprises you, the surprise is information — usually that the service time was a guess. Fix the input, not the formula.

Frequently asked questions