Keep your API fast—even at peak

Requests can arrive from many people at once. Accepting those requests and producing useful reports are different events; more accepted requests do not themselves make reports finish faster.

Capacity describes available finishing ability per unit of time. The request mix, available resources, and other conditions matter; it is not simply the number of requests the system can accept.

Compare the two definitions. Capacity is the available ability to finish work; throughput counts actual completions. A system can finish less than its capacity, for example when less work arrives.

The following arithmetic assumes equal-cost reports and constant conditions. It starts with no unfinished work and retains everything that has not finished; real mixed-cost traffic need not behave this simply.

This is offered demand, not completed reports. We need the finishing rate as well before we can work out whether unfinished work accumulates.

Subtract completions from arrivals: 100 minus 80 leaves 20 extra unfinished requests each second. The finishing rate remains 80/s even though more work was accepted.

Starting empty, 100 arrivals minus 80 completions leaves 20 unfinished requests. One tile represents those twenty requests; it is a count of work, not a memory allocation or a new processing slot.

The same excess of 20/s continues for ten seconds: 200 requests remain unfinished. No request was refused or canceled in this calculation. The worker has still been finishing 80/s throughout.

Unfinished does not mean only requests sitting in a queue. It also includes operations already running whose required result has not completed. Both categories count as work in flight.

Choose a start and finish for the operation being measured. Time spent waiting within that boundary contributes to latency alongside time spent running; the unfinished count is not itself a duration.

The worker still finishes eighty reports each second in these conditions. Accepting more demand increases the unfinished work unless something else changes; acceptance alone is not extra capacity.

Read the rate chart first, then the unfinished-work chart. In this illustration, 100/s arrive for five seconds, then 60/s; 80/s keep finishing. Work grows to 100, then drains to zero: less must arrive than finishes before the old work can clear.

Same time window, same scales, same 80/s finishing rate. Dashed A falls to 60/s and clears. Solid B stays at 100/s: 20 extra requests each second leave 200 unfinished after ten seconds. Under unchanged conditions, waiting longer makes this case worse.