# Keep useful work moving

Content revision: 07c7dd00db17

Public course review copy. Cite the lesson number, revision, and quoted passage in feedback. Illustrative examples are not benchmarks.

## 1 · A useful result starts with one request

[Open this lesson](https://ratelimitly.com/?lesson=request#architecture-graphic) · 1:06 at 1×

Maya needs her sales report before a meeting. Keeping useful results available is the goal of this explanation. You do not need cloud or networking experience to follow it.

When Maya clicks Open report, her browser sends a request: a message asking a computer to do something. The application is the program that receives that message and decides what work is needed. Here, it asks a database, a system that stores and retrieves the sales records.

The database returns the records. The application turns them into a report and sends a response back to Maya. A response is the answer to the request. Maya has succeeded only when she receives a usable report, not when the application merely accepts her message.

What if the application accepts the request but the report never arrives? The computer started something, but Maya still cannot do her job.

### Diagram walkthrough

- **Follow one request, end to end.** Overview of the normal path. Maya uses the browser; the application builds the report; the database supplies records. A started request is not yet a delivered result.

- **Maya has asked. She is still waiting.** The request carries an instruction to the application. Accepting it starts work; it does not mean the report exists or has reached the browser.

- **The application needs records first.** This is a second call inside the original request. The application is a caller of the database while it is still serving Maya.

- **Records have reached the application.** The records have reached the application, not yet Maya. This internal completion is only one step of her request.

- **The application builds the result.** The application still has work to do after the query finishes. Database completion and customer completion are different events.

- **The response travels back to Maya.** The answer must reach the browser and be usable. Counting accepted requests alone misses this last part of the customer’s experience.

- **Success belongs at the customer boundary.** The useful outcome is Maya receiving her report. A query finishing internally is not sufficient evidence of that outcome.

- **No usable report. Maya cannot finish.** Alternative outcome: acceptance happened, but the response never arrived. This does not establish where the path failed or whether database work completed.

### Apply it

Identify the real customer result before choosing counters or failure indicators.

**Boundary:** This example reads a report; a failed response would not by itself establish whether a write committed.

**Technical depth:** Measure successful, useful responses separately from accepted requests and internal completions.

**Check:** The app accepted Maya’s request. Has she succeeded?

**Explained answer:** Only if a usable report reaches her. Acceptance alone is not a completed customer outcome.


## 2 · Unfinished work grows when arrivals outrun completions

[Open this lesson](https://ratelimitly.com/?lesson=capacity#architecture-graphic) · 1:18 at 1×

Now many people ask for reports. Capacity means how much work the system can finish in a given time under the conditions we are considering. Throughput is how much actually finishes.

For a simple worked example, assume every report costs the same amount of work. One hundred requests arrive each second, but only eighty finish. After one second, twenty remain unfinished. After ten seconds, two hundred remain, assuming none were refused or canceled.

This unfinished work is called work in flight. It includes work waiting to start and work already running. Latency is how long a particular operation takes from its chosen start to its finish. More waiting can make that time longer.

Would accepting another hundred requests make the eighty-per-second worker faster? No. A short burst may clear when arrivals fall. If arrivals stay above completions, unfinished work keeps growing until capacity, demand, or the work we accept changes.

### Diagram walkthrough

- **Many people depend on the same finite finishing capacity.** Requests can arrive from many people at once. Accepting those requests and producing useful reports are different events; more accepted requests do not themselves make reports finish faster.

- **Capacity is what the system can finish under these conditions.** Capacity describes available finishing ability per unit of time. The request mix, available resources, and other conditions matter; it is not simply the number of requests the system can accept.

- **Throughput is what actually finishes.** Compare the two definitions. Capacity is the available ability to finish work; throughput counts actual completions. A system can finish less than its capacity, for example when less work arrives.

- **Keep the cost of each report equal for this example.** The following arithmetic assumes equal-cost reports and constant conditions. It starts with no unfinished work and retains everything that has not finished; real mixed-cost traffic need not behave this simply.

- **Count arrivals first: 100 requests every second.** This is offered demand, not completed reports. We need the finishing rate as well before we can work out whether unfinished work accumulates.

- **100 arrive while only 80 finish each second.** Subtract completions from arrivals: 100 minus 80 leaves 20 extra unfinished requests each second. The finishing rate remains 80/s even though more work was accepted.

- **After one second, one tile of work remains unfinished.** Starting empty, 100 arrivals minus 80 completions leaves 20 unfinished requests. One tile represents those twenty requests; it is a count of work, not a memory allocation or a new processing slot.

- **After ten seconds, ten of the same tiles remain.** The same excess of 20/s continues for ten seconds: 200 requests remain unfinished. No request was refused or canceled in this calculation. The worker has still been finishing 80/s throughout.

- **Work in flight includes both waiting and running work.** Unfinished does not mean only requests sitting in a queue. It also includes operations already running whose required result has not completed. Both categories count as work in flight.

- **Latency measures one operation across a chosen boundary.** Choose a start and finish for the operation being measured. Time spent waiting within that boundary contributes to latency alongside time spent running; the unfinished count is not itself a duration.

- **Accepting more work does not increase the finishing rate.** The worker still finishes eighty reports each second in these conditions. Accepting more demand increases the unfinished work unless something else changes; acceptance alone is not extra capacity.

- **A temporary burst can clear when arrivals fall enough.** Read the rate chart first, then the unfinished-work chart. In this illustration, 100/s arrive for five seconds, then 60/s; 80/s keep finishing. Work grows to 100, then drains to zero: less must arrive than finishes before the old work can clear.

- **Sustained excess keeps adding unfinished work.** Same time window, same scales, same 80/s finishing rate. Dashed A falls to 60/s and clears. Solid B stays at 100/s: 20 extra requests each second leave 200 unfinished after ten seconds. Under unchanged conditions, waiting longer makes this case worse.

### Apply it

Measure arrivals, useful completions and unfinished work separately.

**Boundary:** These illustrative rates assume equal-cost work and constant capacity; they are not product benchmarks.

**Technical depth:** Request rate alone is a weak capacity proxy when work mix, resource costs or available capacity change.

**Check:** How many extra requests remain after ten seconds?

**Explained answer:** 200, under the equal-cost, constant-capacity assumptions with no refusal or cancellation.


## 3 · More waiting space does not make work finish faster

[Open this lesson](https://ratelimitly.com/?lesson=queues#architecture-graphic) · 2:00 at 1×

A queue holds requests waiting for their turn. The worker, not the queue, performs the work. A buffer provides waiting space. In our example, one hundred requests arrive each second and eighty finish. Twenty extra requests must wait each second.

Start with one hundred empty waiting slots. After five seconds, all one hundred slots are occupied. With two hundred slots, the same excess fills the buffer in ten seconds. The worker still finishes eighty per second. We bought five more seconds, not a faster worker. Once full, refuse work, slow its supplier, or move waiting elsewhere; that still needs limits.

Now consider a different case: the burst ends after four seconds, with eighty requests waiting. New arrivals fall to forty per second. The worker can now clear forty waiting requests each second, so those eighty waiting requests disappear in two seconds. If arrivals merely fall to eighty, the queue stops growing but never drains. This is when a bounded queue helps: the burst ends, and later capacity can catch up before the result becomes useless.

A durable queue keeps jobs in storage across a restart. It preserves waiting work, not processing capacity. That suits jobs whose results can arrive later. A cache reuses a computed answer, avoiding repeated work. Backpressure asks the supplier to slow down; it helps only if that supplier responds. For sustained overload, change the work required, the work arriving, or the capacity that can finish it.

### Diagram walkthrough

- **Waiting space and processing capacity are different.** Requests wait in the queue until the worker can take them. The tiles represent waiting space, not extra workers or faster execution.

- **100 arrive − 80 finish = 20 more waiting each second.** The worker stays busy, but each second brings twenty requests it cannot yet finish. This simple model assumes equal-cost work, constant rates, and no refusal or cancellation.

- **Start the fill-time comparison from empty.** Reset the illustration to an empty 100-slot buffer. The available slots, not the finishing rate, determine how long this particular excess can be stored.

- **The 100-slot queue is full after five seconds.** 100 free slots ÷ 20 extra requests per second = 5 seconds. Every slot is occupied, while the worker still finishes only eighty requests each second.

- **Twice the waiting space lasts twice as long—not forever.** This alternative starts with 200 empty slots under the same continuing overload. After ten seconds it is full too: the larger buffer has not removed the twenty-request-per-second mismatch.

- **A full queue needs an explicit policy.** Keeping a full queue does not create another slot. Refuse new work, get its supplier to slow down, or recognize that waiting is moving elsewhere and needs limits there too.

- **Alternative: the burst ends before the queue fills.** Reset to a different case: after four seconds of twenty extra requests per second, eighty requests are waiting. The burst now ends; this is not a continuation of the full-buffer example.

- **New work now uses only part of the worker’s capacity.** Forty new requests arrive per second while the worker can finish eighty. The remaining forty completions per second can reduce the already waiting backlog.

- **The queue shrinks by forty requests per second.** After one second of the quieter period, the backlog falls from eighty to forty. Using the full finishing rate as the drain rate would forget the new requests still arriving.

- **Two quieter seconds clear the backlog.** 80 waiting ÷ (80 finishing − 40 arriving per second) = 2 seconds. New requests still arrive, but the old backlog has drained in this constant-rate example.

- **Matching capacity stops growth; it does not clear old work.** Alternative outcome from the same eighty-request backlog: eighty arrivals and eighty completions per second leave zero spare finishing capacity. The old waiting count stays at eighty.

- **A useful buffer needs a burst that ends and capacity to drain it.** Return to the successful forty-arrivals-per-second case. The backlog cleared because capacity remained after serving new arrivals, while the waiting results were still useful.

- **Preserving jobs changes survival—not processing speed.** Durability preserves waiting work across a restart, not processing capacity. It can suit delayed results, but storage limits, waiting age, and sufficient later capacity still matter.

- **Reuse can remove work from the expensive path.** A valid previously computed answer can satisfy a repeat request without recomputing it. That changes the work required; it is different from storing more work for later.

- **Slower input helps only when the supplier responds.** Backpressure communicates that the receiver cannot keep up. A cooperating supplier reduces the rate; otherwise merely sending a signal does not stop accumulation.

- **Change the mismatch, not just the waiting room.** Reduce work per result, reduce the work arriving, or increase capacity at the limiting component. A bounded queue remains useful only within the waiting and recovery conditions you have actually tested.

### Apply it

Bound queue size and waiting age. Test both the expected burst and the quieter period needed to drain it; decide what happens when the queue is full.

**Boundary:** Durable asynchronous work can be useful; a queue is not inherently a bad design.

**Technical depth:** This constant-rate, equal-cost model starts with a busy worker and no refusals or cancellations. Fill time = free slots ÷ (arrivals − completions); drain time = backlog ÷ (completions − new arrivals), only when the denominator is positive. Moving waiting upstream does not remove its storage or age cost.

**Check:** Eighty requests are waiting. Arrivals drop to 40/s while the worker finishes 80/s. When does this queue clear?

**Explained answer:** After two seconds: the worker has 40/s left for the backlog, so 80 ÷ (80 − 40) = 2s. At 80 arrivals/s, the backlog would not shrink.


## 4 · A timeout ends waiting, not necessarily the work

[Open this lesson](https://ratelimitly.com/?lesson=deadlines#architecture-graphic) · 1:16 at 1×

Maya cannot wait forever. Her caller has a deadline: a point after which it stops waiting for an answer. In this example that deadline is thirty seconds. A timeout is the caller reaching that deadline without the expected response.

At five seconds, Maya has twenty-five seconds left to wait. The interface must still look unfinished. At thirty seconds, it shows a timeout, not a successful report and not an error message supposedly sent by the database.

The database may still be doing the work. Cancellation is a separate instruction asking that work to stop, and each part of the system must carry it through and honor it. A connection is the communication link used by two components. If that link fails, an application may learn about the failure immediately rather than waiting thirty seconds.

Does Maya timing out prove that a database change did not happen? No. A caller stopping its wait does not establish whether the operation stopped, completed, or changed stored data.

### Diagram walkthrough

- **Maya needs a result within a finite wait.** Follow the caller separately from the database. Waiting for a result consumes time, but the caller’s decision to stop waiting does not by itself establish what happened to remote work.

- **Start with the full 30-second waiting budget.** At this starting snapshot, zero seconds have elapsed and thirty seconds remain. The bar represents the caller’s elapsed waiting time, not percent completion of a database operation.

- **A timeout means no expected response before the deadline.** This defines the event we will show next. The request is still waiting in the starting snapshot; when its budget is exhausted without the expected answer, the caller times out.

- **Five seconds have passed; Maya still has 25 seconds to wait.** The caller remains visibly pending. Only five of the thirty seconds have elapsed, leaving twenty-five. Nothing in this snapshot says a report was delivered or that the database finished.

- **The caller’s budget reaches zero: TIMEOUT, no report.** The failure is shown on Maya’s caller, not as an error response supposedly sent by the database. The caller reached thirty seconds without its expected report; this is not a successful completion.

- **The caller stops waiting; remote work may continue.** The separate database track remains unresolved. A caller timing out does not automatically cancel the database operation, stop its resource use, or establish whether a write took effect.

- **Cancellation must travel to the work and be honored.** This is a separate signal asking the operation to stop, not an automatic consequence of the earlier timeout. Each component must propagate or handle cancellation and the downstream operation must actually honor it.

- **The communication link can fail separately from the deadline.** The application and database exchange calls and answers over a connection. A connection’s condition is a different fact from the caller’s waiting budget or the database operation’s final outcome.

- **Some failures are learned about before the timeout.** A broken connection may report an error immediately, without consuming the full thirty-second waiting budget. A fast communication error still does not automatically prove that remote work had no effect.

- **A timeout leaves the remote operation’s outcome uncertain.** No. The operation may still be running, may have stopped, or may have completed without its answer reaching the caller. A timeout does not prove that stored data stayed unchanged, or that a write was rolled back.

### Apply it

Propagate deadlines and cancellation where supported. Track caller and downstream outcomes separately.

**Boundary:** Thirty seconds is this example’s chosen deadline, not a universal default or required setting.

**Technical depth:** For writes, an unknown response may conceal a committed side effect. Do not label a timeout as proof of rollback.

**Check:** Maya times out. Can the database still be working?

**Explained answer:** Yes. Cancellation must reach the operation and be honored; the timeout alone does not stop it.


## 5 · Retained work can exhaust a process’s memory

[Open this lesson](https://ratelimitly.com/?lesson=memory#architecture-graphic) · 2:00 at 1×

Memory is an application's working space, not permanent storage. A process is a running program. It uses memory even before customers arrive. While Maya waits for her report, the application can hold her request details, temporary data, and connection information. Waiting is not free.

After the work finishes and cleanup releases those resources, space can become available again. But if more requests remain unfinished, more state stays in memory. Our tiles illustrate that accumulation; they are not measured bytes per request. Maya closing her browser does not itself release this state. The application must finish or cancel the work and clean up.

A container is an environment for running a program. It can have an enforced memory limit. The host machine can still have free memory while this container reaches its limit. Asking for additional memory is an allocation. If the required memory cannot be provided, we have an out-of-memory condition, or OOM. An allocation can fail, or a process can be killed; the outcome depends on the runtime and system policy.

If the serving process is killed, this copy stops serving requests. Maya gets no report from it. The whole machine has not necessarily failed, and stored data has not necessarily been lost. The lesson is to protect room for finishing work, cleanup, and responses before reaching a hard limit. Refusing more work also costs resources. Waiting until memory is completely exhausted is too late to rely on an orderly refusal.

### Diagram walkthrough

- **Working space is different from permanent storage.** The application needs working space to run. This picture represents one application’s memory budget, not its database records or the size of the entire host computer.

- **The running program already uses some memory.** The tiles marked B represent the program’s existing memory use. This is a qualitative illustration: we are not assigning a fixed number of bytes to every request or runtime.

- **Her waiting request retains real resources.** The new R tile represents request details, temporary buffers, and connection state retained while Maya waits. The request can occupy memory without producing a useful result yet.

- **Finishing and cleanup can release working space.** In this successful alternative, the work finishes and cleanup releases the request’s resources. Actual memory reuse and return to the operating system depend on the runtime; completion alone is not proof of immediate reclamation.

- **Alternative: unfinished requests accumulate.** Return to the waiting case. More requests remain in flight and retain more state. The running program has not become larger in this simplified picture; the extra occupied space belongs to unfinished work.

- **The tiles explain the mechanism, not a benchmark.** Different requests can retain very different amounts of memory. These equal visual tiles help track occupied and free space; they are not a prediction of bytes, request capacity, or a safe production threshold.

- **The caller leaving does not clean up the server.** The R tiles remain: Maya’s browser giving up is not evidence that the application cancelled the underlying work. The application must finish or cancel it and release the resources it owns.

- **A program can have its own enforced boundary.** A container is an environment for running a program and may have an enforced memory limit. That limit is a ceiling on this environment’s use, not a claim that the host reserves these exact physical tiles.

- **Host-wide free RAM does not remove a container limit.** The illustrated container has reached its own boundary even though the host may still have free RAM. Looking only at machine-wide memory can miss the limit that actually stops this application.

- **An allocation is a request for more working space.** The application asks the runtime or operating system for additional memory. If the required memory cannot be provided after the applicable reclamation and enforcement behavior, that is an out-of-memory condition.

- **There is more than one possible failure mode.** An allocation may fail and be handled by the program, or system policy may terminate a process. These are alternative possibilities, not a guaranteed two-step chain or a rule that the largest process always dies.

- **This copy stops serving. Maya gets no report.** Follow the process-termination branch: the application copy serving Maya is stopped. That does not automatically mean the whole host failed, every replica stopped, or permanent records were lost.

- **Act before the hard limit leaves no room to respond.** Return to a running application with some free space. Keep enough room for accepted work, cleanup, and responses; even refusing a request consumes resources. A check made only after total exhaustion cannot guarantee an orderly refusal.

### Apply it

Track retained work and the enforced process/container budget. Check cancellation and cleanup, including requests whose callers have gone away.

**Boundary:** The illustrated termination is one possible OOM outcome. Victim selection and failure scope vary.

**Technical depth:** Include garbage collection, native allocations, connection buffers and enforcement scope when selecting headroom; a single ratio cannot cover every runtime.

**Check:** Maya closes her browser. Does the application automatically release everything her request holds?

**Explained answer:** No. The application must finish or cancel that work and release its resources; the browser giving up does not establish that cleanup happened.


## 6 · One failed copy can push its neighbors over capacity

[Open this lesson](https://ratelimitly.com/?lesson=replicas#architecture-graphic) · 1:59 at 1×

A service can run several copies of its application. Each copy is a replica. Here they are servers A, B, and C. A load balancer distributes requests between copies that can serve them. In our example, each can finish one hundred equal-cost requests per second.

With two hundred forty requests per second, divided evenly, each receives eighty. Now server A stops. Once the balancer detects that failure, it redirects those requests. Each survivor now receives one hundred twenty per second, but can still finish only one hundred. Demand did not increase. Available capacity decreased.

If the survivors retain all that work, each accumulates twenty extra unfinished requests every second. Waiting and memory can grow. If server B also stops, server C receives all two hundred forty requests per second. It cannot finish one hundred forty of those each second. If C then fails, no serving copies remain. This spread through the effects of an earlier failure is a cascading failure.

Is that chain inevitable? No. Enough spare capacity, or refusing excess work, can interrupt it. For example, with two survivors, admitting ninety requests per second to each and refusing sixty overall leaves room below their combined capacity. Those numbers illustrate the principle, not a safe production setting. Refusal itself costs resources. Test that remaining capacity can finish accepted work and respond when a replica disappears.

### Diagram walkthrough

- **These are three separate copies of the same application.** A, B, and C are individual serving copies, not three stages in one request. Keeping their names and positions fixed lets us follow what happens to each one as capacity is lost.

- **The load balancer chooses a serving copy.** The branching arrows are alternative destinations for incoming requests. They do not mean that one request is copied to all three servers, and the load balancer does not increase each server’s processing speed.

- **Each server has the same finite finishing capacity.** In this worked example, each copy can finish 100 equal-cost requests per second. We have not assigned demand yet. Three healthy copies supply 300/s of combined capacity under these assumptions.

- **240 requests/s divided three ways is 80 per server.** The same 240 requests per second are distributed evenly: A, B, and C each receive 80. Each is below its 100/s finishing capacity. The gauge scale stays fixed across every later step.

- **A is still on the diagram, but contributes no capacity.** A has stopped. B and C are still in the same positions. This intermediate picture separates the lost path from the following redistribution: 80/s previously directed to A must now be handled or fail.

- **The same demand is routed onto two survivors.** After detection, this example routes all 240/s to B and C. Real detection and routing can take time; the diagram is a causal sequence, not a timing benchmark or a guarantee of even balancing.

- **120 arrive, but only 100 can finish on each survivor.** Both surviving servers cross the 100/s capacity marker. Demand has not increased; total usable capacity fell from 300/s to 200/s. More traffic reaches each remaining copy because there are fewer copies.

- **Unfinished work grows by 20/s on each survivor.** If each server keeps everything it cannot finish, 120 arriving minus 100 finishing leaves 20 extra unfinished requests every second on each. Their waiting work can retain memory; growth is conditional on that work not being bounded or removed.

- **C now receives all 240/s alone.** Follow the severe branch: B also stops and the whole demand is routed to C. C’s capacity has not grown: 240 arriving minus 100 finishing leaves 140/s of excess if it accepts and retains everything.

- **No serving copies remain. Customers get no useful result.** In this explicitly conditional severe outcome, the last application copy also fails. Incoming customer demand still exists, but there is no serving application capacity; failure has spread through the earlier losses.

- **Return to the first loss: the chain can be interrupted.** This is an alternative to the severe continuation, not an automatic restart. A is unavailable, while B and C still have a chance to survive if enough capacity remains or excess work is refused.

- **Refuse some work so the accepted work can finish.** An illustrative alternative admits 90/s to each survivor: 180/s accepted and 60/s refused out of the same 240/s offered. Both stay below the assumed 100/s finishing capacity; this is not a recommended production threshold.

- **Containment needs measured room to respond.** The refused requests still consume some network, routing, and response resources. Test the real request mix and loss behavior: the numerical margin here explains the idea but cannot prove a safe threshold for your deployment.

### Apply it

Test replica loss at expected demand, including detection delay, redistribution, retained work, and the cost of refusing excess on survivors.

**Boundary:** The example assumes even redistribution and no additional capacity. A failed replica need not cause a fleet outage.

**Technical depth:** Aggregate healthy capacity is not necessarily evenly usable. Include uneven routing, expensive request mixes, and resources consumed by refusals. The 90/s admitted example is not a recommended production threshold.

**Check:** Why can one failed replica put healthy survivors at risk?

**Explained answer:** Demand is redistributed onto fewer replicas. Each survivor can exceed its own capacity.


## 7 · Refuse new work while the application can still recover

[Open this lesson](https://ratelimitly.com/?lesson=local#architecture-graphic) · 1:28 at 1×

The application can make a decision before accepting more expensive work. That decision is admission. Load shedding means deliberately refusing some work so the work we keep has a better chance of finishing.

For local memory protection, observe the memory budget of this application process or container. Keep headroom: unused capacity for work already accepted, resource cleanup, and sending responses. When pressure is too high, refuse new work before making large allocations. Middleware is application code that runs around request handling; it is one place to perform this check.

HTTP is the request-and-response protocol used by the web. Its response status code tells the caller what happened. A 503 response can communicate temporary service overload. Producing that response still costs resources, so even refusal needs headroom.

Who owns this protection? Your application does. RateLimitly does not currently supply this memory-monitoring middleware. Local memory checks also do not protect every CPU, connection, or database limit. They address the resources this application can observe and control.

### Diagram walkthrough

- **Admission decides whether new work may start.** Admission is a decision before new work starts. Load shedding deliberately refuses some work so admitted work has a better chance of completing.

- **Keep headroom for accepted work, cleanup, and responses.** Observe the resource limit that applies to this process or container. Leave space for accepted work, cleanup, and the response used to refuse more work.

- **Under pressure, refuse early with an appropriate response.** A temporary-overload response can use HTTP 503. Sending that response still consumes resources, so refusal must remain a cheap path.

- **Local resource protection belongs to the application.** This memory guard belongs to your application, not a current RateLimitly middleware feature. It does not directly protect every shared dependency.

### Apply it

Check local pressure before large allocations; bound retained/concurrent work and define the overload response.

**Boundary:** This is application-owned protection, not a current RateLimitly local-memory feature or a complete resource policy.

**Technical depth:** Choose runtime-aware signals and test threshold behavior under measurement delay, allocation spikes and cleanup costs. Platform concurrency limits can complement memory admission.

**Check:** Does refusing a request cost nothing?

**Explained answer:** No. Refusal still uses resources, so the application needs headroom before it reaches a hard limit.


## 8 · Different compute platforms still have finite resources

[Open this lesson](https://ratelimitly.com/?lesson=compute#architecture-graphic) · 1:28 at 1×

Compute is the part of the system that runs application instructions. That work may run in a fixed pool of servers, containers, or a group that adds and removes servers automatically. That last approach is autoscaling. New capacity takes time to become usable, and adding application copies does not necessarily add database capacity.

Serverless means the provider manages the machines and starts your code when needed; it does not mean no machines or no resource limits exist. An execution environment is one place where a copy of that code runs. Each environment has finite resources. Some may be reused, and some may be new.

Concurrency means how many operations are active at the same time. Limiting concurrency can bound how much work remains active inside an environment or against another component. Available controls and memory signals differ between platforms, so one memory-check recipe is not automatically portable.

If application capacity doubles while database capacity stays unchanged, did the whole system double its useful capacity? Not necessarily. The component that cannot keep up still limits the result.

### Diagram walkthrough

- **Compute is what runs the application instructions.** Maya asks for a report. The application is the program; compute supplies the resources that execute its instructions. Receiving a message is not the same as doing that work.

- **The same application can run on servers or in containers.** The two panels are alternative placements, not a chain of calls. A server pool has separate machines running application copies. Containers provide separate places for programs on a host; they do not remove the host’s resource limits.

- **Autoscaling changes how many application copies are requested.** Requests reach A and B. C has been requested, but it contributes no ready capacity yet. Count the copies that can serve now—not the number you asked the platform to create.

- **A starting copy is not ready capacity.** C is initializing. A and B still handle the incoming work; C has no incoming request path yet. Adding a desired copy does not instantly relieve the existing servers.

- **More ready callers do not enlarge the shared database.** Now suppose C is ready too. All three application copies call the same database. The database is a separate resource: adding an application copy does not automatically add capacity there.

- **Serverless moves machine management to the provider.** Your code still runs on machines, inside the provider-managed boundary. The provider starts that code when needed. The word serverless does not mean machines or finite resources have disappeared.

- **An execution environment is one place where a code copy runs.** A and B are separate execution environments inside the managed infrastructure. Each box is a place for a copy of your application code, not an unlimited shared machine.

- **Each environment still has its own finite resource budget.** Provider-managed execution still has a resource budget. Each environment can run out of its available memory. The strips illustrate finite budgets, not byte measurements or platform defaults.

- **A later invocation may reuse A or need a new environment.** The next invocation may reuse environment A, where code has run before, or start in a new environment B. These are alternatives—not two calls made by the application. Never rely on getting A again.

- **Concurrency counts operations active at the same moment.** Follow the vertical Now line. A, B and C have all started and none has finished: concurrency is three. The earlier completed operation no longer counts. Concurrency measures overlapping work, not requests per second.

- **An active-work limit prevents another operation starting when full.** Choose an illustrative limit of three in this scope. A, B and C are still active, so D cannot start yet. Decide what to do with excess work; moving it to an unbounded waiting area would not solve the resource problem.

- **Choose the control and the scope your platform actually supports.** An application-scoped limit covers work in one environment. Protecting a shared dependency also requires reasoning about all its callers. Platforms expose different memory signals and controls; one server middleware recipe is not automatically portable to every serverless runtime.

- **Two more callers. No larger database.** A and B were already present; C and D are added. All four paths converge on the same database. We doubled the equal-sized application fleet, not the capacity of every component it depends on.

- **More compute does not necessarily mean more useful end-to-end capacity.** If the database already limits completions, more callers do not necessarily produce more reports. They can simply send more work to the same bottleneck. Check the entire request path before treating added compute as added useful capacity.

### Apply it

Use signals and concurrency/admission controls actually available in the selected runtime and platform.

**Boundary:** Do not assume identical per-request memory observability or middleware support across serverless platforms.

**Technical depth:** Measure ready capacity and per-environment limits separately from desired fleet size. Provider limits and application-owned admission have different scopes.

**Check:** Can adding app servers overload a database that did not grow?

**Explained answer:** Yes. More callers can send more work into the same finite database capacity.


## 9 · A shared database failure can spread back into every app

[Open this lesson](https://ratelimitly.com/?lesson=database#architecture-graphic) · 1:32 at 1×

A dependency is another component an application needs to complete some work. A database used by many application copies is a shared resource: all of those copies depend on its finite capacity. A database query is an operation asking it to read or change data.

Many applications send queries at once. If the database cannot finish them fast enough, active queries, connections, and waiting work can consume resources. Under some configurations, memory exhaustion may stop database operations or make the database unavailable. This is a possible severe outcome, not what every database must do.

The applications now wait for answers or receive connection errors. If they keep admitting requests and retaining waiting work, their own memory can grow. Unprotected application processes can fail as a consequence of the database failure.

With local memory protection, the applications may stay alive by refusing excess work. But can they produce a report that requires the unavailable database? No. Keeping the application process alive is useful, but it does not restore the missing dependency. Unavailability alone also does not prove stored data was lost.

### Diagram walkthrough

- **Several applications depend on one database.** Adding application copies gives this shared database more callers. It does not give the database more capacity.

- **Work can accumulate at the shared tier.** Active queries, connection state and waiting work can all consume resources. Where the backlog lives depends on pools, limits and the database configuration.

- **The database may become unavailable.** This severe branch is conditional, not inevitable. Allocation failures or terminated database processes may disrupt service; unavailability does not prove data loss.

- **Failure propagates through dependency calls.** A missing answer can leave callers waiting. A broken connection can instead report an error quickly. Those are different paths with different retained-work costs.

- **Retained work is the link to application OOM.** The cascade requires retained work to keep growing. Fast errors with bounded cleanup do not automatically exhaust application memory.

- **One shared failure can become a fleet failure.** If application processes exhaust their own memory, the outage has spread beyond the database. More restarting app copies alone will not repair that shared dependency.

- **Local shedding contains the secondary failure.** Your application can refuse excess work before its own memory is exhausted. This protects its ability to respond, not the availability of the database.

- **An alive process is not a restored service.** No: a report requiring the unavailable database still cannot be produced. Explicit refusal is different from success, and unavailability does not prove data loss.

### Apply it

Bound application waiting and database concurrency; treat fast connection errors differently from deadline expiry.

**Boundary:** Some databases refuse work or fail only an operation. This scene deliberately illustrates a stated severe database-unavailable branch.

**Technical depth:** Failure scope depends on database/runtime configuration, connection behavior and memory enforcement. App OOM requires retained work to keep growing; fast errors with bounded admission can avoid it.

**Check:** Local memory guards keep the apps alive. Are database-backed reports available?

**Explained answer:** Not while the required database remains unavailable. Process survival and customer functionality differ.


## 10 · Restore capacity before restoring full traffic

[Open this lesson](https://ratelimitly.com/?lesson=recovery#architecture-graphic) · 1:50 at 1×

Restarting processes is not the same as restoring useful service. In our earlier example, two ready copies could finish two hundred requests per second. Sending the old two hundred forty can overload them before the third copy is ready.

Remember that a cache reuses an answer instead of asking the database again. Suppose, among one hundred requests, a warm cache answers eighty and twenty need one database call each. A cold cache has none of those answers: the same hundred requests now need a hundred database calls. That is five times the database work, without more customers.

Reduce or temporarily stop admitted traffic through a tested control. Keep resource limits active during recovery. Restore required dependencies and ready application copies. Where warming caches helps, bound that work too; warming itself loads the database.

Send a small trial of real work. Check that complete reports arrive, waiting clears, and memory stays bounded. A healthy process alone is not enough. Increase in small steps, observing each step before adding more. If waiting or failures grow, hold or reduce traffic again. A timer or fixed ramp percentage is not proof that capacity has recovered. Recovery order and useful cache preparation depend on the application. Rehearse this with its actual dependencies, instead of assuming every restart can immediately take full traffic.

### Diagram walkthrough

- **Starting a process does not prove the service is ready.** The application copies return at different times. C is still starting and supplies no ready capacity. Even a running copy also needs working dependencies before it can complete the customer’s report.

- **The old demand can overload an only partly restored fleet.** Using the earlier equal-cost model, two ready copies provide 200/s while the offered demand is 240/s. A third starting copy adds no ready capacity yet. Actual recovery capacity can be lower while caches or dependencies recover.

- **A reusable answer avoids a database call.** A valid cached answer can serve a repeated request without asking the database again. Restart behavior depends on where the cache lives; the following example explicitly considers a cache whose reusable answers are absent.

- **Warm cache: 100 customer requests need 20 database calls.** In this illustration, eighty requests reuse valid answers and twenty each need one database call. Every tile represents ten customer requests, so the C tiles avoid database work and the D tiles require it.

- **Cold cache: the same 100 requests need 100 database calls.** The same 100 customer requests now find no reusable answers. Assuming one database call per miss, every request reaches the database. We did not add customers; we removed the reuse that previously avoided work.

- **20 → 100 calls: five times the work, not five times the customers.** Compare the two cases for the same one hundred requests. Database calls increase fivefold in this simplified example. That does not imply five times the latency, or establish any safe traffic rate for the recovering database.

- **Control the input while useful capacity is being restored.** Use a recovery control that has actually been rehearsed. Reducing or stopping admitted work keeps the recovering system from being immediately overwhelmed by the old demand; the exact mechanism depends on your application.

- **Recovery still needs room to finish work and respond.** Do not disable the application’s resource bounds just because it is recovering. Accepted work, cleanup, and refusals still need working space while dependencies and traffic conditions are changing.

- **Restore the components that the complete result depends on.** Required dependencies must respond and enough application copies must be ready. This is an end-to-end readiness relationship, not a universal process restart order; your dependency structure determines the recovery sequence.

- **Cache warming is work too; it also needs a budget.** Preparing useful cache entries can reduce later misses, but the preparation itself may query the database. Bound that work against recovering capacity. Some caches remain warm or do not benefit from preparation; warming is not a universal prerequisite.

- **Test with a small amount of real customer work.** Follow the highlighted trial step. A real report exercises the useful path through the application and dependencies. The rest of the diagram is a decision guide, not a claim that its success conditions have already been observed.

- **Observe useful completion and bounded retained work.** Check complete reports reaching customers, waiting work clearing, and memory remaining bounded. A health check proving that a process is alive is not enough evidence that its dependencies and real request path can handle the offered work.

- **Only then try one small increase and observe again.** A successful trial justifies testing a little more traffic, not jumping to the former full load. Allow meaningful request lifecycles and cleanup to reveal the effect of each change before making another increase.

- **Worsening results mean hold or reduce—not keep increasing.** Follow the alternative branch when waiting or failures grow. Pause the increase or reduce admitted work, then reassess. Capacity changes during recovery, and an earlier successful trial cannot justify blindly continuing the ramp.

- **A schedule is not evidence of recovered capacity.** An elapsed timer, fixed percentage, or desired replica count is not proof that capacity has recovered. Rehearse admission controls and dependency-specific recovery, including cache behavior and meaningful observation between traffic changes.

### Apply it

Keep admission and resource bounds active; send a small trial, observe complete customer results and retained work, then increase, hold, or reduce traffic based on the observed effect.

**Boundary:** Cache preparation and recovery order depend on the application. Do not invent one universal restart procedure.

**Technical depth:** Recovery capacity is time-varying. Rehearse cache misses, dependency readiness, caller repeat attempts, routing delays and the cost of warming. Observe each ramp step over meaningful request lifecycles; a timer or desired replica count does not establish recovered capacity.

**Check:** The same 100 requests previously needed 20 database calls. With no reusable answers, each needs one call. What changed before adding any customers?

**Explained answer:** Database calls rose from 20 to 100: five times the work in this illustration. That does not imply five times the latency, or prove that restarting processes restored useful capacity.


## 11 · A circuit breaker stops repeating a failing call

[Open this lesson](https://ratelimitly.com/?lesson=breaker#architecture-graphic) · 1:23 at 1×

Imagine a video website asking another service for suggested videos. That service is a dependency. If its calls repeatedly fail, a circuit breaker can stop the application from making more of those calls.

Closed means calls are permitted. In a simple example, three observed failures within sixty seconds open the breaker. Open means calls are rejected locally, without waiting for the remote answer. The application can show a precomputed list of this week's releases instead.

After a configured wait, half-open allows a small number of test calls, called probes. Successful probes can close the breaker. Failed probes open it again. This is a repeating recovery cycle, not a promise that waiting repairs the service.

Keep three settings separate: the deadline for each call, the window that counts failures, and the time spent open before testing recovery. Modern breakers can also detect slow calls. They do not all wait for complete failure. Local memory and concurrency bounds still protect the caller while outcomes are pending.

### Diagram walkthrough

- **A recommendations dependency can fail while the application still works.** A circuit breaker changes whether the caller attempts a dependency operation. A fallback can avoid waiting for another failing call.

- **Closed → failure threshold → open → wait before bounded probes.** Example trip rule: three observed failures in a 60-second window. Open refuses locally; it does not cancel every call already admitted.

- **Half-open branches on probe results; failures return to open.** After the open wait, half-open admits limited probes. Their outcomes determine the next state; passage of time does not repair the dependency.

- **A deadline, an observation window and an open-state wait are different clocks.** Keep call deadline, observation window, and open-state wait separate. Modern implementations can include slow-call criteria as well as errors.

### Apply it

Choose counted failures, observation window, request deadlines, open-state wait and probe count separately. Keep local resource bounds.

**Boundary:** Three failures in sixty seconds is an illustrative basic policy, not a universal default. Some breakers also use slow calls or resource limits.

**Technical depth:** A sampling window is not a concurrency ceiling. Define failure classification and cancellation; bound calls already admitted before a trip.

**Check:** Does an expired open-state wait prove the suggestions service recovered?

**Explained answer:** No. It permits a limited recovery test. Failed probes can reopen the breaker; the application can keep showing a cheap list.


## 12 · One customer’s excess should not ruin another’s day

[Open this lesson](https://ratelimitly.com/?lesson=tenants#architecture-graphic) · 1:19 at 1×

Return to Maya's report. Her organization, Northstar, shares this service with another organization, Orchid. A tenant is a customer or organization whose usage we want to account for separately.

Orchid accidentally starts a bulk job that sends many expensive requests. Maya sends one ordinary request, but both organizations compete for the same finite capacity. A noisy neighbor is one tenant whose activity makes the service worse for others. It need not be deliberate. Here, abuse means exceeding an intended share, not judging the customer's intentions.

We want to refuse Orchid's excess before it performs expensive work, while allowing Maya's reasonable use. Rejecting every customer equally would miss that goal. The application therefore needs a trustworthy tenant identity attached to each request.

If one bulk job makes a thousand other customers wait, adding a boundary around that job protects real people. Maya should not lose her meeting because somebody else's integration ran out of control.

### Diagram walkthrough

- **Maya’s organization shares the service with another customer.** Northstar and Orchid are distinct organizations using one service. Follow Northstar’s circular N token and Orchid’s square O token; their identities remain readable without relying on color.

- **A tenant is the customer boundary used for separate accounting.** The two customer labels define which organization each request belongs to. The drawn work area is shared; identifying tenants does not automatically create separate servers, memory, or reserved execution slots.

- **Orchid’s bulk job creates many expensive requests.** Orchid’s tokens occupy the shared work area in this illustrative congested state. The number of drawn tokens is schematic, not a traffic rate, limit configuration, or measured processing capacity.

- **Maya waits even though she sent only an ordinary request.** The same shared work area is occupied by Orchid’s activity while Maya’s Northstar request waits. The customer harmed by this congestion is not necessarily the customer producing the excess demand.

- **Protect other customers without assuming malicious intent.** The congestion can come from a mistake in an integration rather than an intentional attack. The protection goal is to enforce intended use and preserve service for other tenants, not to infer the customer’s motives.

- **Refuse Orchid’s excess before it enters expensive work.** This is the desired protected alternative. Orchid’s excess is refused, while permitted Orchid work and Maya’s reasonable request can enter the shared work area. The request tokens do not specify a production quota or guarantee reserved capacity.

- **Blanket refusal would deny Maya as well.** This contrasting policy rejects both tenants. It avoids adding this work but misses the noisy-neighbor goal: allowing reasonable use while containing the organization producing the excess.

- **Use a trustworthy tenant identity before making the decision.** Attach the tenant identity from trusted application context. A caller-provided name alone is not a trustworthy boundary; the admission decision must account against the correct organization before expensive work begins.

- **Contain one runaway job so other people can keep working.** Return to the selective policy: refuse the excess rather than treating every customer as the problem. This protects useful service for people like Maya; it does not promise isolation from every possible shared-resource failure.

### Apply it

Derive tenant identity from authenticated application context. Choose the intended scope: user, organization, or another explicit customer boundary.

**Boundary:** An IP address is not always one customer: people may share one address, and a customer may use many.

**Technical depth:** Do not let an untrusted request choose another tenant’s identity. Combine authenticated policy scope with independent resource bounds.

**Check:** Orchid’s bulk job is accidental. Does the fairness boundary still apply?

**Explained answer:** Yes. Resource impact, not intent, determines whether excess usage needs to be refused.


## 13 · A rate limit checks one customer’s recent allowance

[Open this lesson](https://ratelimitly.com/?lesson=allowance#architecture-graphic) · 1:24 at 1×

A rate limit answers a specific question: has this tenant used its allowance recently? The allowance is how much work the policy permits. A time window is the period over which that use is considered.

For a simple example, suppose Orchid may start sixty ordinary reports per minute. The next report may be refused when its allowance is exhausted. The response can use HTTP status four twenty-nine, meaning too many requests for this policy. No expensive report should start after that refusal.

The exact algorithm matters. Some policies permit a short burst, meaning several requests close together. RateLimitly uses an estimate that decays over time; do not mistake this example for an exact calendar-minute counter. A work weight charges a more expensive operation more than a cheap one, when the application can assign meaningful relative costs.

Does sixty requests per minute guarantee low memory usage? No. Those requests can take different amounts of time and memory. The allowance controls recent use, while the application's local protection still controls its own resource risk.

### Diagram walkthrough

- **Start with the tenant, not the whole service.** Each tenant has its own intended usage record. Orchid’s use does not consume Northstar’s allowance; both still use shared infrastructure.

- **The window says which recent use matters.** Allowance and time scale form a policy. This is a question about recent use, not a measurement of memory or database health.

- **Orchid’s example policy: 60 ordinary reports per minute.** The number describes the intended policy. It is not a display of an exact fixed-window counter. Northstar keeps its own allowance.

- **Refuse Orchid’s excess; do not stop every tenant.** Here Orchid has exhausted its allowance while Northstar remains within its own. Maya’s work may pass this fairness check; it must still pass other required checks.

- **429 identifies the policy refusal.** This is the application responding to Orchid’s excess request. It is not a database error and does not say that the entire service is down.

- **Make refusal a real branch: no expensive report starts.** The refused branch must skip the costly operation. Returning 429 but continuing the same report in the background defeats this protection.

- **The usage estimate decays; it does not reset on a clock boundary.** Time passing reduces estimated recent use. Do not infer a fresh full allowance at the next calendar minute from this example.

- **Charge costly work more when you can assign meaningful weights.** A policy charge can represent relative cost. It still is not a direct reading of current memory or database capacity.

- **A start-rate allowance is not a memory budget.** Longer operations can retain more simultaneous work even at a bounded start rate. Keep application-owned memory and concurrency bounds alongside tenant fairness.

### Apply it

Choose a tenant scope, allowance, time scale, and any meaningful work weights. Check before costly work and handle a policy denial explicitly.

**Boundary:** The sixty-per-minute example illustrates a policy, not an exact fixed-window guarantee of RateLimitly’s decaying estimator.

**Technical depth:** A rate bound is not a concurrency bound. Long-lived accepted work can accumulate even when its start rate is limited.

**Check:** Sixty requests pass the policy. Are their memory costs necessarily equal?

**Explained answer:** No. Request costs and durations vary; weights help express a policy but do not replace resource protection.


## 14 · Copies and languages must share the same intended policy

[Open this lesson](https://ratelimitly.com/?lesson=sharing#architecture-graphic) · 1:28 at 1×

One customer may reach several application copies, written in different programming languages. If each copy gives Orchid its own full allowance, adding copies also multiplies Orchid's effective allowance. The policy has changed without anyone intending it.

RateLimitly is a service that applications ask for rate and latency decisions. A client library is code that lets your program make those calls. The API key defines which RateLimitly servers those libraries use. Keep it on the trusted application side.

A bucket is the named usage record for a rate-limit policy. Use the same API key, the same tenant-derived bucket name, and the same time window and allowance across the client libraries. The libraries then apply the intended policy across your application copies instead of giving each copy its own allowance.

For example, Orchid's report requests should use the same bucket whether they come through Python or JavaScript. Adding an application copy or changing languages should not grant Orchid another share. Use distinct tenant-derived names for different customers so one customer's use is accounted for separately.

### Diagram walkthrough

- **Separate local allowances can multiply one customer’s share.** Independent full allowances per application copy multiply a tenant’s effective budget when the fleet grows.

- **The API key defines which RateLimitly servers the libraries use.** The API key defines which RateLimitly servers the libraries use. Keep the key in trusted application code, not the public browser.

- **Use the same API key, tenant bucket name, window, and allowance.** Use the same API key, tenant-derived bucket name, time window, and allowance across the libraries. Each application copy applies the intended policy.

- **Changing languages does not give Orchid another share.** Orchid uses the same named policy through Python and JavaScript. Different customers use distinct tenant-derived bucket names, keeping their usage separate.

### Apply it

Configure every client library with the same API key, tenant-derived bucket name, time window, and allowance. Keep the key on the trusted application side.

**Boundary:** This example covers one tenant's named policy under the same API key. It does not reserve memory or execution slots.

**Technical depth:** Bucket identity is derived from its logical name, time window, and rate limit. Changing those settings defines a different policy; each request's work weight does not change the bucket identity.

**Check:** Orchid's report moves from Python to JavaScript. Should that give Orchid another allowance?

**Explained answer:** No. Use the same API key and the same tenant-derived bucket name, window, and allowance so both clients apply the same intended policy.


## 15 · Fairness helps contain damage without reserving every resource

[Open this lesson](https://ratelimitly.com/?lesson=isolation#architecture-graphic) · 1:25 at 1×

A ship's bulkhead separates compartments so trouble in one does not flood every other compartment. In software, a bulkhead separates resources or work so one part cannot consume everything. Resource isolation means setting aside or bounding resources for a particular part.

A tenant rate limit is bulkhead-like: it contains excessive arrivals from one customer and makes other customers less likely to suffer. That is a valuable fairness boundary. But it does not necessarily reserve a processor, memory, database connections, or active execution slots for Maya.

Imagine two tenants each start ten requests. One tenant's requests finish quickly. The other's requests remain active and retain much more memory. Equal request counts did not create equal resource use. Work weights can improve the allowance, but strict resource separation may require separate pools or additional concurrency bounds.

Choose the strength of the promise carefully. Say that the rate limit contains a tenant's excessive use. Do not say that rate limiting alone makes every tenant fully isolated from every other tenant.

### Diagram walkthrough

- **A bulkhead contains the spread of trouble.** A software bulkhead separates or bounds resources so trouble in one part does not consume everything in another part.

- **A rate limit contains arrivals, not every resource.** A tenant rate limit is bulkhead-like: it contains excessive arrivals. It does not necessarily reserve memory, connections, or execution slots.

- **Ten quick requests and ten long requests are different.** Ten short operations and ten long operations have equal counts but different retained-work costs. Weights improve accounting; they do not create strict isolation.

- **Match the promise to the protection.** Name the actual guarantee: recent-use fairness, concurrency separation, or a dedicated resource budget. They are different protections.

### Apply it

Use tenant rate limits for excessive arrivals; add separate bounded pools or concurrency controls when you need stronger isolation.

**Boundary:** A rate limit alone does not reserve CPU, memory, active slots, or database connections.

**Technical depth:** Distinguish admission accounting from resource scheduling. State which resources are actually bounded or separated.

**Check:** Does an equal request allowance reserve equal memory for each tenant?

**Explained answer:** No. Duration and per-request cost differ. Strict resource isolation needs additional resource controls.


## 16 · A legitimate crowd can exceed shared capacity

[Open this lesson](https://ratelimitly.com/?lesson=surge#architecture-graphic) · 1:20 at 1×

Now the service hosts a popular live show's video suggestions. A commercial break brings thousands of viewers at once. Aggregate demand means their combined demand, not what any one viewer sends.

Each viewer requests one suggestion and stays within the individual allowance. Every fairness check can pass. But all those accepted requests still reach the same database. If combined work exceeds the database's capacity, unfinished work accumulates even though nobody broke a rule.

More application copies and local memory protection can keep the application layer alive. They do not automatically increase the database's capacity. A fixed database-wide rate cap can be useful, but its safe value depends on request cost and available capacity. One simple lookup is not the same work as a large report.

This is a different problem from Orchid's bulk job. We are no longer asking which tenant exceeded an allowance. We are asking whether the shared resource can handle more work now. Passing the fairness check alone does not answer that question.

### Diagram walkthrough

- **A live show brings a legitimate crowd.** A live-event spike can contain only reasonable individual requests. Aggregate demand is their combined work.

- **Every fairness check can pass while aggregate work is too high.** Every per-tenant check can pass while the shared database receives more work than it can finish.

- **Adding application copies does not add database capacity.** Application scaling and local guards do not add database capacity. A fixed aggregate cap also depends on request cost and available capacity.

- **Fairness and current shared capacity are different questions.** Fairness asks who exceeded a policy. Shared-resource admission asks whether adding work looks safe now. One answer cannot substitute for the other.

### Apply it

Keep tenant fairness, but also bound and observe work sent to shared dependencies. Test changes in workload cost and available capacity.

**Boundary:** A fixed shared-resource cap can be a useful guardrail; it is not automatically well-sized for every workload mix.

**Technical depth:** A user-count spike, costlier queries, or reduced database capacity can all make the same per-tenant policy insufficient.

**Check:** Every viewer is within their allowance. Can the shared database still overload?

**Explained answer:** Yes. The sum of individually reasonable use can exceed the database’s available capacity.


## 17 · Measure the operation whose pressure you want to observe

[Open this lesson](https://ratelimitly.com/?lesson=measure#architecture-graphic) · 1:24 at 1×

Latency is how long an operation takes. To learn about a shared database, measure the database operation, not the whole page including unrelated work. A measurement boundary says exactly when that stopwatch starts and stops.

For example, start just before asking for a database connection, and stop when the required database result arrives. This boundary includes waiting for a connection and the query itself. Another boundary could start after the connection is acquired. Neither number is meaningful unless you know which boundary produced it and use it consistently.

A report is an observation sent to RateLimitly: this operation took this much time. Report after the measured work reaches its defined endpoint. Reporting an observation is not consuming a rate-limit allowance. Do not send a made-up zero duration when a call fails or a measurement is missing.

If the measured time rises, work inside that boundary is taking longer. That may include waiting, actual computation, or communication. The number is useful evidence, but it does not directly reveal the database's memory usage or the exact cause of the delay.

### Diagram walkthrough

- **Latency is the elapsed time between a start and a finish.** Follow one operation from START to STOP. Its latency is the time between those two events, not the number of operations performed.

- **Measure the database work, not the entire page.** The page also does unrelated work. The solid bracket isolates the database operation; dashed boxes remain outside that interval.

- **A boundary is a precise choice of start and stop.** Decide which events begin and end the stopwatch. Without that definition, a duration cannot tell another developer which work it describes.

- **Start A before asking for a database connection.** Connection acquisition is inside this chosen boundary. Time spent waiting for an available connection therefore contributes to A, before the query can run.

- **Stop A when the required database result arrives.** Submitting a query is not the endpoint. Keep measuring through the query and the return of the required result; later page work is excluded.

- **A includes connection acquisition plus the query and result.** Read the full solid bracket. Both acquisition and database work contribute to this observation; unrelated page work does not.

- **B starts later, after the connection has been acquired.** This is the same operation, not a second request. A and B stop at the same result, but B deliberately leaves connection acquisition outside its stopwatch.

- **Compare matching boundaries, not just duration values.** A longer A does not by itself mean its query was slower than B: it includes additional work. Use a consistent definition across clients, or deliberately separate different observation scopes.

- **The application sends the measured duration to RateLimitly.** The stopwatch belongs around the chosen operation. Reporting communicates its observed duration; RateLimitly does not execute the database query shown here.

- **Stop the timer first. Then report the observation.** The reporting arrow lies outside the stopwatch bracket. Sending the observation must not extend the duration of the database operation being reported.

- **Reporting duration is not consuming a rate-limit allowance.** This message says how long the measured operation took. It is an observation, not a request to consume tenant allowance and not an admission decision.

- **Missing timing is not zero milliseconds.** A failed call or missing measurement does not prove that no time elapsed. Do not fabricate a zero-duration observation; define how valid timing, failures, and cancellations are handled.

- **Longer time means more time inside the same boundary.** Keep the definition fixed when interpreting a change. The observation is that the measured operation took longer; it does not yet identify why.

- **Waiting, computation, and communication can all contribute.** These are possible contributors, not three measured components or a diagnosis. Which are included depends on the chosen stopwatch boundary.

- **A duration is evidence, not a memory reading or a diagnosis.** The stopwatch tells us elapsed time within its boundary. It does not directly measure database memory, reveal an exact queue length, or establish the cause of the delay.

### Apply it

Start before the protected operation, report its actual measured duration after the defined endpoint, and define how failures and cancellations are observed.

**Boundary:** Do not manufacture a zero-latency report for missing or failed measurements. Reporting semantics must match the measured operation.

**Technical depth:** Including pool acquisition can expose upstream contention, while excluding it isolates another boundary. Heterogeneous operation costs may need separate trackers.

**Check:** One client includes connection waiting and another excludes it. Are their durations directly comparable?

**Explained answer:** Not necessarily. They measured different operations. Use consistent boundaries or separate appropriately scoped trackers.


## 18 · A shared tracker observes; a guard decides; your app acts

[Open this lesson](https://ratelimitly.com/?lesson=tracker#architecture-graphic) · 1:30 at 1×

A latency tracker is a shared record of reported operation durations. It observes. A guard is the check that uses those observations to decide whether new work may start. A threshold is the configured duration at which that guard refuses work. These are three distinct things: observations, a decision, and the application's action.

Before starting a protected database operation, the client asks for admission. With a positive threshold of five hundred milliseconds, a reported signal at five hundred or above causes the current guard to refuse. Setting this threshold to zero disables that latency guard; it does not mean refuse every request.

If admitted, the application does the database work, measures its duration, and reports afterward. If refused, it does not start that protected work. The client library does not automatically rewrite the application into a cheaper one. Your application handles the decision.

A tracker by itself does not stop anything. A guard checked after the expensive work is too late. Put the decision before the work it is meant to prevent, and make refusal a real branch in the program.

### Diagram walkthrough

- **First: the tracker observes reported durations.** The tracker holds a shared record of measurements supplied by clients. Observing those durations is separate from making an admission decision or stopping application work.

- **Second: the guard uses the observations to decide.** The guard uses the tracker’s observations to decide whether new work may start. The measurement record and the admission check have different responsibilities; they are not interchangeable names for the same thing.

- **A threshold sets the duration at which the guard refuses.** The threshold supplies the guard’s configured comparison boundary. It is distinct from the observed durations: observations describe measured work; the threshold is the policy applied to those observations.

- **Third: your application must act on the answer.** Observations inform the guard, but application code must start or skip the protected operation. Merely recording a latency metric does not prevent work from reaching a database.

- **Ask for admission before calling the database.** Only the first side call exists at this step: application to RateLimitly. The database query has not begun. Put this check before acquiring or consuming the resource the guard is intended to protect.

- **At or above 500 ms, this guard refuses new work.** The comparison includes equality. Below this illustrative positive threshold, the latency guard allows on this signal; at 500 ms or above it refuses. Other required checks can still refuse an operation.

- **Zero disables this guard; it is not a zero-time deadline.** The zero setting is an explicit configuration exception. It disables this latency guard rather than refusing every nonzero-latency operation. It does not disable other required admission checks.

- **Allow lets the application make the measured database call.** Follow the direct application-to-database call. RateLimitly is a separate admission side call, not a proxy carrying the query. The result and subsequent report have not been shown yet.

- **Measure across the chosen database-work boundary.** The result returns directly from the database to the application. Measure from the agreed start to finish; this is the duration of the protected work, not the duration of the admission side call.

- **Only after measuring the work does the application report duration.** The last arrow reports the measured duration from the application to RateLimitly. That observation can inform future admissions; it does not retroactively prevent or cancel the work just measured.

- **The application does not start the protected database work.** The refusal comes back to application code, which does not start that database operation. There is no query arrow to the database on this branch. Return a correct cheap fallback or an explicit refusal.

- **Your application implements the refusal branch.** The library does not automatically rewrite an expensive request into a cheap alternative. Your application handles the answer and must not perform the same protected operation while pretending to fall back.

- **Observing without acting is not load shedding.** All three responsibilities are needed: an observation, a guard decision, and application code that obeys it. Recording durations while unconditionally running the expensive operation leaves that work unprotected.

- **Check before work, not after the cost has already been paid.** Return to the first call in the sequence. A guard evaluated after a database operation cannot prevent that operation. Test the refused path by checking that the protected call was never made.

### Apply it

Use the same API key with matching tracker identities and settings across client libraries. Check before the protected operation, branch on the result, and report measured admitted work afterward.

**Boundary:** A positive threshold refuses at equality or above; zero disables this guard. A tracker is not an automatic proxy or application rewrite.

**Technical depth:** Thresholds belong to the guard and need not be part of tracker identity; operations can use different thresholds against the same shared observations.

**Check:** You report durations but never check before work. Is the database protected by those reports alone?

**Explained answer:** No. Reports provide observations. An admission check and an enforced refusal branch are needed before new work.


## 19 · Even the fastest recent observations can become slow

[Open this lesson](https://ratelimitly.com/?lesson=minimum#architecture-graphic) · 1:36 at 1×

Which latency number does the guard use? RateLimitly uses the minimum among recent eligible observations: the fastest measurement that still counts under the tracker's configured history rules. It is not an average or a percentile. Freshness means how recent an observation is.

Imagine the eligible measurements are eighty, ninety, and one hundred milliseconds. The minimum is eighty. Now imagine the eligible set contains six hundred, six hundred twenty, and seven hundred. Its minimum is six hundred. Even the fastest eligible operation is now slower than a five-hundred-millisecond threshold, so that guard refuses new work.

But if an old eighty-millisecond sample still counts, the minimum stays low. That old fast result can mask newer slow results until it stops being eligible. Sparse history means too few eligible observations. The current implementation treats insufficient history optimistically; a zero signal there is not proof of a healthy database.

Sharing reports improves what clients can observe together. Freshness, history settings, and independent resource bounds still matter. Do not interpret an old fast result or missing evidence as guaranteed spare capacity.

### Diagram walkthrough

- **Minimum means the fastest observation that still counts.** Eligibility determines which observations still count under the configured history rules. The minimum is the fastest of those—not an average, percentile, or exact queue length. Freshness describes how recent an observation is.

- **First example: 80, 90 and 100 milliseconds.** These three eligible measurements are 80, 90 and 100 ms. Find the smallest duration among the measurements that count under the configured history rules.

- **The fastest of these observations is 80 ms.** Of 80, 90 and 100 ms, the smallest duration is 80 ms. That is this eligible set’s minimum; it does not describe every request or prove spare database capacity.

- **Second example: 600, 620 and 700 milliseconds.** Now the eligible set contains 600, 620 and 700 ms. This is a different example: the earlier 80-ms observation is not part of this set.

- **Even the fastest observation is now 600 ms.** The minimum of 600, 620 and 700 ms is 600 ms. All eligible observations in this example are at least that slow.

- **Compare the 600-ms signal with the 500-ms threshold.** The fastest eligible observation is 600 ms, above this guard’s illustrative 500-ms threshold. The comparison includes equality; the threshold is not a timer that cancels work.

- **The guard refuses new work.** Because 600 ms is at or above the positive 500-ms threshold, the guard refuses this new operation. Your application must obey the decision and skip the protected work.

- **An old fast sample can keep the minimum low.** If an old 80-ms observation still qualifies alongside newer 600, 620 and 700-ms observations, the minimum remains 80 ms. That can mask current pressure until the old observation stops being eligible.

- **Sparse history: too few eligible observations.** This is a different limitation from an old fast observation: too few measurements qualify to satisfy the tracker’s required history. Missing evidence is not a measured fast response.

- **A zero signal is not proof of a healthy database.** The current implementation treats insufficient history optimistically. Its zero signal in that case is not proof of a healthy database or measured spare capacity; independent resource bounds still matter.

- **Shared evidence still has limits.** Sharing real observations improves the common view. Freshness and history settings still determine the signal; keep independent resource bounds rather than treating old or missing evidence as guaranteed capacity.

### Apply it

Keep tracker history settings consistent, supply actual observations, and test old fast samples, sparse history, and sudden slowdowns.

**Boundary:** This is the eligible recent minimum, not an average, percentile, memory meter, or exact queue-depth measurement.

**Technical depth:** A lingering fast sample can delay refusal; optimistic sparse history can reopen admission. Bound resources independently and test the full feedback delay.

**Check:** A zero signal came from insufficient history. Does zero prove the database is fast?

**Explained answer:** No. It represents optimistic handling of missing evidence, not a measured guarantee of health.


## 20 · Separate local histories can reopen into the same burst

[Open this lesson](https://ratelimitly.com/?lesson=oscillation#architecture-graphic) · 1:34 at 1×

A serverless execution environment can sometimes retain local state between requests. Keeping latency history there seems convenient: no additional service to run. But each environment knows only its own past, and a new one may know nothing.

Suppose many environments still remember fast work. They admit a burst into the same database. Slow results arrive only after that work finishes. This is feedback delay: decisions happen before the consequences become visible.

Those environments can then refuse work together. Few fresh results arrive while they refuse. Their old observations expire, and optimistic empty histories can let many environments reopen together. Another burst reaches the database. Repeated opening and refusing is an oscillation: the system swings instead of settling. This is a possible failure pattern, not a claim that every local controller behaves this way.

A shared tracker lets these clients contribute to and read the same intended observation scope. It reduces fragmented histories without operating a separate history store in each service. Sharing does not remove feedback delay or guarantee stability. Freshness, bounded work, and careful recovery remain important.

### Diagram walkthrough

- **A reused execution environment may remember its own past work.** Each labeled box is a separate execution environment. A reused environment can sometimes retain local history between requests, but those local records are not automatically shared with the other environments.

- **A new environment may start with no latency evidence.** C illustrates a new environment with an empty history while A and B have their own past observations. Missing measurements do not establish that the shared database is healthy or has spare capacity.

- **Worked case: local histories still describe earlier fast work.** For the following possible failure pattern, consider environments whose histories retain old fast observations. Their earlier experience is evidence about the past, not a direct measurement of the shared database’s current pressure.

- **Several environments admit work using old observations.** Follow the three work routes into the same database. A, B, and C still have old fast evidence when admitting this burst; slow results from the newly admitted work have not returned yet.

- **The evidence arrives after the expensive work has already run.** Each returned result can update its originating environment’s local history. That result is delayed feedback about work already admitted; whether it changes the guard’s signal depends on eligibility and history rules.

- **The decision happened before its consequences were visible.** The new evidence comes from an operation admitted earlier. A client cannot use that operation’s final duration before the measurement finishes. Sharing evidence later does not remove this causal delay.

- **Possible next state: local guards refuse together.** Here the eligible histories cause the local guards to refuse new protected work. No new work routes reach the database. This is a possible branch, not a claim that every slow observation necessarily trips every guard.

- **Refusing work also reduces the supply of fresh measurements.** While these clients skip protected work, they produce few new completions to measure. Existing work may still return results, but a refusal itself is not a new database latency observation.

- **The histories lose evidence; expiry does not prove recovery.** The old observations stop qualifying under the history rules. That leaves too little current evidence in this illustrative branch. A history aging out does not itself make the database faster or add processing capacity.

- **Optimistic empty histories can reopen into another burst.** Follow the work routes returning together: the histories are empty, not freshly measured fast. Optimistic reopening can admit another combined burst before useful current evidence has accumulated.

- **The cluster can swing: admit, refuse, then admit together again.** Each row is one environment and each column is a stage, not a measured time interval. The matching shapes show a possible synchronized pattern: a burst, common refusal, and another burst instead of settling.

- **Share the intended observation scope instead of isolated histories.** The arrows now represent reports and admission checks, not database query traffic. A, B, and C contribute to and consult the same intended shared observation scope; they no longer rely only on their separate private past.

- **Sharing helps the evidence; it does not guarantee stability.** Sharing reduces fragmented histories but does not guarantee stable control or remove delayed feedback. Freshness rules, bounded work, and careful recovery remain necessary; there is deliberately no promised flat healthy throughput line here.

### Apply it

Use a shared, consistently scoped tracker across execution environments; test stale samples, new environments, expiry, and coordinated reopening.

**Boundary:** Shared history helps but does not guarantee stable feedback, eliminate delay, or automatically make independent server state global.

**Technical depth:** The closed loop includes admission, work completion, reporting, retention, and recovery. Test the whole loop under rapid changes, not only steady overload.

**Check:** All environments stopped sending work. Why might they reopen into another burst?

**Explained answer:** While refusing, they may receive few fresh completions. Expired local histories can trigger optimistic reopening at similar times.


## 21 · Replace optional expensive work before refusing essential work

[Open this lesson](https://ratelimitly.com/?lesson=degradation#architecture-graphic) · 1:27 at 1×

Some work matters more than other work. Criticality is that difference in importance to the customer's task. A fallback is an acceptable alternative result. Graceful degradation means doing less expensive work while still delivering a useful, correct experience.

During the live show, personalized video suggestions are optional. A precomputed list of recent releases is a cheaper fallback. It must not call the same overloaded database to create that list on demand, or it has not removed the expensive work.

Suppose optional suggestions use a five-hundred-millisecond threshold, while essential reads use nine hundred. These are illustrative policy choices, not recommended settings. At a shared signal of six hundred, the optional guard refuses the expensive suggestions, but the essential guard can still admit its work. The application returns the cheap list instead.

At one thousand milliseconds, both positive thresholds are exceeded, so essential work can be refused too. Degradation is not a promise that everything succeeds. Protect more important work for longer, and never silently substitute an incorrect answer for a correctness-sensitive operation.

### Diagram walkthrough

- **Not every operation has the same customer importance.** Criticality describes importance to the customer’s task. Graceful degradation removes optional work while preserving a correct useful result.

- **The alternative must avoid the expensive operation.** A fallback must avoid the expensive dependency. Building the supposedly cheap list from the same overloaded database defeats the purpose.

- **At 600 ms, shed optional work before essential work.** Illustrative policy: at 600 ms, the 500-ms optional guard refuses while the 900-ms essential guard still admits. Your application chooses the fallback.

- **At 1,000 ms, essential work can also be refused.** At 1000 ms both positive thresholds are exceeded. Essential work may also be refused; never substitute an incorrect result for a correctness-sensitive operation.

### Apply it

Define business criticality, safe precomputed or cached alternatives, and per-operation thresholds. Return the alternative only where it preserves the intended contract.

**Boundary:** 500 ms and 900 ms are illustrative, not sizing guidance. Different thresholds do not guarantee essential work always succeeds.

**Technical depth:** Track degraded-but-useful responses separately from failures. Avoid fallback stampedes and preserve correctness-sensitive write semantics.

**Check:** A fallback makes another expensive call to the same overloaded database. Did it remove that pressure?

**Explained answer:** No. It moved the same burden into a different branch. A useful fallback must actually avoid the protected expensive work.


## 22 · A local breaker learns after calls have already piled up

[Open this lesson](https://ratelimitly.com/?lesson=stall#architecture-graphic) · 1:38 at 1×

Now the suggestions service stops answering. Consider a basic breaker that counts only completed timeouts, each with a thirty-second deadline. For almost thirty seconds, calls can overlap before the first timeout is known. Their waiting state retains memory. Opening the breaker does not erase calls already in flight.

If two hundred application copies each require three observed failures, that is six hundred threshold observations, provided every copy reaches its threshold in the window. It is not a cap on failed calls. A new copy with no history can repeat discovery. A shared circuit breaker can reduce this duplication too.

Consider a different case: the dependency slows down but still answers successfully. A shared latency guard can refuse new work once its eligible signal reaches the configured threshold, before a failure-only breaker has a timeout to count. Modern slow-call breakers can also respond before complete failure.

RateLimitly's benefit is shared latency evidence for admission, not advance knowledge of every outage. If a sudden failure produces no fresh signal, do not assume immediate protection. Bound waiting and retained work, and show a cheap fallback when possible.

### Diagram walkthrough

- **At 0–30 s, overlapping callers can wait before the first timeout becomes observable.** A failure-only detector cannot count an outcome that has not arrived. Many calls can overlap before the first 30-second timeout is observed.

- **Independent discovery can repeat across copies; 200 × 3 is not a failed-call ceiling.** Derived example: 200 copies × 3 observations = 600 threshold observations if every copy reaches its threshold. That is not a cap on total failed calls.

- **Gradual slowdown can produce usable latency evidence while calls still return successfully.** Gradual successful slowdown is different from a sudden silent outage. Latency-based admission can act on slow observations; modern slow-call breakers can too.

- **Use shared evidence with bounded waiting and application-owned fallback.** Without fresh evidence, do not assume immediate refusal. Bound waiting and retained work independently of the detector.

### Apply it

Load-test abrupt stalls and gradual slowdown separately. Share the intended observation scope, bound concurrency and keep a fallback path.

**Boundary:** Shared evidence has freshness and routing boundaries. Shared breakers exist; low-latency shared admission is not an instantaneous outage oracle.

**Technical depth:** Detection delay, admitted concurrency and cancellation determine retained work before a trip. Test new replicas without history and delayed observations.

**Check:** Are six hundred threshold observations a maximum of six hundred failed requests?

**Explained answer:** No. The arithmetic assumes all two hundred copies reach three observed failures. Overlapping calls, new copies and probes can add more failures.


## 23 · Fast errors explain why the protections remain complementary

[Open this lesson](https://ratelimitly.com/?lesson=errors#architecture-graphic) · 1:32 at 1×

A service can fail quickly. Suppose the suggestions service returns an error in five milliseconds. That is a failed result with low latency, not an overloaded queue we have measured.

With a five-hundred-millisecond guard, a five-millisecond observation is below the threshold. RateLimitly's latency report carries a duration, not an HTTP success or failure classification. A latency guard alone therefore does not replace an error-aware circuit breaker.

An error-aware breaker can count those failures and open. Half-open can later restrict recovery probes. By contrast, insufficient or expired tracker history produces a zero signal in the current implementation. That is not proof of health, and it is not the same as a bounded recovery probe.

Choose protection by the question it answers. Is one tenant taking too much? Use its rate limit. Is shared work getting too slow? Use shared latency admission. Are operations failing? Consider an error-aware breaker. Is this application retaining too much work? Bound its local resources. Your application chooses the cheap alternative, whichever protection declines the expensive operation.

### Diagram walkthrough

- **Fast is not the same as useful.** A call has both a duration and an outcome. Looking at only the duration loses the fact that the customer did not receive a usable result.

- **The same failed call returned in only 5 ms.** Keep these two facts attached to the same call. The short duration does not turn its failed result into success or prove that a queue is overloaded.

- **5 ms passes this duration comparison, even though the call failed.** Five is below five hundred. This latency check alone may admit later work. The report protocol carries duration, not an HTTP success/failure classification.

- **An error-aware breaker can act on the failure signal.** Opening depends on its configured criteria and sufficient observations, not one universal number of failures. Low duration does not erase classified errors.

- **A half-open breaker deliberately limits recovery probes.** After the configured open wait, a bounded number of test calls can supply recovery evidence. Waiting alone does not repair the dependency.

- **No eligible tracker history is a different case, not a recovery test.** The zero signal is optimistic behavior when insufficient history qualifies. It is not a measured five-millisecond result and not proof of health or a bounded half-open probe.

- **Choose the signal for the problem, then implement the fallback.** Tenant fairness, shared latency, classified errors, and local retained work answer different questions. Application code chooses a correct cheap alternative when protected work is declined.

### Apply it

Record both duration and outcome without disguising an error as fabricated latency. Configure bounded recovery and cheap alternatives explicitly.

**Boundary:** RateLimitly does not claim every circuit-breaker feature. Shared state and graceful fallback are possible with other designs too.

**Technical depth:** The current latency-report protocol contains duration but no outcome field; an insufficient-history minimum of zero is optimistic admission, not a half-open state machine.

**Check:** Will a 500 ms latency guard necessarily reject a service returning errors in 5 ms?

**Explained answer:** No. Five milliseconds is below that latency threshold. Classify errors separately; keep an error-aware breaker when the failure pattern needs one.


## 24 · Latency feedback helps without measuring exact spare capacity

[Open this lesson](https://ratelimitly.com/?lesson=limits#architecture-graphic) · 1:33 at 1×

The latency signal is useful because longer operations can keep more work unfinished. Consider a system completing one hundred operations per second. In a stable period, if each operation spends one tenth of a second inside the chosen boundary on average, there are ten operations inside it on average.

In a different stable period, the same completion rate and half a second of average time correspond to fifty operations inside. Average time means the total time divided by the number of operations. Little's law is the relationship between these averages: work in flight equals throughput multiplied by time.

RateLimitly does not insert its minimum into that formula. A minimum and an average are different numbers. The tracker therefore does not calculate an exact queue size or read the database's remaining memory. Communication delays, slow computation, and connection waiting can all affect the chosen measurement.

Use latency feedback to decide when adding work looks unwise, not to claim perfect knowledge of spare capacity. A sudden failure can happen before enough new reports arrive. Independent resource bounds and operational monitoring still have jobs to do.

### Diagram walkthrough

- **Stable example: throughput × average time = average in flight.** For a stable period and a consistent boundary, average work in flight equals throughput multiplied by average time.

- **A longer average time retains more work at the same throughput.** Derived examples: 100 × 0.1 = 10; 100 × 0.5 = 50. Longer time can retain more simultaneous work even with the same completion rate.

- **The tracker minimum is not the average in this equation.** RateLimitly does not substitute its minimum sample into Little’s law. Minimum and average are different statistics.

- **Combine feedback with independent safeguards.** The signal is evidence for admission, not an exact measurement of queue length, spare memory, utilization, or stability.

### Apply it

Calibrate and test thresholds against real workload behavior while keeping independent resource limits and monitoring.

**Boundary:** The 10- and 50-operation examples describe separate stable periods, not an exact estimate during a growing transient queue.

**Technical depth:** Use consistent measurement boundaries and compatible averaging intervals. Neither latency alone nor a minimum establishes cause, utilization, memory, or stability.

**Check:** Can you substitute the tracker’s recent minimum for the average time in Little’s law?

**Explained answer:** No. Little’s law relates compatible averages under appropriate conditions. The tracker’s minimum is a different signal.


## 25 · Ask three different questions before expensive work

[Open this lesson](https://ratelimitly.com/?lesson=combine#architecture-graphic) · 1:43 at 1×

Before expensive work, ask three different questions. Does this application have enough local headroom? Is this tenant within its allowance? Is the shared dependency's latency signal below this operation's threshold? The first question is application-owned local protection. The other two are RateLimitly's fairness and latency decisions.

If these required checks admit the work, perform the operation, measure its duration, and report it to the correct tracker. A policy refusal means a check answered no. Use a safe cheap fallback where the product permits one, or return an explicit refusal. Do not secretly start the refused operation anyway.

A transport failure is different: your application could not obtain a reliable answer from the decision service. A failure policy specifies what to do then. Allowing work despite that uncertainty favors immediate access; refusing favors protecting the dependency. Neither choice is universally correct. Choose and test it deliberately, with local bounds still active.

Keep the API key on the trusted application side, match identities and settings, and observe failures separately from ordinary policy refusals. The decision service also needs capacity and failure testing. A protection that gates work becomes part of the service you operate.

### Diagram walkthrough

- **Three different questions protect three different boundaries.** Local resource safety, tenant fairness, and dependency pressure answer different questions. No one check establishes all three.

- **Admitted work is measured; refused work does not start.** Only start protected work after required checks allow it. A refusal must branch around the work, not merely add a log entry.

- **No reliable answer is not a known refusal.** An unavailable decision service is not an ordinary policy refusal. Choose and test failure behavior per operation while retaining independent local bounds.

- **Operate the guardrails as part of the service.** Keep identity, state scope, credentials and observed errors explicit. The decision service is itself a dependency whose failure behavior needs testing.

### Apply it

Local resource check → tenant/latency admission → branch → protected work → measured report. Keep credentials server-side and test refusal and transport-failure branches.

**Boundary:** RateLimitly does not currently provide the application’s memory guard. A decision-service failure cannot safely be interpreted as automatic admission in every product.

**Technical depth:** Choose fail-open or fail-closed per operation, observe admission overhead and failures, and keep independent bounds when decisions are unavailable.

**Check:** A decision-service call failed. Is that the same as a policy refusal?

**Explained answer:** No. A refusal is a known decision. A transport failure is missing or unreliable decision evidence and needs an explicit failure policy.


## 26 · Recovery attempts need limits too

[Open this lesson](https://ratelimitly.com/?lesson=attempts#architecture-graphic) · 1:31 at 1×

After a refusal or a lost response, a client may try again. A retry is another attempt at the same operation. If many clients repeat immediately, they add load just when the service has less capacity.

Backoff means waiting longer between repeated attempts. Jitter means varying that wait so clients do not all return at the same instant. Bound the number of attempts and the total time available; a request whose deadline has already passed should not create endless background work. Honor the service's guidance where its contract provides it.

A missing response does not prove a write failed. Idempotency means repeated attempts have the same intended effect as one attempt. For example, a payment operation must not charge twice simply because the first response was lost. The application must provide the appropriate operation identity and duplicate handling; a rate limit cannot add those semantics for it.

During recovery, new arrivals and repeated attempts both count toward load. Restore traffic gradually, watch useful completions, and keep admission controls active. Another attempt is useful only if it has a realistic chance of delivering a correct result.

### Diagram walkthrough

- **Immediate repeated attempts can amplify pressure.** Retries are new load. Immediate repeated attempts can concentrate demand while useful capacity is already reduced.

- **Backoff waits; jitter spreads; bounds stop endless attempts.** Backoff spaces attempts; jitter prevents every caller choosing the same return instant. Bound attempts and the total remaining deadline.

- **A lost response does not prove a write failed.** A lost response does not prove a write failed. Correct duplicate handling needs operation identity and idempotency semantics supplied by the application.

- **Count new and repeated work during recovery.** Include retries at every layer in the load budget. Another attempt is useful only while it can still produce a correct result within its limits.

### Apply it

Use bounded attempts, total deadlines, backoff and jitter. Define duplicate-write handling and keep refused or expired work from becoming an unbounded background queue.

**Boundary:** A rate limit is not an idempotency mechanism. A timeout does not prove cancellation or a failed write.

**Technical depth:** Include retries at every layer in the total load budget; nested client/service/database attempts can multiply demand.

**Check:** A payment response was lost. Is immediately repeating the write always safe?

**Explained answer:** No. The first write may have committed. Correct duplicate handling and operation semantics are needed, not just another request.


## 27 · Measure useful service, not how much work you accepted

[Open this lesson](https://ratelimitly.com/?lesson=value#architecture-graphic) · 1:52 at 1×

Return to the people using the service. Maya needs a report. A viewer wants something useful to watch. An operations team wants a system it can recover without repeatedly losing every application copy. The business value is useful service delivered, not the largest possible count of accepted requests.

Local resource protection preserves the application's ability to respond. Tenant fairness contains one customer's excess. Shared latency feedback helps avoid adding work to a dependency that is already slow. A cheap, correct fallback can keep a customer moving even when optional features are reduced. Refused or failed requests must still be counted honestly.

These protections do not absorb unlimited traffic. Edge protection means filtering traffic before it reaches the application. A distributed denial-of-service attack is a flood intended to overwhelm availability; specialized network protection helps with that scale. A web application firewall filters configured categories of unwanted web requests. Those defenses complement application-aware decisions.

Measure useful responses, delays, explicit refusals, degraded responses, and failures separately. No layer promises total protection. The goal is to preserve more useful, correct service while containing overload. Next, choose one important customer operation. Set its protections and safe fallback, then test both refusal and recovery.

### Diagram walkthrough

- **Useful customer outcomes are the goal.** The goal is a useful result for a person, not a large count of accepted requests or internally completed operations.

- **Three protections contribute different kinds of value.** Measure correct full responses, useful degraded responses, refusals, and failures separately. A cheap fallback can preserve value without pretending every request succeeded.

- **Outer defenses address other threats and traffic scales.** Edge filtering and large-scale network defenses complement application-aware admission. No layer can absorb unlimited traffic.

- **Count the outcomes customers actually experience.** Choose one important customer operation. Identify local bounds, tenant policy, and the shared dependency. Decide what can safely degrade; test refusal and recovery.

### Apply it

Define customer-success metrics and distinguish normal success, safe degraded success, policy refusal, transport error, timeout, and resource failure.

**Boundary:** No layer guarantees total protection. DDoS/WAF defenses and application controls solve complementary problems.

**Technical depth:** Compare useful throughput and recovery behavior under realistic mixed workloads. Treat marketing performance numbers as claims requiring workload-specific evidence.

**Check:** Accepted requests increased, but useful responses fell. Did availability improve?

**Explained answer:** Not for customers who cannot finish their task. Evaluate useful completed outcomes, latency, and deliberate service reductions separately.