Every rate limiter you've ever built has an implicit subject: the user. Requests per second, per user. Requests per minute, per API key. The mental model is a person on the other end, clicking at human speed, and the limiter's job is to keep any one of them from ruining it for the rest. Agents quietly invalidate that model, and they do it in a single move.
Give one user one agent, and that user's single request becomes a burst of tool calls — a web search, then six follow-up fetches, then a dozen parallel database reads, then an LLM call to summarize each result. Hundreds of outbound calls, often concurrent, all traceable to one human who pressed "go" once. A per-user request counter looks at this and reports one active user. Meanwhile the API on the receiving end is being saturated by that one user. The load an agent generates has come unglued from the number of users generating it, and the limiter is measuring the wrong thing.
The ceiling that binds is not yours#
Here's the reframe that changes the design. When you throttle a normal web service, you're protecting your own capacity from your own users. When you throttle an agent, the thing you're usually protecting against is a limit you don't own. Your agent calls an LLM provider, a search API, a SaaS endpoint — and each of those enforces its own quota and hands you a 429 the instant you cross it.
So the limiter's real job flips. It isn't defending your service from inbound traffic; it's shaping your outbound traffic to stay under someone else's ceiling. That ceiling is shared across everything you run: every concurrent agent, every user, every background job hitting the same provider draws from one quota. A per-user limiter can't express that — it partitions the budget by the wrong dimension. What you need is a single shared egress budget for each downstream, with fair-share allocation across runs on top.
And for the LLM calls specifically, the unit isn't the request. Anthropic's Messages API meters requests per minute and input tokens per minute (ITPM) and output tokens per minute (OTPM) — three independent counters that do not share a budget, each with its own 429 and retry-after. OpenAI does the same with RPM and TPM. A request-counting bucket will happily wave through a handful of enormous calls that blow the token quota while the request count still looks healthy. The bucket has to be sized in estimated tokens: debit the input side before dispatch, reserve against the output side for the largest plausible response, and settle up when the real usage comes back on the response.
Rate is the wrong axis; concurrency is the right one#
The classic answer to bursty traffic is the token bucket — a bucket that refills at a steady rate and lets an idle client spend the whole thing at once. It's what Stripe and AWS use, because real traffic is bursty and a token bucket absorbs bursts gracefully. Keep it. But for agents it's not sufficient, because a token bucket controls the average rate and says nothing about how many calls are in flight at once.
That distinction is the whole game for agent tool calls, which are long-lived and heavy — a five-second search, a thirty-second model call. Under a pure rate limiter, a burst of these can all be admitted within the rate and then pile up concurrently, and it's the concurrency, not the rate, that takes the downstream down. Stripe's own post is explicit that they run three separate limiters: a request-rate limiter, a concurrency limiter that caps simultaneous in-progress requests, and a load shedder for when things go wrong anyway. Agents need the middle one most. (The GCRA variant of the token bucket, which tracks a theoretical arrival time instead of physically counting tokens, is a cheap way to run these limiters in a distributed setup without a hot shared counter.)
A token bucket answers "how fast." A concurrency limiter answers "how many at once." For agents, the second question is the one that pages you.
The limiter belongs to the run#
Put it together and the right granularity falls out. Not per user — that misses the runaway agent. Not per process — a single global limiter starves parallel runs unfairly and turns throttling into a lottery. The unit is the run. Give each downstream a shared quota bucket sized to the provider's real ceiling (in tokens where the provider meters tokens), and layer a per-run concurrency cap on top so one runaway agent can't drain the whole budget and starve its siblings. Fair-share the shared bucket across active runs; cap each run's in-flight calls locally.
And when you do hit the ceiling, do the agent-native thing. A 429 carries a retry-after and reset-timestamp headers that tell you exactly when the specific limit you tripped refills — honor them instead of guessing with blind exponential backoff (worth pairing with a retry budget so a thundering herd of retries doesn't deepen the outage). Better still, apply the throttle as backpressure at the tool-dispatch layer: queue the outbound calls and let the agent keep reasoning about what it already has, so a rate limit slows the run instead of freezing it. The user pressed go once. The least you can do is not make them watch it hang.
