Every Inference & Gateways comparison and buyer's guide for building AI agents — 80 pieces and counting. Each is a head-to-head or a “best X for Y” roundup with a sources-backed verdict.
Setting temperature to 0 doesn't make an LLM deterministic. The real culprit isn't sampling or 'random' GPU math — it's that your request's output depends on who else is in the batch.
A frontier mixture-of-experts model has 671B weights but touches only ~37B per token. That gap is why you can serve DeepSeek-scale models on a single consumer GPU — if you split by tensor role, not by layer.
The neocloud pitch started as 'cheap raw GPUs vs AWS.' In 2026 the scarce input isn't price — it's powered, networked racks — and the category has quietly split into two businesses that barely compete.
The people who build vLLM raised $150M and became a company. The money isn't the story — who now sets the roadmap of an engine half the industry serves on is.
Per chip, Google's Ironwood and Nvidia's B200 are now within ten percent of each other on every number that used to decide this. The real fork is the interconnect — and vLLM just deleted the reason you couldn't cross it.
Hugging Face's TGI went read-only in March. The way it wound down — not the fact that it did — tells you where model serving actually settled.
SGLang v0.5.14 reports 5x throughput serving DeepSeek-V4 on GB300. The lever isn't Blackwell Ultra — it's a per-batch linear program that reroutes tokens across expert replicas. Static replication plans for the average; no batch looks like the average.
The load balancer you already trust is the wrong tool for a fleet of inference servers. Spreading requests evenly is exactly what destroys the cache that sets your latency and your bill.
OpenAI's first silicon claims roughly 50% cheaper inference than Nvidia. The number is self-reported and unverifiable — but the vertical-integration bet underneath it is the part actually worth understanding.
The per-token dashboard is lying to you. An agent's cost lives in the trajectory, not the request — and the only number that aligns finance with engineering is dollars per resolved task.
Token prices are falling and agent bills are still exploding. The reason isn't the model getting pricier — it's that an agent re-pays for its entire history at every step, so cost scales with the square of the loop, not its length.
An 11.7k-star, Rust-based open-source LLMOps stack archived itself on June 12 — not because it ran out of adoption or cash, but because the wedge it was built on is closing from both ends.
The 'AI gateway' stopped being a cost-tracking load balancer and turned into the policy layer for autonomous agents — and that shift is why the newcomers are all written in Go and Rust, benchmarking themselves against LiteLLM.
The headline in SGLang's June release isn't a speed number — it's a deprecation. Speculative decoding stopped being an expert knob and became the default path, and the old one is on the way out.
Sonnet 5's rate card matches Sonnet 4.6's — $3/$15 per million tokens. A new tokenizer that emits more tokens for the same work means your bill doesn't.
With TGI archived and Hugging Face pointing everyone at vLLM and SGLang, the open-source serving field narrowed to three real choices. They hit nearly the same throughput ceiling from opposite directions — so speed is not the thing you're actually picking.
A router picks a model before it sees the answer; a cascade tries the cheap one first and escalates only if a judge says so — and that judge, not the models, decides whether you actually save.
Most 'agent budgets' are alerts wearing a brake's uniform: they tell you after the money is gone. Real enforcement is a prediction problem, because the cost of the next step is a bound you can only ever estimate — never a number you can look up.
Every provider now sells the same deal — hand over your requests, wait up to 24 hours, pay half. The savings are real, but the reason to reach for batch is the thing nobody puts on the pricing page.
The invoice arrives and no one can say which customer spent the money. Cost attribution isn't a report you run later — it's a schema decision you make at request time, and for agents the gateway total lies about where the spend went.
A cut-off completion isn't an error your code catches — it's a 200 OK whose only tell is a stop-reason field most callers never read. And on reasoning models, the fix everyone reaches for can hand you an empty response.
A web server drains its in-flight requests in 30 seconds and restarts. An agent's in-flight request is a multi-hour, side-effecting loop — so graceful shutdown stops being a deploy setting and becomes an architecture decision you had to make weeks earlier.
Round-robin is the wrong way to route an LLM request. Kubernetes now has a GA'd standard that lets the gateway pick a model server by live KV-cache pressure and queue depth instead — and it changes what a load balancer is.
The Kubernetes-native serving stack got very good at spreading a model across a cluster. But in 2026 your GPUs aren't in one cluster — they're scattered across clouds by price and availability, and that's a different problem.
Interruptible GPUs scare people because of training horror stories. For stateless inference the math inverts — there's nothing to checkpoint, so the only real tax is cold start.
Swapping LLM providers in one line is true for a chatbot and a lie for an agent. The cage is one layer up, in tool-calling behavior — and no gateway unlocks it for you.
The metric you'd reach for first — CPU, then GPU utilization — is the one that lies. A 70B pod can read 5% CPU and a calm GPU dial while its request queue backs up for miles. Scale on queue depth instead.
Mozilla's any-llm and LiteLLM get pitted against each other constantly, but they answer different questions — the only one that matters is whether you actually need a proxy.
Every other latency fix speeds up the typical request. Hedging is the only one that attacks the slow tail — by firing a duplicate after your p95 and taking whichever finishes first.
The SDK's 10-minute default times out one call; an agent makes dozens. You need a deadline the whole loop shares — and cancelling to enforce it still costs tokens and can corrupt state.
For an app built on a hosted LLM API, the wall you hit under load isn't the model's speed — it's the provider's rate limiter and your own retry policy. Test for the ceiling and the fall, not the throughput.
The two halves of every LLM request fight each other on the same GPU. Disaggregated serving splits them onto separate hardware — and the win is real, but only past a certain scale.
Trainium2 and Inferentia2 sell real price-performance and AWS capacity. NVIDIA sells CUDA. The decision is whether the Neuron SDK supports your model and serving stack — and how much engineering you'll spend finding out.
Three of these throw tokens away to save memory. One keeps them all and just reads less — and for a long-running agent that revisits its own past, that difference is the whole game.
One of these isn't an inference engine at all — it's a wrapper around the other two. Sorting that out is the whole decision, and it just got simpler because one contender quietly left the race.
Decode is memory-bandwidth bound, so a GPU's TFLOPs barely predict serving capacity. What caps concurrency is the KV cache. Here's the actual arithmetic, with a worked example.
The provider's per-user field won't give you an invoice, and raw token counts lie. The honest unit of attribution is the priced token — after caching, batching, and hidden thinking.
One speeds up the attention math; the other stops your KV cache from wasting most of the GPU. You run both — and the friction where they meet is the actual story.
A trillion-parameter MoE only fires a fraction of itself per token. Expert parallelism scatters those experts across dozens of GPUs — but the hard part was never the split. It's the all-to-all traffic and the hot experts, and they only pay off when you're drowning in load.
The textbook breaker opens when calls start failing. The incident that actually bankrupts an agent is a loop where every call succeeds — so you need a second breaker that watches money, not errors.
It isn't a FLOPS race. Decode is memory-bound, and the MI300X's 192 GB lets a model live on fewer GPUs than an 80 GB H100 can. The catch was never the silicon — it was ROCm. Here's where that tax stands in 2026.
The cost of scaling a self-hosted model to zero isn't compute or container boot — it's the seconds-to-minutes of shoving tens of gigabytes of weights into empty GPU memory. That's the number that decides warm-vs-zero.
When an orchestrator spawns twenty sub-agents that each retry on 429, the retries compound into a self-inflicted DDoS. The fix is upstream flow control, not smarter backoff.
They get filed as rivals because both promise "one API for every model." But one is a hosted marketplace you buy from, the other is infrastructure you run — and the smart move is often to use both.
Your engine computes a KV cache, uses it once, and throws it away. Offloading turns that scratchpad into a shared storage tier — and changes the question you should be asking.
The B200's headline 5-6x throughput jump is two different upgrades wearing one number — bigger HBM and FP4 compute — and which one matters depends entirely on whether your workload is memory-bound or compute-bound.
Is it cheaper to run an open model on your own GPUs than to call an API? The deciding number isn't the token price — it's how busy the GPU stays.
The cheaper-model reflex is the wrong first move. An agent's bill is dominated by the transcript it re-sends on every step — so the money is in the context, not the price card.
Buying a faster model is the reflex, and usually the wrong first move. An agent's wait is a chain of serial round-trips — so the latency is in the loop, not the tokens-per-second.
Every agent in production eventually meets a 429. The naive fix — just retry — is also the most expensive bug in modern LLM apps. Here's the layered pattern that survives the limit instead of paying triple for it.
A 429 means wait; a 400 means stop; a 200 from your backup model can be the most dangerous answer of all. The reliability layer every agent needs and most skip.
Greedy decoding should give the same answer every time. It doesn't — and the usual 'floating-point' excuse is wrong. The real culprit is what else is in the batch with you.
Three libraries everyone compares as if you get to choose. You don't — your model already chose for you. The real question is what that choice costs, and who pays it.
Three of these knobs do the same job — truncate the unreliable tail of the next-token distribution. The differences are smaller, and more contested, than the tutorials admit. And if you build agents, you probably want almost none of it.
If your agent has a fixed set of tools and intents, you probably don't need a model to pick between them. An embedding lookup is faster, cheaper, and the same input lands the same way every time.
Stretching a model past its trained context length isn't a memory problem — it's a positional-encoding generalization problem. The methods that work all interpolate instead of extrapolate, and the good ones interpolate unevenly.
"Dynamo vs vLLM" is a category error. One is an orchestrator across pools of GPUs; the other is the engine inside a single replica. Sort that out and the real choice gets clear.
Three ways to put more than one workload on one accelerator — and a reason most LLM serving shouldn't use any of them. Choose by failure domain, not utilization.
Every attention variant since 2019 has been one argument about the same scarce resource — the key-value cache — and the newest answer changes the terms of the deal.
Pure Mamba never beat the Transformer outright — but a wave of hybrids that keep ~8% of layers as attention now cut long-context memory 70%+ and triple decode throughput.
The three numbers everyone quotes measure three different bottlenecks — and per-user speed and system throughput move in opposite directions, so a vendor's headline tok/s can mean whatever flatters it.
Stop choosing between them. FlashAttention is the compute kernel, PagedAttention is the memory layout, FlashInfer is the engine — a modern stack runs all three at once.
Diffusion language models generate every token at once instead of left-to-right, which sounds like a guaranteed speedup. The early open models were slower than the autoregressive baseline anyway — and the reason they finally got fast is the opposite of what the pitch implied.
Static batching wastes the GPU because LLM outputs are variable-length — short replies idle while the batch waits for the longest. Continuous batching schedules at every token step instead. The catch is that the same trick that wins throughput can spike latency.
Three startups built custom silicon to outrun the GPU on token generation. The speed is real, the SRAM is tiny, and that tradeoff decides everything.
When one model won't fit on one GPU, you have two ways to cut it up — and the right cut is a description of your interconnect, not a tuning knob you guess at.
A single tokens-per-second number hides two workloads pulling in opposite directions — and the whole arc of serving optimization is the field admitting they should never share a GPU.
Ollama just ripped out llama.cpp and bolted in Apple's MLX on the Mac. The switch is a tell about where your bottleneck actually lives — and when the older engine still wins.
Every major provider sells inference at roughly half price if you can wait up to 24 hours. The discount isn't the point — the contract is, and it tells you which agent work was never realtime to begin with.
You quantized the weights to 4-bit and thought memory was solved. At long context the KV cache dwarfs the weights — and it needs a different kind of quantization to shrink safely.
The weights are the easy part — the math you can do on a napkin. What silently OOMs your server in production is the KV cache, and almost nobody budgets for it.
Three engines, one job: turn a model into a high-throughput endpoint. The feature gaps are closing — what's left is portability, vendor lock-in, and which project is still being built.
Speculative decoding makes a single LLM response 2–6x faster without changing a token of the output. The reason it works — and why the newest method wins — is a fact about your GPU, not your model.
Buyers shop for these cards by peak FLOPS. Token generation barely uses them. The spec that actually moves inference throughput is the one most spec sheets bury — and a single NVIDIA card proves it.
Three ways to put a model behind an endpoint — and they increasingly run the same engine underneath, so the thing you are actually choosing is not speed.
They all wrap roughly the same inference engine, so they all run the same model at roughly the same speed. The thing that actually separates them is what shape they want to be — a daemon, a polished app, or an open one.
Three ways to rent open-weight inference without owning a GPU — and why the fastest of them just licensed its speed to Nvidia instead of competing with it.
Per-prompt model routing promises GPT-quality answers at a fraction of the bill. The honest 2026 answer is that it's a cost lever with a threshold, not a free one — and a neutral benchmark disagrees with the marketing.
Every agent ends up talking to more than one model provider. The library you put in the middle decides whether that seam stays a proxy or quietly becomes your control plane.
The benchmark everyone argues over is the wrong one. The engine you should run is decided by how much context your requests share — not by whose tokens-per-second screenshot is biggest.