Most agent frameworks answer one question well: how fast can you build the thing. NVIDIA's NeMo Agent Toolkit — shipped as the nvidia-nat package, latest release 1.4.0 — is interesting because it answers the one almost nobody else touches — what does the thing cost to run once a thousand people use it at once. And the way it answers is the tell. It doesn't hand you a formula. It hands you a load test.

The tool: nat sizing calc#

The toolkit is not another orchestrator. It's framework-agnostic — it wraps agents you already built in LangChain, LlamaIndex, CrewAI or Semantic Kernel and adds a measurement and optimization layer on top. Two pieces of that layer matter here.

The profiler records usage on a per-invocation basis: tokens consumed, time between calls, number of LLM calls. It stores those for offline analysis, computes workflow-level latency and throughput, and reports them with 90%, 95% and 99% confidence intervals. It draws a Gantt chart of every LLM and tool span so you can see which segment is eating the wall clock. In NVIDIA's own worked example, that turned out to be response generation — the longest LLM segment ran roughly 61 seconds while HTTP overhead sat under two.

The sizing calculator sits on top of the profiler and does the thing the name promises. You run nat sizing calc across a sweep of concurrency levels — the docs use 1 to 32 simultaneous users, several passes each — and it captures how the workflow behaves as load climbs. Then, offline, you give it a target: N concurrent users at a target workflow runtime. The docs' example asks for 25 users at a 50-second SLO. Out comes an estimate of the GPU cluster you need.

That's a load test that emits a bill of materials.

Why it has to be a load test#

Here is the non-obvious part, and it's the whole reason this tool exists in this shape.

If you're serving a single LLM endpoint, capacity planning is arithmetic. Decode is memory-bandwidth bound, the KV cache caps concurrency, and you can work the numbers out on paper before you rent a single GPU. One request per turn, a token count bounded by the context window, a latency you can decompose into time-to-first-token and time-per-output-token.

A multi-agent workflow breaks every one of those assumptions. A single user turn is not one request — it's a planner call, then some variable number of tool calls, then maybe a reflection pass, then a retry when a tool returns garbage. The token count isn't bounded by the context window; it's an emergent property of the trajectory, and the trajectory length depends on the input. Latency isn't TTFT plus TPOT; it's the sum of however many sequential agent hops this particular question happened to require, plus the I/O of whatever tools got called along the way.

You cannot compute the cost of a behavior you haven't observed. You can only measure it and extrapolate.

None of that is knowable from a config file. You cannot sum per-call numbers into a workflow cost, because the number of calls is itself a random variable set by the model's choices. So the honest options are two: measure it under representative load, or guess. NVIDIA built the tool that measures.

The uncomfortable corollary#

Once you accept that an agent's footprint is emergent, prompt design stops being a quality decision and becomes a capacity-planning decision.

Add one self-reflection step to your loop. On a single-LLM mental model, that's "a few more tokens." On the trajectory model, it's another sequential hop on every request, which lengthens the critical path, which shifts your entire latency-versus-concurrency curve, which changes the cluster size the calculator spits out. The same is true of a retry policy, an extra sub-agent, a more permissive tool budget. Each one is a slider that moves your GPU count, and none of them announces itself as such in code review. This is the mechanism behind why agent costs scale worse than you expect — the fan-out is invisible until you profile it.

The sizing calculator makes that visible in the only currency that ends an argument: rack units. Change the prompt, re-profile, watch the number move. It turns "should we add a critic agent" from a taste debate into a before-and-after on the cluster estimate.

What to actually do with it#

Don't treat the output as a precise procurement order. The estimate is only as representative as the traffic you replayed through it, and agent workloads are notoriously bimodal — a few pathological trajectories dominate the tail. Profile with inputs that include your ugly cases, not just the happy path, or the calculator will confidently under-size you.

But use it as a differential instrument and it earns its place. Baseline your current workflow. Then, every time you're tempted to bolt on another reasoning step, run the sweep again and read the delta. The absolute GPU number will drift; the direction and magnitude of each change won't lie.

The broader signal is what NVIDIA is implicitly conceding by shipping this. If the cost of an agent could be calculated, they'd have shipped a calculator. They shipped a load harness instead — an admission that in agentic systems, the only reliable way to know what something costs is to run it and watch. The spreadsheet era of capacity planning ended the moment the model started deciding how many times to call itself.