There are now two ways to say "your agent has too many tools," and the industry has only solved one of them.
The solved half is loading. Connect an agent to a dozen MCP servers and their tool definitions eat the context window before the first user message is read — a typical GitHub/Slack/Sentry/Grafana setup runs tens of thousands of tokens up front, and selection accuracy falls off a cliff once more than a few dozen tools are visible at once. The fixes are mature and shipping: defer the schemas and load the three-to-five a request actually needs. Anthropic's tool-search and code-execution patterns do this at the schema layer; retrieval systems do it with embeddings. We covered that front half in June, and ScaleMCP pushed it to the extreme — an agentic-RAG retriever indexing 5,000 MCP servers and binding only the matches for the current turn.
Here is the half nobody optimized. Retrieval adds. Almost nothing removes. In a single-shot request that's fine — the context resets next call. But a real agent runs a conversation, and in a conversation the tools you retrieved on turn 3 are still bound on turn 40, alongside everything turns 4 through 39 pulled in. Loading-side cleverness gets you a small starting context; without an eviction policy, it creeps right back up. The scarce operation isn't finding the next tool. It's forgetting the last twenty.
MemTool measures the forgetting#
MemTool is the first framework I've seen that benchmarks eviction head-on. It treats an agent's bound tool set as short-term memory — something to be actively managed across turns, not a monotonically growing pile — and it measures tool-removal efficiency: of the tools that should have been dropped by now, how many actually were. The evaluation is deliberately punishing: 13-plus LLMs, driven across 100 consecutive user interactions, each turn adding and retiring tool needs, over the ScaleMCP suite of 5,000 servers. It's the multi-turn stress test the loading-side papers never ran.
MemTool ships three architectures for who does the removing, and the distinction is the whole point:
- Autonomous Agent mode hands the model full control. Alongside its task tools, it gets tool-management tools, and it's on the model to notice "I'm done with the Jira set" and call the function that unbinds them.
- Workflow mode takes the model out of it. A deterministic control layer decides what to evict on a fixed policy — no autonomy, no judgment call.
- Hybrid mode runs both: the model prunes what it can, and a deterministic backstop catches what it misses.
The capability cliff#
The result that should change how you build: in Autonomous mode, reasoning models reach 90-94% tool-removal efficiency (on a three-window rolling average) while medium-sized models land between 0% and 60%. Same mode, same benchmark — the only variable is whether the model reasons. Self-pruning your own toolset turns out to be a planning skill, not a lookup, and it's gated on exactly the capability that separates a reasoning model from a cheaper one.
Retrieval is a lookup any model can do. Deciding which of your loaded tools are now dead weight is a judgment about the whole conversation — and the benchmark says only reasoning models reliably make it.
That asymmetry is easy to miss because loading-side optimization doesn't have it. Embedding a query and pulling the top-k servers works fine on a small model; retrieval quality barely tracks reasoning. So teams reasonably assume tool management in general is model-agnostic. MemTool's finding is that the moment you ask the model to manage its own context — to look at what it's holding and decide what to put down — you've quietly added a reasoning dependency that your model tier may not clear.
The other half of the result is the reassuring half. Workflow and Hybrid modes remove tools reliably across every model class, because the eviction logic no longer depends on the model choosing to run it. And on task completion — did the agent still get the job done — Autonomous and Hybrid come out ahead, since fully-deterministic Workflow eviction can prune something the model wasn't finished with. So the trade is legible: determinism buys you guaranteed cleanup, autonomy buys you task performance, and Hybrid is the mode that refuses to choose.
The rule this gives you#
Pick your tool-management architecture by your model's class, not by taste:
- Running a frontier reasoning model? Autonomous mode is viable — you'll get near-total cleanup with minimal orchestration, and you keep the task-completion edge.
- Running a cost-tier or non-reasoning model? Do not let the agent manage its own tools. That 0-60% range is a context budget quietly leaking until a long session degrades or overflows. Use Workflow or Hybrid so deterministic code owns eviction.
- **Want both — reliable cleanup and task performance, without betting on the model's judgment?** Hybrid. It's the safe default precisely because it doesn't assume the model will prune, but lets it help when it can.
And keep both halves of the problem in view. Schema deferral and retrieval keep your first context small; dynamic removal keeps it small on turn 100. A long-running agent that only optimizes loading has fixed the leak at the faucet and left the drain plugged. The interesting frontier in "too many tools" has quietly moved from which tool do I load next to which of these am I finally allowed to forget — and it turns out not every model is smart enough to answer.
