The cheapest large-language-model call is the one you never make. That is the entire pitch for semantic caching: embed the incoming query, run an approximate-nearest-neighbor search over the embeddings of past queries, and if the closest match clears a similarity threshold, return the stored answer without touching the model. For a customer-support chatbot answering "what's your refund window?" for the ten-thousandth time, it works beautifully. Tools like GPTCache made it a one-import feature.

Then people started wiring the same layer into agents, and it started quietly breaking things.

A near-miss stops being a sentence#

The reason is not subtle once you see it, but almost nobody designs for it. In a chatbot, a cache is a retrieval layer: a false hit — a query that clears the threshold but isn't actually the same question — returns a slightly-off paragraph that a human reads, frowns at, and moves past. The failure is legible and cheap.

An agent does not read the cached value. It acts on it. The cached output is a tool call, a function argument, a routing decision, a write to some external system. So a false cache hit is not a wrong sentence; it is a wrong action, executed automatically, with the same confidence as a correct one. "Cancel order #4471" and "cancel order #4417" sit a hair apart in embedding space and a world apart in consequence. The cache that saved you a dollar just refunded the wrong customer.

The benchmark nobody wanted#

For a while this was folklore. In February 2026 it got measured. The paper "Why Agent Caching Fails and How to Fix It" ran the standard approaches against real agent workloads and reported numbers that should stop anyone mid-integration: GPTCache landed at 37.9% accuracy, and adaptive prompt caching at 0–12%. These are not tuning-away-able margins. On agent tasks, the off-the-shelf semantic cache is closer to a coin flip than a component.

The authors' diagnosis is the genuinely non-obvious part, and it inverts the instinct every engineer brings to the problem.

A cache doesn't need to understand the query correctly. It needs to map the same intent to the same key every single time.

We reach for a cache and immediately start optimizing how accurately it classifies a request. Wrong target. What a cache actually requires is key consistency. Consider a method that mislabels every email-related request as, say, "calendar" — but does so identically every time. Its cache hits are perfect: the same intent always lands on the same key, so the stored answer is always the right one for that intent. Now consider a "smarter" method that correctly recognizes email requests but oscillates between two valid labels depending on phrasing. It has higher classification accuracy and a useless cache, because the same intent scatters across multiple keys and never hits. Precision and consistency win; raw understanding is beside the point.

Canonicalize the intent, not the string#

That reframing points straight at the fix: intent canonicalization. Instead of comparing raw query embeddings by cosine distance and praying about the threshold, you collapse the many surface forms of a single intent down to one stable, discrete key — via few-shot classification, or by normalizing the request into a structured canonical form — and cache on that.

The payoff shows up cleanly in an adjacent domain. A companion 2026 paper on caching analytical (OLAP) queries via LLM canonicalization reported an 82% cache-hit rate against 28% for plain text matching and 56% for syntax-tree matching — with zero false hits across nearly 1,400 controlled queries. Zero. That is the number that matters for agents: not how often you hit, but how often you hit wrong. A separate line of work, SemanticALLI, pushes the same idea further — caching the agent's reasoning structure, not just its final response, so that decomposed sub-decisions become reusable keys.

And now it's an attack surface#

There is one more reason to be careful that has nothing to do with hit rates. Because cache keys are derived from embeddings, they can be engineered. The 2026 work "From Similarity to Vulnerability" demonstrates key-collision attacks: an adversary crafts queries deliberately tuned to collide with a victim's cache key, poisoning what gets retrieved or probing what's stored. For a passive chatbot cache that's an annoyance. For an agent that executes cache hits, it's a way to steer behavior through the cache itself. A semantic cache is untrusted-input infrastructure, not a free performance win.

The rule that falls out#

None of this argues against caching. It argues for scoping it like a load-bearing decision instead of a decorator. If you already run a general LLM cache, the mechanics of thresholds and tools still apply — but agents change the acceptance criteria. The line worth writing on the wall:

Cache the deterministic, read-only periphery — reference lookups, documentation, idempotent retrievals — where a stale-but-close answer costs nothing. Never cache a step whose output changes the world. And when you do cache, key on canonicalized intent, not on how close two sentences happen to sound. The chatbot cache optimized for finding a similar question. The agent cache has exactly one job: give the same intent the same key, every time, or stay out of the loop.