Every good talk on agent memory ends at the same cliff. You learn the three tiers — session for this thread, state for scratch key-values, and long-term memory for facts that survive a reset — and then someone asks the only question that costs money: where does tier three actually live? That's where the tidy diagram stops helping, because the answer is a backend decision, and the three real options optimize for different things.
Start with the insight that reframes the whole choice: a vector database is not a memory system. It's the storage layer under one. The genuinely hard part of agent memory is everything around the store — deciding which turns are worth keeping, merging a new fact that supersedes an old one, forgetting what's gone stale, and retrieving the right slice without flooding the window. Call it the invisible 80%. Whichever backend you pick, what you're really choosing is who writes that 80%.
Option 1 — Vertex AI Memory Bank: let Google write it#
If you're already building on Google's Agent Development Kit, Memory Bank is the path of least code. It plugs into ADK's MemoryService interface: short-term session history gets distilled into persistent, semantically-searchable long-term memories, extracted asynchronously so the agent keeps working while memories are generated. You read it back with a PreloadMemoryTool — which pulls relevant memory in at the start of every turn — or a LoadMemoryTool, which the agent calls only when it decides it needs to dig.
The upside is that extraction, semantic retrieval, and the write pipeline are handled. The cost is coupling: tier-three memory now lives in Vertex AI, on Google Cloud. That's a fine trade if you were going to be there anyway, and a real lock-in tax if you weren't.
Option 2 — Mem0: the logic, without the lock-in#
Mem0 is the option most teams should default to, and the reason is portability. It's an open-source memory layer (Apache 2.0) that does the invisible 80% for you — a two-phase pipeline that automatically extracts salient facts from each turn, consolidates redundant or superseded ones, and retrieves with a hybrid of vector similarity, keyword match, and an entity graph for multi-hop queries. It runs across models and clouds, and you can self-host it with unlimited memories or use the managed Mem0 Cloud, which starts free at 10K memories and tiers up (roughly $19/month for vector + key-value, ~$249/month once you want the graph).
A vector DB gives you the shelf. Mem0 gives you the librarian who decides what goes on it, merges the duplicates, and finds the right book without reading you the whole aisle.
On the public memory benchmarks — LoCoMo, LongMemEval — Mem0 reports category-leading scores for its 2026 token-efficient algorithm while keeping retrieval under ~7,000 tokens a call. Those are vendor-reported numbers, so validate them on your own traffic; but the architecture is the point. You get the retrieval logic without betting your data on one cloud.
Option 3 — a plain vector DB: own it on purpose#
Reaching straight for pgvector, Pinecone, or Redis is the right call in exactly two situations: your memory is genuinely simple (store a handful of facts, retrieve by similarity, done), or it's so specialized that a general-purpose extractor would fight you and you'd rather write the rules yourself. The appeal is total control and data ownership — no third party sees your memories, and every knob is yours.
The catch is the one this whole piece is about: a bare vector DB hands you the shelf and none of the librarian. Extraction, dedup, temporal reasoning, forgetting — you build all of it. That's a bigger project than "add Postgres" makes it sound, which is precisely why the industry default has shifted from raw store to memory library over the last year.
The decision#
- Default to Mem0 (or a comparable memory library) when you want the memory logic done but refuse to be locked to one cloud. It's the best fit for the most teams.
- Use Vertex AI Memory Bank when you're already committed to ADK and Vertex — the integration is tight and the code you don't write is real.
- Go straight to a vector DB only when memory is simple, or when you've decided, deliberately, to own the retrieval logic as a core competency rather than inherit it.
The mistake isn't picking wrong among the three. It's not realizing you were choosing at all — wiring up a vector DB, calling it "memory," and discovering three months in that the shelf was the easy part. If you're still upstream of this decision, start with the tiers themselves in three kinds of agent memory, then come back and pick your backend on purpose.
