Three Memory Architectures for Production AI Agents (And When Each Fails)

The three architectures and what they actually are

Before getting into failure modes, here is what we mean by each term, because the industry uses these words loosely:

In-context memory means putting everything the agent needs to know directly into the prompt. Previous messages, prior decisions, user preferences — all of it in the context window. The agent can "remember" because everything is visible right now.

Vector memory means storing past interactions or knowledge as embeddings in a vector database, then retrieving the most semantically similar chunks when the agent needs them. The classic RAG pattern applied to episodic memory rather than documents.

Structured fact memory means extracting specific, typed facts from interactions and storing them in a relational or key-value store. Not "here is everything we discussed" but "supplier_preferred_payment_terms = 60_days" and "user_approval_threshold = 50000_USD".

In-context memory: the approach that destroys itself at scale

We shipped a procurement agent in May 2024 that used pure in-context memory. Every prior message in the conversation was included in the next call. Simple to implement. Works perfectly for the first four or five turns.

By turn fifteen, context was eating 40,000 tokens on every call. By turn thirty, we were hitting context limits on some suppliers with verbose pricing schedules. The cost per session climbed to roughly $0.90 in API spend for a 50-turn conversation. That is fine for a demo. It is not fine for an operation running 2,000 procurement sessions per month.

We tried summarizing old context to trim it. That introduced the first hallucinations. The summarizer was a separate LLM call that occasionally got details slightly wrong — a price quoted as $4,200 would become "approximately $4,200" in the summary, then get quoted as $4,250 in a later turn when the model rounded differently. Small errors. But in procurement, small errors in quoted prices are not acceptable.

In-context memory has one legitimate use: short, bounded tasks where the conversation will not exceed 15–20 turns and where the cost per session is not a scaling concern. Everything else is scope creep.

The deeper problem is that in-context memory treats all information as equally important. The model's decision about which parts of a 50-turn conversation are relevant to the current question is uncontrolled. You have no way to say "the approved budget from turn 3 is more important than the small talk in turn 24." Everything is in there and the model weighs it as it sees fit.

Vector memory: the approach that retrieves the wrong thing confidently

Version two used a vector database. After each conversation turn we embedded the user's message and the agent response and stored them. When a new query came in, we retrieved the top-5 most semantically similar past interactions and injected them into context.

This worked well in single-user testing. It broke badly in multi-user production.

The retrieval does not know why two messages are semantically similar. When a new procurement manager asked "what is our standard payment term for construction suppliers?", the vector search returned three conversations from a different user who had been discussing payment terms for IT equipment. The cosine similarity between those conversations was high. The actual answer was completely different. The agent confidently synthesized the wrong information from the right-sounding context.

We spent three weeks adding scope filtering (org ID, user ID, agent type all required to match), recency decay (interactions older than 90 days weighted at 30%), and permission checks (only retrieve from users with equal or higher permission level than the current user). After those fixes, hallucination rate dropped significantly.

Vector memory is genuinely useful for document retrieval and for surfacing similar past decisions. It is a poor architecture for factual state — things that are simply true about an entity right now. If you need to know that a supplier's lead time is currently 45 days, you do not want that retrieved probabilistically from a semantic search. You want a lookup.

Structured fact memory: the approach that actually scales

Version three runs a fact extraction step after every task completion. The LLM gets the full conversation summary and outputs a structured JSON object of things it learned: entity facts (supplier attributes, user preferences, org policies), confirmed decisions, and corrections the human made to prior outputs.

These facts get stored in a typed key-value store keyed by (org_id, entity_type, entity_id, fact_key). When a new conversation starts, we do a deterministic lookup — no embeddings, no retrieval uncertainty — and inject the relevant facts into the system prompt as a structured block.

This architecture has three properties that make it production-stable:

• →Deterministic injection. You know exactly what the agent knows. It is not a function of which previous conversations happened to score highest on cosine similarity today.
• →Human correction propagates. When a user corrects the agent ("no, the lead time is actually 30 days, not 45"), that correction gets extracted as a fact override and changes all future behavior for that entity.
• →Auditable. Every injected fact has a source (which conversation, which user, when) and a confidence score. You can explain exactly why the agent made a specific claim.

The failure mode of structured memory (yes, it has one)

Structured memory assumes the fact extraction step works correctly. It mostly does. But the LLM doing extraction occasionally misattributes a fact — extracting a statement the user made hypothetically ("what if we extended terms to 90 days?") as a confirmed policy.

We fixed this with an extraction confidence threshold (extractions that the model scores below 0.75 confidence go into a review queue rather than directly into the fact store) and a human confirmation layer for facts that affect financial approvals.

The other failure mode: fact stores get stale. A policy that was true six months ago may have changed. We run a staleness check — any fact not confirmed in the last 90 days gets a flag, and the agent adds a caveat when using it. Overkill for some domains, essential for anything in a regulatory or financial workflow.

The practical architecture we use now

The right answer for enterprise agents is not one of these three in isolation. It is a layered system:

• 1.Current session context — last 8 turns maximum, recency-trimmed, never summarized.
• 2.Structured facts — deterministic injection of typed, confidence-scored entity facts. This is the stable memory layer.
• 3.Vector retrieval — used only for document search and for surfacing similar past decisions as context ("here is how we handled a similar supplier situation"), never as authoritative fact.

Layers one and two run on every call. Layer three runs only when the query explicitly involves historical precedent or document lookup. Average context window per call: 6,000–8,000 tokens, down from the 40,000+ we hit with pure in-context memory.

What this means for your architecture decisions

If you are building an agent that will run thousands of sessions per month on business-critical workflows, the memory architecture decision matters more than the model choice. Claude 3.5 Sonnet with a bad memory architecture will underperform GPT-4o mini with a good one on any task that requires recalling specific facts about entities.

The three questions to answer before you commit to an architecture:

1. 1.How many turns do your sessions typically run? Under 15, in-context is probably fine. Over 30, you need something else.
2. 2.Does the agent need to remember specific facts about entities (suppliers, customers, policies) across sessions? If yes, you need a structured store.
3. 3.Does the agent need to find relevant precedents or similar past decisions? If yes, vector retrieval is appropriate — but scoped and filtered.

We got this wrong twice before we got it right. The mistakes were not expensive in terms of money lost — they were expensive in terms of time spent debugging behavior that looked like model quality problems but were actually architecture problems. Build the memory layer intentionally.