Before You Give an AI Agent Memory, Fix What It Retrieves

An agent that cites an obsolete policy in a customer response does not necessarily need a better memory. It may need a current index, a narrower search tool, or a way to inspect the passage it found. Add persistent memory before diagnosing those failures and you risk making a bad answer available for reuse across sessions.

The failure that looks like missing memory

Suppose an agent drafts a response using policy documents and the customer’s current CRM status. It cites a policy that was superseded and misses a condition in the current version. The team’s first instinct might be to give it long-term memory so it can retain corrections. That treats the symptom, not the failure.

There are at least three possibilities. The current policy never reached the index. It reached the index but the search did not return it. Or it was returned alongside so much other material that the agent failed to use it. LangChain separates inefficient searching, oversized results, and irrelevant or stale context because each calls for a different fix. Re-indexing will not improve a poorly scoped query; a better query will not repair stale source data.

Saving a corrected answer as memory can make the incident harder to investigate. On the next run, did the agent use the current policy, or repeat an earlier answer? If the policy changes again, which copy wins? Keep the correction attached to its authoritative source, and make the retrieval path observable. Memory has a place, but it should not become a second, agent-writable policy repository.

That distinction matters most when the agent works across systems. A document search index can answer what a policy says; it may not be the right place to determine a customer’s current account status. Before changing the agent’s memory architecture, list the facts the workflow needs and identify where each fact should come from.

Give each fact a source of truth and a freshness requirement

For a customer-response workflow, separate policy text, operational state, and user preferences. A versioned policy document may be authoritative for the rule. The CRM may be authoritative for the customer’s current status. A user-approved preference about response format may legitimately persist between runs. These facts have different owners, update paths, and acceptable ages.

Write those requirements down for one consequential workflow. For each required fact, record its source, the actor allowed to read it, how old it may be, and what the agent must do when it cannot verify it. If the task requires current operational state, a document index must not silently stand in for a live business record. A live query can reduce staleness, but it also brings latency, rate limits, and integration work. Indexed retrieval is a sound choice when measured update and deletion lag fits the workflow.

Index freshness is more than an ingestion schedule. Test edits, superseded versions, deletions, and permission changes. Azure AI Search indexers use source change detection for incremental updates, but Microsoft warns that resetting and rerunning an indexer does not remove every orphaned document. Deletion needs its own handling. An index that updates new text while retaining a deleted policy is not current enough for an agent that presents it as evidence.

Access control has a similar synchronization problem. Azure AI Search describes document-level security filtering using caller identity, while its native token-based document-permission approaches are marked preview and depend on permission metadata in the index. A permission revoked in the source is not necessarily reflected before synchronization. Define the acceptable window and the fallback behavior; do not assume a successful search result proves the caller is entitled to see it. For a fuller treatment of authorization at the tool boundary, see why retrieval is not permission to act.

Do not replace these checks with a newest-first sort. Publication date, effective date, source-system update time, and index time answer different questions. The authoritative rule might be in an older document still in force. Azure’s freshness-aware retrieval is a preview feature using the 2026-08-01-preview REST API; its freshness policy biases ranking rather than enforcing a date cutoff. If a workflow requires a date range, use a filter. Test conflicting versions instead of assuming recent means applicable.

Design search tools that return evidence, not exports

Give the agent tools that match the lookup it needs. For a known customer ID or document filename, use structured or exact lookup. For a policy question phrased differently from the source text, use semantic or hybrid search. LangChain’s retrieval example starts with filename and literal-text search, then introduces semantic retrieval where wording is unpredictable. That is a useful design sequence, not a rule that every corpus needs the same search stack.

Each retrieval tool needs a contract. Require an authenticated actor and tenant; constrain the sources the tool can search; expose an effective-date filter where it is meaningful; and bound the number and size of results. Include pagination and a way to open the full passage. Return source ID, title, version or update timestamp, and a location the agent or reviewer can use to verify the claim. A concise preview is useful only if the decisive evidence remains reachable.

Large responses are not harmless just because the model accepts them. Anthropic recommends pagination, range selection, filtering, or truncation for large tool outputs, and contrasts targeted log search with returning every log line. Its concise and detailed response formats let an agent request more when needed. Claude Code’s default restriction of 25,000 tokens per tool response is a product behavior, not a context budget to copy into your system. Likewise, LangChain’s Deep Agents example returns 100 lines by default from read_file and can offload results over about 20,000 tokens to files; those numbers describe that implementation, not an optimum for yours.

Do not optimize result size in isolation. A tight top-k limit can hide the relevant passage on the next page. A terse summary can omit the exception that changes the answer. The right limit is the smallest result set that still supports the task on your evaluation cases. If the agent needs more detail, it should be able to request a specific page or passage rather than repeating a broad search.

Tool choice also affects predictability. LangChain distinguishes two-step RAG from agentic RAG: the former retrieves before generation; the latter lets the agent decide when and how to search, with variable latency as a trade-off. If every customer response must check current policy and CRM status, make those checks mandatory rather than hoping an agent chooses them. Leave discretionary search for genuinely open-ended questions.

Evaluate retrieval before adding another context layer

Build a small test set from failures that would change a business decision: conflicting policy versions, similarly named customers, a revoked document permission, an oversized export, and a relevant passage beyond the first page. Include a source that is unavailable or older than the workflow permits. For each case, specify which evidence is authorized and current, what answer it supports, and when the agent should decline to assert a fact.

Measure separate outcomes. Was the required evidence retrieved? Was stale or unauthorized material returned? Was the answer grounded in the evidence, and did the workflow succeed? Track latency and token use alongside those measures. LangChain’s RAG evaluation tutorial separates answer correctness, grounding, answer relevance, and retrieval relevance. A single answer score cannot tell you whether to fix ingestion, search, or generation.

Compare changes against that baseline. Try narrower tools and better filters before adding a reranker or a subagent to sift results. Extra stages can help, but add calls, cost, latency, and another point where decisive evidence might be discarded. Anthropic reported lower top-20-chunk retrieval failure rates with contextual embeddings plus BM25, and further improvement with reranking, on its tested datasets and configurations. Those results justify testing the approach, not forecasting the same gain for your documents.

Record enough telemetry to distinguish a missing source from a buried result without copying confidential passages into every log. Capture source identifiers, versions, retrieval position, filters, and tool outcomes where appropriate. If engineers cannot inspect customer data directly, the approach in debugging AI agents without customer data is a useful companion to this evaluation work.

Add memory only for a cross-run requirement

Once retrieval works, ask what information the workflow genuinely must carry forward. An unfinished task, a user-approved preference, or a handoff across sessions may need durable storage. Current policy text and CRM status usually need revalidation against their sources instead. Retrieval and memory are complementary, but they should have different jobs.

Make memory scope explicit. Deep Agents distinguishes thread-scoped StateBackend from StoreBackend storage across threads, and describes user-scoped and agent-scoped memory. It also warns that writable memory shared between users creates a prompt-injection risk. For every persisted item, define who can write it, which user or tenant can read it, where it came from, when it expires, how it is deleted, and which authoritative system must be checked before reuse.

Treat retrieved text as evidence, not instructions to update memory. LangChain warns that retrieved material can contain indirect prompt injection and that prompts or delimiters alone do not reliably prevent it. Keep shared policy read-only to the agent; validate claims against sources before using them. If the work spans existing systems, access controls, evaluations, and monitoring, that is the appropriate scope for AI agent development, not simply adding a memory store.

Release checklist for a document-and-systems agent

• Choose one consequential workflow and list every fact it needs, its authoritative system, acceptable age, and whether it must survive a session.

• Test edits, superseded versions, deletions, and revoked permissions. Measure source-to-index lag and define what happens when a source is missing or too old.

• Require actor and tenant scope on retrieval tools. Return bounded, paginated evidence with source and version information, plus a path to the full passage.

• Test conflicting versions, similar names, oversized results, and evidence beyond the first page. Score evidence recall, freshness, authorization, grounded success, latency, and tokens separately.

• Add durable memory only for an identified cross-run need. Set write authority, scope, provenance, expiry, deletion, and revalidation rules before enabling it.