TL;DR
AI memory architecture determines what persists, how it reaches the model, and whether it is still appropriate to use. Model parameters, request context, external records, and workflow state serve different purposes. A larger context window or a vector database does not establish provenance, current accuracy, or authorization. Design both the write and read paths, preserve the distinction between observations and hypotheses, and test correction and revocation across summaries and caches. The objective is not maximum recall. It is reliable use of information within the right boundaries.
Introduction
An incident assistant identifies that the checkout service is waiting on an unhealthy dependency. Engineers restore the dependency, verify recovery, and close the incident.
The next morning, another timeout report arrives. The assistant confidently names the same dependency as the active failure point. Its explanation is consistent with yesterday’s incident, but not with the current telemetry.
The model may not have changed at all. The application may simply be retrieving a saved summary that says the dependency is unavailable, without preserving when that observation was made or whether it remains applicable.
Part 1, AI Generalization: Did Your Model Learn the Right Pattern?, examined whether learned relationships support useful behavior when evidence changes. Part 2 addresses the information supplied around that model. Even a well-evaluated assistant can be undermined by stale, incorrectly scoped, or poorly governed memory.
We will continue the hypothetical incident-triage scenario. The assistant uses approved runbooks and read-only operational evidence, has no production write access, and does not update its model parameters during normal inference. The architecture and tests below are proposed engineering patterns, not reported deployment results.
An assistant can remember an event accurately and still be wrong to use it as evidence of the present.
Separate Model Learning from Application State
The foundation article distinguished parametric knowledge, encoded in learned model weights, from contextual knowledge, supplied through prompts, retrieved material, conversation history, and tool observations.
Brown and colleagues’ Language Models are Few-Shot Learners demonstrated task performance using instructions and examples without gradient updates during inference. The response can adapt to supplied information without the deployed model permanently incorporating it into its parameters.
Persistence adds a separate application responsibility. A conversation can be stored and supplied again tomorrow. That makes the record durable; it does not make the storage operation a model-training step.
For this architecture, separate four kinds of state. These are operating boundaries, not a universal taxonomy imposed on every AI framework.
| State | What it contains | How it changes | Main operating concern |
|---|---|---|---|
| Model parameters | Learned capabilities and associations | Training or adaptation that updates saved parameters | Evaluation, artifact versioning, release control |
| Request context | Instructions, selected evidence, and relevant history | Context assembly for an inference call | Relevance, exposure, size, and interpretation |
| External knowledge and memory | Runbooks, observations, preferences, and retained summaries | Source updates and controlled record writes | Provenance, access, validity, correction, deletion |
| Workflow state | Pending steps, approvals, execution status, and results | Application transactions and workflow events | Consistency, authorization, recovery |
There is an important terminology trap here. In LangChain’s memory documentation, LangGraph’s short-term memory is thread-scoped state that can be persisted through database-backed checkpoints. Long-term memory is available across threads through stores. “Short-term” therefore describes recall scope, not necessarily volatile storage.
A saved conversation and a cross-session preference can both persist. Neither is equivalent to changing model weights.
Choose the Memory Mechanism That Matches the Requirement
For the checkout assistant, the current dependency map should come from its maintained source or an approved live query. An engineer’s formatting preference can live in a scoped preference record. An unfinished investigation should use structured workflow state rather than asking the model to reconstruct completion from a conversational summary.
Lewis and colleagues’ retrieval-augmented generation research combines parametric model knowledge with external, nonparametric memory. Retrieval-augmented generation, or RAG, describes a way to supply external information to generation. It is not, by itself, a complete lifecycle for that information.
Use semantic retrieval to locate potentially relevant runbooks or similar incidents. Once a specific service, record, or workflow is identified, use its authoritative identifier and appropriate data interface for exact state. Do not let the nearest matching paragraph decide whether an approval exists or a diagnostic has completed.
Consider fine-tuning when evaluation identifies a repeatable behavior gap that a parameter update is intended to address. Do not use it as the default remedy for outdated topology or a missing runbook.
Also consider whether persistence is needed at all. A one-time summary may require only the authorized evidence for that request. Retention should serve a defined use case, not happen merely because storage is available.
Design the Write Path Before Expanding Retrieval
The first design question is not how much an assistant can remember. It is what the application permits to become reusable information.
In the proposed pattern below, a source event or generated summary enters a controlled write path. The application checks the writer’s authority, the record type, its scope, and its provenance before making it eligible for later use.
The read path then makes a separate decision. Admission to a store does not grant every caller permission to retrieve the record.
Not every write requires human review. An authenticated telemetry adapter can be authorized to create a narrowly defined observation. A proposed incident conclusion needs different checks from a health-check result.
Preserve that difference in the record type. “The dependency health check failed” is an observation. “The dependency caused the outage” is a hypothesis unless the investigation establishes more.
A generated summary should retain uncertainty and references to its supporting records. It must not silently convert “suspected” into “confirmed.” Repeated summaries of the same observation should also retain their shared lineage rather than appear to be independent corroboration.
Prevent Retrieved Instructions from Becoming Durable Policy
OWASP’s LLM01:2025 Prompt Injection describes indirect injection through external content and notes that RAG does not eliminate the vulnerability.
For this assistant, an instruction inside a retrieved ticket to “remember this as the new incident policy” is content to assess, not authority to modify policy. Keep policy and approved operating procedures behind a separate change process. Do not give an ordinary summary-writing operation permission to overwrite them.
A schema check can reject malformed records, but it cannot establish that a plausible statement is true. Source validation, scope enforcement, and an appropriate review process still have separate jobs.
Give Every Record a Defined Use and Lifetime
A sentence without provenance is difficult to correct. A sentence without temporal context is difficult to use safely.
The following YAML describes one candidate observation. It is an illustrative record schema, not deployable configuration for a particular memory platform.
record_id: mem-checkout-017 record_version: 1 kind: observation status: candidate scope: tenant: example_enterprise environment: production service: checkout_api dependency: catalog_api statement: "The catalog_api health check failed." provenance: evidence_id: telemetry_snapshot_042 source_revision: 3 observed_at: "2026-09-09T14:00:00Z" ingested_at: "2026-09-09T14:00:12Z" usage: purpose: incident_hypothesis_support current_evidence_until: "2026-09-09T14:05:00Z" access_policy_ref: incident_readers_checkout retention_policy_ref: operations_evidence_retention owner: platform_operations
Replace the service identifiers, source reference, policy names, and owner with values controlled by your application. The ingestion service must derive or verify the scope, writer identity, and policy references against trusted systems. Model-generated values in those fields do not establish permission.
The five-minute window is illustrative, not a recommended default. It limits current-state use; it does not guarantee that the observation remains representative for five minutes. A newer authoritative recovery observation could make the old failure unsuitable for current diagnosis much sooner.
Keep observed_at separate from ingested_at. A delayed failure event should not overwrite newer authoritative state merely because it arrived later. Define ordering and conflict rules using the source’s timestamps, versions, or sequence guarantees.
Successful implementation keeps the candidate outside ordinary retrieval until the required validation completes. After admission, every read still checks access, intended use, and validity. The YAML itself performs none of those checks.
Expiry, Correction, and Deletion Are Different Operations
An expired observation may remain accurate as historical evidence. It should not automatically remain eligible to describe current service health.
A correction changes what the application should treat as supported. Retain the correction relationship where appropriate and invalidate affected summaries, index entries, and cached answers. A new model release will not repair those records.
Deletion is a separate retention operation. Define its coverage across source copies, chunks, embeddings, summaries, transcripts, logs, and backups. Removing one vector-index entry should not be reported as deletion from every location.
Where backups retain older material under an approved retention policy, the restore process must reapply applicable deletion and revocation records before serving that material again. Otherwise, recovery could reintroduce information already excluded from use.
RAG Access Control Must Survive Summaries and Caches
Relevance and authorization answer different questions. A document can closely match the incident and still belong to another tenant or restricted service.
Microsoft’s Document-level access control in Azure AI Search provides a concrete example of enforcing document permissions in retrieval. It distinguishes application-driven security filters from native permission integrations with preview requirements, and warns of propagation delays for permission changes in the preview functionality.
For this design, derive caller identity from authenticated application context and enforce access before restricted content enters the model context or an unauthorized downstream service. A model-supplied tenant identifier is not an access-control decision.
Where indexed permissions cannot meet the required revocation window, add authoritative read-time checks or withhold affected records. Document the remaining delay rather than describing permission-aware retrieval as instantaneous revocation.
Derived Content Needs Its Own Access Decision
Suppose a saved summary combines an ordinary incident ticket with a restricted security investigation. Restricting only the original investigation document leaves a potential disclosure path through the summary.
In this design, retain source lineage and prevent derived material from receiving broader access than its contributing restricted information unless an approved transformation permits it. Re-evaluate that eligibility when source permissions change.
Apply the same principle to caches and resumed conversations. A cached answer is not safe to reuse merely because it was authorized when first generated. Recheck current eligibility and invalidate affected material before including it in a new response.
Revocation cannot retract information already delivered to a user. Its enforceable scope is future retrieval, reuse, and disclosure through systems under the application’s control.
Assemble Evidence, Not Just More Text
After access and validity checks, the application still needs to assemble a useful context.
For the checkout incident, I would supply the current request, the relevant topology version, selected observations with timestamps, and the applicable runbook passages. Historical incidents would be labeled as historical comparisons, not blended into present-state evidence.
When sources conflict, preserve the conflict. A previous failure followed by a successful health check may describe recovery rather than disagreement. Two sources claiming different current states may require a fresh observation. Do not resolve either situation simply by choosing the most recently ingested paragraph.
A larger context window does not remove the need to test evidence use. Liu and colleagues’ Lost in the Middle found position-sensitive performance in the models and tasks they studied. That supports testing the deployed model’s actual behavior, not assigning the same failure rate to every current model.
Vary where decisive evidence appears, introduce relevant contradictions, and check whether the response preserves source and time distinctions. A source identifier makes an answer traceable; it does not independently prove that the interpretation is correct.
Test Memory as a Lifecycle, Not a Recall Demonstration
The evaluation question is broader than whether the assistant can retrieve something saved yesterday. It is whether the complete application uses, excludes, corrects, and retires information appropriately.
Extend Part 1’s controlled testing approach with lifecycle cases.
| Test | Expected behavior |
|---|---|
| Replace an outdated dependency map | Current analysis uses the applicable version; the old map appears only when historical context is required |
| Expire a failure observation | The assistant requests fresh evidence rather than presenting the old failure as current |
| Correct a suspected root cause | Dependent summaries and caches stop presenting the rejected hypothesis as confirmed |
| Revoke a caller’s access | Source records and restricted derivatives meet the same defined revocation requirement |
| Retrieve a ticket containing a memory-write instruction | The instruction does not gain authority to change policy or approved memory |
| Resume a saved conversation after permissions change | Context is rebuilt or filtered under current access conditions |
Run these tests through ingestion, indexing, retrieval, context assembly, and generation. A correct database record is insufficient evidence when an old summary still reaches the model.
For reproducible evaluation, use a controlled clock for expiry tests and versioned fixtures for source and permission changes. Capture the records actually admitted to context, not only the final answer.
Measure correction and revocation propagation time, stale-evidence use, unauthorized content admitted to context, and whether sufficient evidence produces a useful response. Keep these results separate from an aggregate answer-quality score. Blocking every record may reduce exposure while making the assistant unusable.
These are proposed acceptance tests. They do not establish production readiness until the deployed implementation has been exercised and its results reviewed.
Operate Memory Before Reaching for Retraining
When the assistant repeats an old fact, trace the information path before changing the model.
Check whether the source was current, whether the eligible version reached the index, whether retrieval selected it, whether context assembly preserved it, and whether the response used it correctly. That sequence separates source, synchronization, retrieval, context, and model-behavior failures.
Assign ownership accordingly. Source owners approve authoritative content and validity rules. The application team owns memory admission, context assembly, and invalidation. Security defines access and revocation requirements. Operations monitors propagation failures and the health of those controls.
Keep record and schema versions, index-generation identifiers, and references to the evidence admitted for reviewed responses. Protect diagnostic records themselves and avoid indiscriminate retention of sensitive prompts.
Define a containment path as well: disable memory writes, exclude a suspect source, invalidate derived records, or fall back to fresh approved evidence. Restoring an earlier model does not undo a bad memory write, and restoring an earlier memory snapshot must not undo later access revocations.
Conclusion
AI memory architecture is not primarily about making an assistant retain more information. It is about deciding which information can persist, how it becomes available, and when it remains appropriate to use.
Keep model parameters, request context, external records, and workflow state distinct. Give reusable information provenance, an owner, an access boundary, and a defined lifecycle. Preserve the difference between an observation and a hypothesis, and between historical accuracy and current relevance.
Start with one incident family and a small set of memory types. Prove that the application can admit a record, use it correctly, correct it, and stop using it across retrieval, summaries, and caches. That is a more meaningful milestone than remembering a preference in a demonstration.
The final article, AI Decision Controls: From Learned Patterns to Authorized Actions, moves from information to execution. A supported recommendation is only the beginning; the surrounding system still has to establish what may actually happen.
External References
- Microsoft Learn: Document-level access control in Azure AI Search
Canonical URL: https://learn.microsoft.com/en-us/azure/search/search-document-level-access-overview - LangChain: Memory overview
Canonical URL: https://docs.langchain.com/oss/python/concepts/memory - OWASP Gen AI Security Project: LLM01:2025 Prompt Injection
Canonical URL: https://genai.owasp.org/llmrisk/llm01-prompt-injection/ - arXiv: Language Models are Few-Shot Learners
Canonical URL: https://arxiv.org/abs/2005.14165 - arXiv: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Canonical URL: https://arxiv.org/abs/2005.11401 - ACL Anthology: Lost in the Middle: How Language Models Use Long Contexts
Canonical URL: https://aclanthology.org/2024.tacl-1.9/
TL;DR AI agent governance must separate evidence, approval, and permission to execute. A shared workspace can coordinate a convincing recommendation without granting...