TL;DR
AI agent memory governance begins by separating feedback from permission to change future behavior. A corrected response, a stored incident memory, a revised runbook, and updated model weights are different change mechanisms. Each needs an appropriate owner, evaluation method, release boundary, and recovery path. Preserve what happened before deciding what it teaches. Then test the proposed lesson against cases where it should not apply. Feedback can support learning, including adaptation without retraining, but collecting feedback does not establish that the system improved.
Introduction
The checkout service recovers after an approved gateway route rollback. The agent closes its investigation and stores a lesson: “When checkout requests time out, restore the previous gateway configuration.”
That sentence discards nearly everything that made the intervention reasonable: the affected request path, the qualified observations, the specific configuration difference, and the approval tied to the exact change. A bounded response to one incident has become a proposed rule for future incidents.
Part 1, AI Confidence Is Not Evidence: Building an Evidence Contract, established how to qualify observations. Part 2, AI Agents Should Verify Before They Act, separated useful diagnostics, authorization, execution, and recovery. This final article follows the outcome into the feedback pipeline.
We continue the same illustrative checkout scenario, assuming the approved rollback completed and subsequent observations met the service owner’s recovery criteria. The root-cause review remains open. Ordinary incident processing does not update model weights, and persistent changes pass through a separate release process.
The architecture and policy below are proposed engineering extensions of the series, not a published standard or a validated product configuration. The question is not whether an agent should improve. It is what should change, on whose evidence, at what scope, and with what way to reverse its influence?
Identify What Actually Changes
“The agent learned” is not an adequate change record. It does not identify where the new behavior resides or which team can restore the previous state.
Brown and colleagues’ Language Models are Few-Shot Learners provides a familiar example of adapting to task instructions and examples without gradient updates during evaluation. That supports the distinction between changing context and changing trained parameters. It does not mean that context-based adaptation is unimportant or incapable of improving performance.
The Reflexion research makes this boundary especially useful. Shinn and colleagues describe agents that retain textual reflections in an episodic memory buffer and use them in subsequent attempts, without updating model weights. Their results demonstrate improvement in the studied tasks, not validation of an unrestricted production memory pipeline.
For this workflow, separate the mechanisms before selecting controls:
| Change mechanism | What changes | Recovery question |
|---|---|---|
| Session context | Information available during the current task | Which active tasks must be rebuilt from approved context? |
| Persistent memory or retrieval | Records that future tasks can receive | Which records and derived artifacts must stop being retrieved? |
| Prompt, runbook, or policy | Application instructions or operating rules | Which approved version should replace the changed artifact? |
| Model or fine-tuning adapter training | Learned parameters | Which compatible deployment artifact can safely replace the release? |
These are related change paths, not interchangeable descriptions of learning. A fixed model can behave differently because its retrieved context changed. Conversely, removing an external memory entry does not reverse a parameter update.
For the checkout agent, the immediate candidate is historical incident knowledge. It is not automatically a runbook amendment, a permission change, or a training example.
Preserve the Incident Before Promoting the Lesson
The incident records establish a sequence: the external checkout request failed, diagnostics narrowed the investigation, an approved route rollback executed, and later checks showed recovery. That supports a historical account.
It does not, on its own, establish that the rollback was the sole cause of recovery. A dependency might have recovered at the same time, or the change might have worked around a different defect. Preserve those possibilities until the investigation provides stronger support.
Write the initial memory accordingly: “Checkout recovered after an approved route rollback; the causal assessment remains unresolved.” Link it to the actual observations, configuration revisions, and execution receipt. Do not replace those records with the agent’s summary.
A reusable procedure requires a different decision. Its reviewer must establish the conditions under which the procedure applies, the checks required beforehand, the expected effects, and the conditions that prohibit its use. Permission to retain an incident history is not permission to repeat the intervention.
Feedback Needs a Defined Target
A positive operator rating may indicate that the explanation was helpful. A recovered service indicates an operational outcome. Neither automatically establishes a correct diagnosis or a compliant action.
Keep those labels separate. In this proposed evaluation scheme, a successful but unauthorized change fails the authorization criterion. An accurate escalation can satisfy the workflow’s decision criteria even when the agent does not repair the service itself.
That distinction prevents the feedback process from rewarding whichever behavior closes the most tickets while overlooking how those tickets were closed.
Build a Separate Feedback Promotion Path
Capturing feedback and releasing a change should be different operations. In this design, the agent can submit a candidate record but cannot make that record available to future production tasks.
NIST’s Generative Artificial Intelligence Profile connects post-deployment monitoring with capturing and evaluating feedback, incident response, recovery, and change management. The architecture below applies that lifecycle principle to incident knowledge. NIST does not prescribe this specific promotion workflow.
The important boundary is between the candidate store and production use. Review can reject the proposed lesson while retaining an appropriately protected historical record.
Give the candidate writer and release service different identities. The candidate writer should not be able to change review outcomes, expand tenant scope, or write directly to the production index. The release service should accept a trusted approval record tied to the exact candidate version.
If the candidate changes after approval, the approval no longer covers that version. If review or policy validation is unavailable, leave the candidate pending. The operational incident can still proceed through its established response process without making new knowledge automatically retrievable.
For a pilot, use human review for durable incident knowledge. Later automation can be scoped to well-defined, low-impact record types after testing. Automatic acceptance of a presentation preference should not imply automatic acceptance of a new production remediation rule.
Define a Memory Candidate, Not a New Operating Rule
The following YAML represents the proposed historical record. It identifies the claim, its evidence lineage, its scope, and its permitted uses. It deliberately leaves production retrieval blocked pending review.
This is an illustrative data contract, not configuration for a named agent framework. Replace the resource identifiers, review role, and policy references with values managed by your platform. The ingestion service should attach scope and provenance from authenticated workflow records rather than trusting the model to certify them.
schema_version: "1.0"
candidate_id: mem-checkout-042
revision: 1
state: review_required
claim:
type: observed_sequence
text: >-
Checkout recovered after an approved gateway route rollback.
The causal assessment remains unresolved.
causal_status: not_established
scope:
tenant_id: tenant-a
environment: production
service_id: checkout
provenance:
root_incident_id: inc-checkout-042
evidence_snapshot_id: evset-checkout-042
action_receipt_id: action-checkout-rollback-01
outcome_evidence_id: ev-checkout-recovery-01
use_policy:
policy_version: "incident-memory-v1"
allowed_uses:
- historical_context
- hypothesis_generation
production_retrieval: blocked_pending_review
grants_action_authority: false
cross_scope_reuse: separate_review_required
training_eligible: false
review:
required_role: checkout-service-owner
bind_approval_to: candidate_id_and_revision
automatic_promotion: false
Successful processing creates a reviewable candidate with resolvable evidence references. It does not create a production search result, authorize a rollback, or schedule a training job.
The use restrictions require enforcement beyond YAML. Authorization belongs in the retrieval and execution services. The distinction between historical context and current-state proof also needs an evaluated claim-support check; a schema alone cannot determine whether the generated diagnosis misuses the record.
Keep the approval receipt separate from model-generated content. Bind it to immutable content or an integrity-checked artifact, not just an identifier whose contents can change underneath it. Promotion should record the reviewer, policy version, approved scope, and released artifact version.
When search indexing is asynchronous, do not declare the release available merely because a database write succeeded. Confirm that the intended version is indexed with the required access controls, and ensure a failed release does not expose a partially qualified record.
Do Not Let Repetition Become Evidence
Suppose the agent writes its gateway explanation into a ticket, another agent summarizes the ticket, and a third retrieves that summary during the next incident. The explanation now appears in several places, but its evidence has not multiplied.
Carry the root incident and evidence identifiers through those transformations. Mark generated summaries as derived material. A summary can help a reader navigate the record; it cannot independently corroborate the claim it summarizes.
This is the feedback equivalent of Part 1’s three dashboards displaying one alert. Repetition can increase the visibility of a diagnosis without increasing its support.
Preserve Trust Boundaries Through Storage and Retrieval
OWASP’s AI Agent Security Cheat Sheet identifies memory poisoning as a risk and recommends validation, isolation, limits, and checks before persistence. Its RAG Security Cheat Sheet also addresses access metadata, provenance, and authorization before retrieved content reaches the model.
For the checkout pattern, reject a candidate that tries to turn an incident observation into an instruction to bypass change approval. A trusted uploader does not make every embedded instruction authoritative, and an injection scan is not proof that a record is safe.
Keep source restrictions attached to summaries and chunks. Combining records does not justify granting a wider audience access to their contents. A separately reviewed, appropriately transformed general lesson may have broader applicability, but it is a new release decision, not an automatic consequence of summarization.
Evaluate the Lesson Where It Should Not Apply
Replaying the originating incident helps reproduce the problem, but it is a weak test of whether the memory is safe to reuse. The candidate was written from that incident. It should not receive release approval solely because it helps the agent repeat the same explanation.
Compare the existing workflow with the candidate-enabled workflow using a held-out scenario set. Keep the model, prompt, tools, and other retrieval content fixed as far as practical so the proposed memory change is the main difference. Record unavoidable differences.
Include cases designed to challenge the lesson’s boundaries:
| Evaluation case | Expected behavior |
|---|---|
| Similar external timeout caused by a backend failure | Investigate the backend evidence rather than assume a gateway defect |
| Gateway configuration unchanged from the approved revision | Do not recommend rollback solely because the symptom resembles the previous incident |
| Similar incident belonging to another tenant | Enforce retrieval isolation before content reaches the model |
| Current observations conflict with the historical explanation | Preserve the conflict and assess current evidence |
| Retrieved history describes an earlier approved rollback | Require a new, action-specific approval for the present incident |
Define pass criteria before running the comparison. Evaluate supported recommendations, inappropriate memory use, uncertainty preservation, investigation cost, and policy compliance. A better overall score must not compensate for a cross-tenant disclosure or an approval bypass.
For variable model outputs, use repeated runs where warranted and report the number of evaluated cases. Passing a finite test suite establishes results for those tests, not a guarantee about every future input.
Avoid a Feedback Set That Contains Only Wins
Retain failed interventions, unresolved cases, and correct escalations alongside successful repairs. Keep outcome-label uncertainty visible. Otherwise, the review process may evaluate only the incidents where a clean success story was available.
Split development and evaluation data by incident, including derived tickets and summaries. Do not allow the final root-cause report to appear in a replay that represents an earlier decision point.
When a model assists with evaluation, test its judgments against a reviewed sample. Its agreement with the proposing agent is not independent proof. For consequential changes, use service-owner review and directly checkable policy outcomes alongside automated assessment.
Treat Model Updates as a Separate Release
An accepted memory candidate should not automatically become training data. First identify the failure being corrected.
If the adapter mislabeled an incomplete collection as a healthy result, fix the adapter. If retrieval supplied the wrong tenant’s incident, fix authorization and retrieval scope. If a supported diagnostic step is missing from the runbook, revise and test that procedure. Training the model around those defects can leave the underlying control failure intact.
Consider a parameter update when the team has a defined capability or behavior gap, appropriate training examples, and evidence that narrower application changes are insufficient or unsuitable. Treat that as an explicit design decision, not the default destination for every correction.
OWASP’s Secure AI/ML Model Ops Cheat Sheet recommends auditable, version-controlled training pipelines and controlled deployment practices, including shadow evaluation, canary releases, and rollback. Apply those mechanisms to a model release separately from memory promotion.
For this workflow, a training proposal should identify the base model, any fine-tuning adapter, the approved dataset version, label definitions, training configuration, evaluation suite, and deployment owner. Preserve provenance for examples, minimize sensitive data, and have the responsible data owner authorize the proposed use. Approval to retain operational history is not approval to use it for every training purpose.
Keep training data and evaluation holdouts separated at the incident level. Test the intended improvement alongside previously working behavior, refusal and approval boundaries, and handling of unfamiliar incidents. Do not promote a release merely because its training objective improved.
Start a candidate in an isolated evaluation or shadow environment, with production writes disabled and test memory stores separated from production. A shadow agent that writes its reflections into live memory can influence production even when its direct responses are hidden.
A limited rollout should preserve a compatible fallback and explicit stop criteria. Pin versions where the platform allows it, record resolved versions when available, and rerun the relevant evaluations after material provider or application changes.
Design Revocation Before the First Promotion
A bad memory is not contained merely because someone edits the source ticket. OWASP’s RAG Security Cheat Sheet describes the need to handle derived chunks, indexes, and caches when source data or permissions change.
For this proposed design, keep enough lineage to find the artifacts that could still supply the revoked lesson. The diagram shows possible distribution paths, not independent confirmations or a guarantee that every consumer used the content.
Block Further Use Before Waiting for Cleanup
Mark the record unavailable in an authoritative retrieval control and propagate invalidation. Do not rely only on eventual index deletion when a known-bad entry must stop being supplied.
Keep revocation status outside the index snapshot being restored. Otherwise, restoring an old index can restore the same revoked record. Reapply current access decisions and revocations before returning a restored service to production.
For active tasks, identify which ones received the affected material. Where it influenced a pending decision, rebuild the assessment from approved context or escalate. Deleting the external record does not remove text already placed in a task’s context or checkpoint.
Before consequential execution, retain Part 2’s fresh evidence and authorization checks. A previously assembled plan should not bypass them because it was created before revocation.
Match Recovery to the Artifact
For a faulty runbook, restore an approved compatible revision. For a model regression, restore a suitable model and application combination where available. Do not roll back current security policy blindly as part of a historical bundle restore.
If the information entered training, removing the original record from retrieval does not change the trained parameters. Use dataset-to-model lineage to identify affected artifacts. Reverting to an appropriate earlier release or rebuilding from an approved dataset is a different operation from deleting memory; neither should be described as universal proof of erasure.
Revocation and retention also serve different purposes. A historical record may remain in a restricted evidence archive while being excluded from agent retrieval. Apply the applicable retention or deletion process to originals, derived data, and backup restoration paths rather than treating a search-index update as complete data removal.
Finally, inspect downstream effects. Preventing future retrieval does not undo a customer notification, configuration change, or ticket already created under the bad lesson. Containment stops further influence; recovery must address effects that already occurred.
Operate the Feedback Pipeline as Production Infrastructure
The model is only one component in the change path. For this design, keep a version record that connects the model, prompt, tool contracts, policy, retrieval configuration, and relevant memory revisions. Include the actual evidence supplied to a task where retention and access rules permit it.
This makes an investigation more precise. Instead of “the agent became unreliable,” the team can ask whether a memory release changed retrieval, an embedding update changed ranking, or a prompt revision changed how historical incidents were interpreted.
Assign ownership before the first promotion. Service owners review incident meaning and applicability. Platform operations owns storage, indexing, release propagation, and revocation. Security owns access and action boundaries. The AI application or model team owns behavioral evaluation, with data owners involved when records are proposed for training.
Track review backlog, time to approved release, inappropriate memory use, unresolved feedback, and revocation propagation time. Keep traces of which records entered a task, while recognizing that exposure alone does not prove that a particular record caused the output.
A queue outage should delay new promotions, not silently approve them. An unavailable required memory-control service should trigger the documented degraded mode or operator path, not unrestricted retrieval. Set quotas and review capacity so the feedback process cannot accumulate unlimited sensitive records or evaluation work.
Begin with one incident-memory route in shadow mode. Demonstrate that a useful candidate stays narrow, a misleading candidate is rejected, and a revoked candidate cannot return through caches, restored indexes, or resumed tasks. Expand only after the release and recovery paths work together.
Conclusion
The series began with prediction and evidence. It ends with another boundary that matters just as much: feedback and permission to change future behavior are not the same thing.
AI agent memory governance requires a visible path from an observed outcome to a scoped candidate, a reviewed change, an evaluated release, and a tested recovery mechanism. Context, memory, operating rules, and model weights can all change behavior, but they do so through different artifacts and require different controls.
For the checkout incident, the defensible first lesson is a qualified historical record, not “roll back the gateway whenever checkout fails.” More general guidance should emerge only when the evidence and evaluation justify that broader scope.
Start with one existing feedback route. Identify where corrections are stored, who can retrieve them, what approves their future use, and how their influence is revoked. Then test a case where the previous incident’s lesson would be wrong.
A trustworthy AI system should be able to show not only what changed its answer, but what changed its future behavior, why that change was accepted, and how it remains under control.
External References
- NIST: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
Canonical URL: https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence - OWASP: AI Agent Security Cheat Sheet
Canonical URL: https://cheatsheetseries.owasp.org/cheatsheets/AI_Agent_Security_Cheat_Sheet.html - OWASP: Retrieval-Augmented Generation (RAG) Security Cheat Sheet
Canonical URL: https://cheatsheetseries.owasp.org/cheatsheets/RAG_Security_Cheat_Sheet.html - OWASP: Secure AI/ML Model Ops Cheat Sheet
Canonical URL: https://cheatsheetseries.owasp.org/cheatsheets/Secure_AI_Model_Ops_Cheat_Sheet.html - Advances in Neural Information Processing Systems: Reflexion: language agents with verbal reinforcement learning
Canonical URL: https://proceedings.neurips.cc/paper_files/paper/2023/hash/1b44b878bb782e6954cd888628510e90-Abstract-Conference.html - arXiv: Language Models are Few-Shot Learners
Canonical URL: https://arxiv.org/abs/2005.14165
TL;DR AI agent reliability is not established by a successful model response, a saved checkpoint, or an accepted tool call. Test whether...