TL;DR
AI agent learning needs a boundary between using feedback now and changing future behavior. Preserve operational evidence without automatically turning it into trusted memory, procedures, policy, or training data. Qualify durable changes through provenance, scope, evaluation, approval, and controlled release. Keep reusable knowledge separate from execution authority, and make revocation reach cached content and active workflows. A practical starting point is one service-specific procedure update that can be tested, traced to its evidence, and withdrawn without losing the incident record.
Introduction
The staging service recovered. The approved fallback passed verification, the deployment episode closed, and the agent prepared a lesson for future use:
“When catalog-api exceeds its error-rate limit, restore the previous release.”
That sentence discards most of the controls that made the recovery acceptable. The fallback was explicitly identified, checked for compatibility, separately authorized, and evaluated under a defined workload. The original failure also did not establish its own root cause.
Part 1, AI Agent Verification: Prove the Outcome, Not the Tool Call, established what the runtime may claim about an outcome. Part 2, AI Agent Stability: When Retries Become the Incident, bounded the next intervention. This final installment addresses what the system is allowed to carry into the next episode.
We retain the hypothetical catalog-api staging workflow and its existing verification and execution contracts. The qualification process below is a proposed architecture, not a vendor standard or a reported deployment. Here, “permanent” means persistent beyond the current task, not exempt from expiry, correction, or removal.
A verified recovery can justify closing an incident without justifying a permanent recovery rule.
Preserve the Observation Without Approving the Explanation
The foundation article separates feedback-driven correction from durable learning. For this workflow, make that distinction concrete by preserving three different records: what was observed, how someone interpreted it, and what change they propose.
The observation records that the original release failed its acceptance check and that the authorized fallback later passed its own check. It retains resource identity, release identifiers, measurement windows, verdict reasons, and execution history. The original failure remains a failure after recovery.
The interpretation might suggest that the new release caused the errors. That is a hypothesis unless the investigation establishes it. Dependency recovery, changing demand, or another intervention could affect the result. The existing scenario does not resolve those possibilities.
The proposal asks for a specific future change. It might add a diagnostic step, correct documentation, or request a new recovery policy. Those requests require different evidence and decision-makers.
A reviewer should be able to reject the explanation or proposal without deleting the observation. Likewise, an operator’s approval of one fallback is evidence of that action’s authorization, not standing approval for similar-looking incidents.
For catalog-api, a narrower proposal is more useful: prepare a read-only recovery-review packet containing the failed verdict, per-route errors, dependency observations, and current fallback compatibility information. That can be evaluated as a workflow improvement without claiming that the incident proved a universal cause or remedy.
Identify Which Durable Layer Is Changing
Not every persistent write is learning, and not every behavioral change requires model training.
Anthropic’s Effective context engineering for AI agents describes structured notes stored outside the context window and retrieved later. That mechanism can influence subsequent decisions through context. It does not require the notes themselves to update model parameters.
For this article, separate the following layers:
| Layer | Example in this workflow | Required boundary |
|---|---|---|
| Durable task state | Preserve an unresolved restoration across a worker restart | Accurate continuation of the same episode, not general-purpose guidance |
| Evidence archive | Retain the original and fallback verdicts | Access, retention, and integrity controls |
| Reusable memory or retrieval content | Store a scoped observation about the service | Provenance, permitted use, applicability, and expiry |
| Procedure or orchestration | Add the recovery-review packet | Versioned behavior change with regression evaluation |
| Policy or permissions | Permit another recovery action | Approval by the authority responsible for that permission |
| Model parameters | Include qualified examples in a training release | Separate dataset, training, evaluation, and deployment controls |
Part 2’s durable execution ledger remains necessary. Requiring review before it can record a timeout would break recovery. The promotion boundary applies when information gains a new purpose, audience, or ability to shape future tasks.
A note does not become low impact merely because it is text. Equally, an approved memory entry does not become executable policy. Keep the mechanisms separate even when one platform stores several of them together.
The implementation below targets a procedure update. Model training is outside its scope; changing model parameters would require an additional release process rather than reuse of this procedure approval.
Put a Qualification Gate Beside the Operational Loop
The operational loop should continue recording evidence while a separate process evaluates reusable changes. A successful task must not write directly into the approved procedure catalog.
Notice the two branches below. One supports continuity and review. The other requires qualification before future use.
NIST’s AI RMF Playbook, under Manage, addresses evaluating feedback, documenting system changes, and monitoring deployed systems. It supports the broader change-management discipline here. It does not prescribe this diagram, candidate schema, or approval workflow.
Separate the ability to propose from the ability to release. In this design, the agent can create a candidate; the release service accepts only the required evaluation and approval records. Neither an agent-written approved label nor an approving sentence inside an incident note is sufficient.
Use proportional gates. A factual correction within an established, low-risk data contract may qualify through automated checks. A procedure affecting recovery decisions warrants broader evaluation. Authority expansion requires its own approval path and cannot be bundled into a routine memory refresh.
Keep Provenance and Scope Through Every Transformation
Suppose the agent’s incident summary is copied into a knowledge article and then summarized by another agent. There are now three documents, but they may all depend on one original observation.
For this architecture, retain that lineage. A derivative document must not count as independent corroboration merely because its wording or author changed. Model agreement can help critique an explanation, but it does not create another measurement of the service.
Preserve qualifications through summarization as well. “Fallback passed in staging under workload W” must not become “fallback is safe” after compaction. Test whether scope, uncertainty, exceptions, and negative results survive each transformation used by the actual runtime.
Bind approved content to its immutable artifact identity or digest. An approval for one revision should not follow a later edit automatically. Integrity checks establish which content was approved; they do not establish that its claims are true.
Similarity Is Not Permission to Retrieve
Keep candidate artifacts outside the default runtime retrieval path. A separate folder or status field is insufficient if the agent can search around it using broader credentials.
Enforce environment, tenant, role, and release eligibility before content reaches the model. For this example, staging recovery material must not appear as production guidance merely because the same service name occurs in both environments. An authorized incident investigator may have a separate read-only evidence route, but archived records remain evidence rather than approved instructions.
OWASP’s Memory Is a Feature. It Is Also an Attack Surface describes the risk of untrusted content persisting and influencing later agent behavior. Its LLM01:2025 Prompt Injection guidance also addresses indirect instructions arriving through external content and recommends controls including least privilege and separation of external content.
The corresponding design rule here is to keep retrieved content from granting permissions or overriding executable policy. Screening content can add a defense, but a classifier’s approval cannot replace tool-boundary enforcement. Incorrect guidance can still produce a poor choice among permitted actions, which is why outcome checks and bounded execution remain necessary.
Evaluate the Proposed Change, Not Its Supporting Story
The original episode is useful for developing a candidate. It is not an independent test of the rule derived from it.
For the proposed recovery-review packet, compare the existing procedure with the candidate using the same declared model, tools, policy, workload fixtures, and knowledge baseline. Keep the verification criteria fixed unless changing them is separately approved. Otherwise, an apparent improvement could come from weakening the definition of success.
Anthropic’s Demystifying evals for AI agents recommends testing both when behavior should occur and when it should not, running multiple trials where outputs vary, and isolating trials from unintended shared state. Apply that discipline to the complete workflow, not just the text of the proposed lesson.
The following are proposed acceptance cases, not executed results:
| Evaluation case | Expected behavior |
|---|---|
| Conclusive rollout FAIL with complete diagnostics | Prepare the scoped review packet without initiating restoration |
| Missing required verification evidence | Preserve UNKNOWN; do not convert it into a recovery authorization |
| Fallback compatibility cannot be established | Report the gap and require the existing recovery review |
| Previously recorded approval has expired or been revoked | Do not reuse it to authorize a new action |
| Similar incident belongs to another environment or tenant | Exclude the entry from that task’s retrieval context |
| Candidate contains instructions to bypass controls | Prevent its release or privileged use; preserve the finding for review |
| Critical incident path is already active | Do not make completion of the new diagnostic packet a prerequisite for containment |
Score the useful outcome: whether the reviewer receives the required evidence and whether existing boundaries hold. Do not reward the procedure merely for generating longer explanations or fewer escalations.
Test deterministic controls separately from model behavior. An agent proposing a prohibited action and a gateway successfully rejecting it are two different results. A blended task score should not conceal either one.
Test Memory Across Episodes Deliberately
Reset unintended state between independent trials. Then create separate multi-episode tests that intentionally preserve memory: introduce a scoped entry, begin another task, revoke the entry, and verify its later treatment.
That tests persistence without allowing accidental leftovers to inflate evaluation results. Keep development examples separate from held-out cases, and recognize that a case becomes development material once it is used to tune the procedure.
A procedure can earn a scoped release without proving the root cause of the original incident. It must demonstrate its own usefulness and acceptable failure behavior, not borrow confidence from the recovery narrative.
A Qualification Contract for the Procedure Update
The YAML below describes a proposed candidate record and its release requirements. It is not a deployable vendor schema. It deliberately leaves the change in candidate status and contains no claim that evaluation or approval has occurred.
The example adds diagnostic context to recovery review. It does not expand tool permissions, change Part 1’s acceptance thresholds, or increase Part 2’s action budget.
change_id: catalog-learning-candidate-017
status: candidate
target:
layer: procedure
artifact: catalog-recovery-review
base_version: v1
proposed_version: v2
source:
episode_id: staging-rollout-042
evidence_refs:
- original-release-verdict-042
- restoration-verdict-042
- authorization-record-042
root_cause_status: not_established
scope:
tenant_id: bind_from_authoritative_episode_record
environment: staging
service: catalog-api
purpose: prepare_read_only_recovery_review
proposal:
add: route_errors_and_dependency_evidence
new_tool_permissions: []
authority_expansion: forbidden
diagnostic_gaps: preserve_as_unknown
critical_incident_path: never_block
qualification:
required_suite: catalog-recovery-review-regression-v1
evaluation_manifest: bind_before_approval
bind_approval_to_content_digest: true
required_reviewers: [service-owner, platform-owner]
proposer_may_self_approve: false
release:
publisher_identity: approved-release-service
strategy: shadow_then_scoped_canary
require_unexpired_scoped_release_receipt: true
runtime:
require_scope_and_bundle_match: true
require_current_release_eligibility: true
on_unknown_eligibility: exclude_and_handoff
revocation:
deny_future_reuse: true
rebuild_affected_active_context: true
revalidate_before_material_actions: true
Replace the evidence identifiers with access-controlled records and bind the tenant from the authoritative episode, not from planner-supplied text. The named test suite must resolve to actual fixtures, expected results, and retained evaluation output. The candidate’s proposed content needs its own immutable artifact; the short add field is not a complete implementation.
The evaluation manifest should identify the model or deployment revision, instructions, retrieval baseline, procedure, tool schemas and adapters, policy, and test-suite versions. Record what the provider actually exposes; do not invent a model revision behind a mutable endpoint.
The release receipt binds the approved content and evaluated configuration to its permitted cohort and validity period. A canary needs that authorization just as a wider release does. Existing diagnostic permissions must suffice, or the proposal needs a separate access review.
Reject missing bindings, unsupported fields, and incomplete approvals. Set the diagnostic collection deadline and call limits before the canary; a read-only packet must not create an unbounded investigation.
Successful enforcement means a candidate cannot enter operational use without a valid release record. Accepting the YAML or changing its status field establishes none of that.
Release the Change Without Rewriting the Incident
For the running example, reject the broad “always restore the previous release” instruction. Retain the original failure, authorized recovery, and uncertainty about causation.
The narrower review-packet proposal can proceed to evaluation. Start in read-only shadow mode: compare the packet it would prepare with the existing workflow without allowing it to initiate changes. That tests evidence assembly and reviewer usefulness, not the consequences of recovery actions that never execute.
A subsequently authorized staging canary can expose real users of the review workflow to the new packet. Define stop conditions, reviewer ownership, and the affected service scope before rollout. Keep the verification contract and execution policy unchanged during this trial unless another approved change explicitly covers them.
Release the evaluated combination, not several independently updated components that have never been assessed together. Record which artifact version the runtime actually loaded, rather than assuming that the catalog’s desired version reached every worker.
A passing canary would support that scoped procedure release. It would not establish a new rollback permission or turn the initial incident into proof that the fallback is suitable for every future release.
Revocation Must Reach Active Context, Not Just Storage
Deleting a knowledge entry is not a complete withdrawal process when copies have already entered caches, summaries, plans, or queued work. Design revocation around where the content could still influence decisions.
For this pattern, record the artifact versions supplied to each task. This identifies potential exposure, not proof that every supplied entry caused the model’s decision. Use those dependencies to invalidate affected plans conservatively when necessary.
The withdrawal path has three distinct responsibilities:
Do not rely on adding “ignore the old note” to a conversation already containing it. For consequential workflows, rebuild from eligible sources and the authoritative task ledger. Preserve unresolved execution state and consumed budgets while doing so; a clean context window must not create a fresh action allowance.
At the execution boundary, revalidate current policy and the eligibility of the artifacts on which the request depends. Define when revocation becomes effective and how cached status expires. A control that permits stale eligibility for a period must acknowledge that period, not claim instantaneous withdrawal.
For this example, inability to establish required eligibility blocks new dependent writes and triggers handoff. It does not cancel an operation already accepted downstream. Reconcile those operations using Part 2’s execution record.
Rollback of knowledge also differs from rollback of model parameters. Removing an entry from retrieval changes that retrieval path; it does not establish removal from a model trained on the content. Training releases need separate remediation. Neither action reverses external effects that already occurred.
Review Changes to the System That Uses the Knowledge
A procedure’s applicability depends on more than its text. Model updates, changed retrieval behavior, new tool schemas, different service dependencies, or altered permissions can invalidate assumptions made during evaluation.
NIST’s Manage guidance includes monitoring pretrained models and connected components. In this design, use material component changes as review triggers in addition to periodic review. A model update may change behavior without changing an access-control entry; it does not follow that every update increases capability or expands effective authority.
Pin approved artifacts for reproducibility, but do not freeze authorization. A long-running task may retain its procedure version while still obeying a current revocation or tighter policy. Define when compatibility must be reassessed rather than silently switching versions mid-episode.
The service owner approves operational applicability. The platform owner implements release, retrieval, and revocation. The authority owner approves permission changes. Those responsibilities should remain identifiable even when existing delivery infrastructure performs several of the functions.
Start With One Durable Update Path
Begin with one service and one artifact type, such as this recovery-review procedure. Inventory where its content is stored, who can promote it, how the runtime loads it, and how its use can be stopped.
Reuse existing repositories, review workflows, artifact storage, and identity controls where they meet the requirements. A separate database for every conceptual boundary is unnecessary; an unenforced label in a shared database is insufficient.
Track rejected candidates, missing provenance, out-of-scope retrieval attempts, release-version mismatches, and revocation propagation alongside useful outcomes. For this procedure, reviewer rework and missing diagnostic evidence are more meaningful than the number of memories created.
There is a real cost to qualification: evaluation effort, release latency, storage, and operational ownership. Apply lighter controls to bounded factual updates and stronger controls to changes with wider consequences. When the organization cannot identify consumers or withdraw harmful guidance, keep new material in a review-only path rather than calling it approved learning.
Conclusion
AI agent learning becomes governable when recording, interpreting, approving, and deploying a lesson are different operations. Preserve the evidence, qualify the proposed change, and keep its scope and authority visible when it reaches a future task.
Across this series, verification establishes what happened. Stable execution bounds what may happen next. Qualified learning determines which changes may influence tomorrow’s decisions. None of those boundaries replaces the others.
The practical starting point is one versioned procedure with traceable evidence, a representative evaluation suite, a scoped release, and a tested withdrawal path. Improve that mechanism before opening a broader route from incident feedback into persistent operating instructions.
The right lesson is not whatever the agent remembers. It is what the evidence supports, the organization approves, and the runtime can still revoke.
External References
- NIST AI Resource Center: Manage
Canonical URL: https://airc.nist.gov/airmf-resources/playbook/manage/ - OWASP Gen AI Security Project: Memory Is a Feature. It Is Also an Attack Surface
Canonical URL: https://genai.owasp.org/2026/05/13/memory-is-a-feature-it-is-also-an-attack-surface/ - OWASP Gen AI Security Project: LLM01:2025 Prompt Injection
Canonical URL: https://genai.owasp.org/llmrisk/llm01-prompt-injection/ - Anthropic: Effective context engineering for AI agents
Canonical URL: https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents - Anthropic: Demystifying evals for AI agents
Canonical URL: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
TL;DR AI decision controls separate what a model proposes from what an application may execute. A useful recommendation still needs appropriate evidence,...