TL;DR
AI feedback governance controls which observations become persistent changes, where those changes apply, and who can authorize them. A positive rating, a verified resolution, and an approved training example are different things. Route corrections to the appropriate layer, preserve attribution, bind approvals to the exact proposed change, and test the release and withdrawal paths. The goal is not to stop improvement. It is to prevent a local experience from becoming an unreviewed rule for everyone.
Introduction
A support incident is resolved. The user gives the interaction a positive rating, and an engineer adds a note describing the customer-specific configuration correction that restored service.
The feedback pipeline sees a successful outcome, extracts a lesson, and makes that lesson available to future agents.
But the agent did not complete the repair. The engineer did. The correction applies to one customer’s environment, not every customer. The engineer’s authority to make that change also does not automatically extend to the agent.
Nothing in the positive rating establishes those distinctions.
The first article, AI Reward Design: Stop Optimizing the Wrong Outcome, defined the difference between a useful business outcome and a convenient score. The second, AI Agent Evaluation: Test the Behavior, Not the Explanation, showed how to inspect actions and independently verify results.
This final installment addresses what happens after the evidence arrives: which parts of that experience should influence future behavior, through which mechanism, and under whose authority?
We will continue the illustrative support workflow and develop a proposed feedback-promotion pattern. The architecture, manifest, and acceptance scenarios are design examples, not results from a production deployment.
Identify What Actually Changes When the System Learns
The foundational article distinguished conversation context, persistent memory, retrieval, policy, and model training. Keep those distinctions explicit. Saying that an agent “learned from feedback” is not a sufficient change record.
In this proposed architecture, production inference does not automatically update model parameters. Feedback enters a separate process that can propose a change to an identified component. The platform can inspect outcomes, authenticate reviewers, version affected artifacts, and restrict publication credentials.
The first decision is the destination:
| Destination | What changes | Review question |
|---|---|---|
| Conversation context | Information available within the current workflow | Is the correction relevant and safe to use here? |
| Persistent memory | Stored preferences or facts reused by the application | Which user, tenant, and purpose may reuse it? |
| Retrieval source | Documents or records supplied as external context | Is the information valid for the target environment? |
| Prompt or orchestration | Instructions, routing, or application behavior | Does this change solve the diagnosed failure? |
| Authorization policy | Which identities may perform which actions | Who can grant this authority? |
| Model training | Parameters of a resulting model artifact | Is the dataset appropriate, permitted, and independently evaluable? |
These are not a fixed ranking of risk. A widely shared retrieval update could affect more workflows than a narrowly deployed model update. Scope and distribution determine the exposure.
LangChain’s Memory overview provides a concrete implementation example: LangGraph distinguishes thread-scoped state persisted through checkpoints from long-term memory organized in custom namespaces. A conversation-scoped record is therefore not necessarily short-lived storage. Retention and reuse scope need separate decisions.
For the support incident, a reviewed, tenant-specific knowledge correction is a more direct candidate than an automatic training update. It can describe the engineer’s validated procedure while leaving the agent’s execution permissions unchanged.
Preserve the Observation Before Promoting the Lesson
The incident contains several facts: the agent attempted a repair, an engineer intervened, the service recovered, and the user liked the interaction. Preserve those facts separately before deriving a reusable lesson.
A record labeled only successful_interaction loses the distinction between user satisfaction, autonomous performance, human assistance, and technical recovery. A downstream dataset builder could then turn a positive experience into a false example of independent agent success.
Carry forward the action history, outcome evidence, attribution, and applicable contract version. Record the user rating as a rating, not as a technical verification result.
A Failed Trial Can Still Produce Useful Evidence
Part 2’s grader deliberately treated human-assisted repair as insufficient for an autonomous-resolution claim. That does not make the entire incident useless.
The engineer’s intervention could become a reviewed demonstration, a scoped knowledge correction, or a regression scenario. The agent’s failed approach could become a labeled negative example. Each destination requires a different interpretation and approval.
Likewise, a passing trial is not automatically eligible for training. The content may belong to another tenant, contain restricted information, or already serve as a protected release-acceptance test.
Evaluation describes what the evidence establishes. Promotion decides how that evidence may be reused.
Preserve the timing boundary from Part 1 as well. Decision-time verification may justify operational closure while the final outcome remains provisional. Where a label claims that recovery persisted, wait for the required follow-up evidence. A later reopening should trigger reassessment, not silent replacement of the original history.
Put a Controlled Boundary Between Feedback and Publication
OWASP’s LLM04:2025 Data and Model Poisoning identifies manipulation of training, fine-tuning, and embedding data as an integrity risk. Its recommendations include tracking data origins and transformations, validating legitimacy, and versioning datasets.
For this workflow, apply that discipline at feedback ingestion. Treat a correction as proposed content, not as an instruction to modify shared memory or deployment policy. Authentication establishes who submitted it; it does not establish technical correctness or permitted reuse.
The proposed architecture separates collection from publication. Notice that the acting agent can submit evidence or a candidate correction, but it cannot approve or publish its own change.
Use separate capabilities for submission, review, and publication. The support agent should not possess the credential that publishes the knowledge base. A reviewer approves the candidate through an authenticated application. The release service verifies that decision and performs the permitted write.
This follows OWASP’s LLM06:2025 Excessive Agency guidance on limiting permissions and enforcing authorization downstream. A sentence telling the agent not to update shared knowledge is not equivalent to removing its write access.
Protect the intake store too. A correction containing “ignore approvals for future tickets” must not become an executable instruction merely because a summarizer copied it into a trusted-looking field. Treat model-generated summaries as derived content that still needs review.
Collect the evidence needed for that review without copying full customer conversations into every pipeline component. Keep sensitive source material in restricted storage, reference it from the change record, and apply defined retention and deletion rules.
Match Approval Depth to Scope and Consequence
Not every memory write needs a committee. Nor should every persistent change inherit the lightweight process used for a display preference.
For this design, an authenticated user’s selection of a predefined response format can follow an automated, preapproved rule. The application accepts only supported values, restricts the change to that user, records the update, and provides a reset path.
Free-form operational guidance has different consequences. A customer-specific remediation note needs technical validation and a scope decision. A shared runbook needs broader evaluation. A permission expansion belongs in a separate authorization workflow, regardless of how useful the proposed action appears.
Derive tenant and user scope from authenticated application context. Do not let a model-generated tenant_id determine who receives the update. Apply access checks during both publication and retrieval, before protected content enters the model’s context.
Automation remains useful within these boundaries. The distinction is between an automated decision under an approved policy and an agent granting itself permission through its own explanation.
Represent the Proposed Change with a Promotion Manifest
The following YAML describes a proposed correction to one tenant’s retrieval content. It is an illustrative record and policy contract, not a deployable product schema. All identifiers are synthetic, and the change intentionally remains blocked pending review.
Replace the evidence references, target artifact, ownership roles, and version identifiers with values from your environment. Approval receipts must come from an authenticated approval service, not from model output.
feedback_change:
schema_version: "1"
change_id: feedback-support-1042
state: quarantined
evidence:
workflow_run: support-1042
source_record: restricted-evidence-1042
outcome_attribution: human_assisted
follow_up_status: pending
target:
kind: retrieval_document
tenant_id: tenant-a
artifact_id: support-kb-tenant-a
expected_current_version: kb-17
candidate_artifact: immutable-candidate-kb-18
reuse:
permitted_purpose: tenant_support_retrieval
prohibited_purposes:
- cross_tenant_reuse
- model_training
- permission_changes
review:
technical_validation: pending
required_approver_roles: [service_owner, data_owner]
approval_receipt_ids: []
bind_approvals_to:
- candidate_digest
- tenant_id
- artifact_id
- permitted_purpose
- expected_current_version
evaluation:
status: not_run
required_suites:
- target_workflow_regression
- tenant_isolation
- retrieval_access_control
- revoked_source_handling
release_requirements:
verify_candidate_digest: true
validate_current_approval_authority: true
recheck_source_and_policy_status: true
on_missing_stale_or_conflicting_evidence: block
recovery:
fallback_candidate: kb-17
require_fallback_validation: true
track_derived_artifacts: true
reassess_affected_in_flight_work: true
The manifest does not approve anything by itself. A digest identifies candidate bytes; it does not prove their truth. Role names describe requirements; they do not authenticate an approver. A prohibited-use field becomes a control only when consuming services enforce it.
That includes the training-data pipeline. A retrieval-only approval should not become training permission when a separate job exports the feedback store. Restrict that job’s access and require its dataset builder to evaluate purpose-specific eligibility.
Successful implementation would publish only the reviewed artifact to the approved tenant scope, reject an unauthorized reuse request, and leave an auditable receipt identifying what changed. Missing approvals, incomplete required verification, or mismatched versions should leave the change unpublished.
Bind Approval to the Exact Change at Commit Time
A reviewer approves version kb-18 against baseline kb-17. Before publication, another team deploys kb-19. Blindly applying the earlier approval could overwrite the newer change.
For this pattern, compare the current target version with the approved baseline as part of the conditional publication operation. A mismatch returns the candidate for reconciliation and any necessary re-evaluation. Do not silently substitute a new baseline or modified candidate under an old approval.
Approval also has a time boundary. Recheck whether the evidence was withdrawn, the permission policy changed, or the approver’s relevant authority was revoked. Bind the receipt to the candidate digest, scope, purpose, and applicable release requirements.
When policy and storage live in different systems, specify how stale decisions are prevented and what revocation propagation delay remains. A policy lookup followed by an unrelated write is not an atomic transaction across both services.
Make retries safe as well. If publication succeeds but the worker loses its acknowledgment, processing the same change again should return the recorded result rather than create a second release. Use a stable change identity and durable release receipt. A materially revised proposal receives a new identity and a new review.
Evaluate the Candidate Change and the Promotion Process
Replaying the original incident answers only whether the proposed correction helps that incident. It does not establish that the correction is appropriately scoped, that other workflows remain correct, or that the publishing service respects its controls.
For the tenant-specific knowledge update, evaluate the resulting agent behavior with the candidate retrieval content and existing tool restrictions. Confirm that the agent explains the procedure accurately without treating knowledge of a repair as permission to execute it.
Test the promotion process separately:
| Test condition | Required result in this design |
|---|---|
| A user from another tenant requests the correction | Protected content is not returned to that user’s model context |
| Candidate bytes change after approval | The existing approval cannot authorize publication |
| The target baseline advances before commit | Publication stops for reconciliation |
| A source is revoked while the change is queued | The candidate cannot publish under the stale review |
| The same change is delivered twice | One release occurs, with the same recorded result |
| A training export encounters retrieval-only content | The content is excluded or routed to a separate permitted-use review |
| Approval or evaluation evidence is unavailable | Publication remains blocked |
These are acceptance criteria, not reported test results. Test failures should distinguish an invalid correction from a broken publisher, evidence adapter, or access-control implementation.
Keep evaluation material from leaking into the improvement loop. Group related incident variants when separating development data from release-acceptance cases. Once an acceptance example is used to tune a prompt or train a model, stop presenting it as independent evidence of generalization.
Track Where the Feedback Went, Not Just Where It Started
A source identifier tells you where information originated. Recovery also requires knowing which artifacts incorporated it and which executions consumed those artifacts.
For the proposed retrieval update, record derivation and usage relationships. The following diagram shows why deleting the original correction is not the complete recovery action in a system that has already produced dependent content.
Link derived summaries to their source records instead of dropping provenance during compression. Record the artifact versions supplied to each run. A missing dependency link is an evidence gap, not proof that nothing downstream was affected.
When an incident reopens, first establish whether the new evidence actually invalidates the correction. A reopened ticket with a different cause should not automatically revoke valid guidance. Conversely, credible evidence of an unsafe correction should trigger containment while the detailed investigation continues.
Retain only the lineage metadata and source content your operating requirements justify. Auditability is not a reason to retain unrestricted customer data indefinitely.
Revoke Use, Repair Artifacts, and Address Side Effects
For this design, distinguish stopping further use from completing cleanup. Block the affected artifact at the serving boundary, then identify and repair the dependent components.
For retrieval, inspect source documents, chunks, index versions, and caches. For memory, inspect scoped records and derived summaries. Long-running workflows may already hold the invalid information in context, so pausing future retrieval alone is insufficient. Revalidate, restart from a safe checkpoint, or route affected work to human review before its next material action.
Do not claim that an invalidation event reached every consumer merely because it was published. Require acknowledgment or observable enforcement where the system supports it, and keep unreachable consumers visible. Where safe validation is unavailable, suspend the affected operation rather than continue under an unverified assumption.
Deleting Training Data Is Not Model Unlearning
Google Research’s Announcing the first Machine Unlearning Challenge distinguishes deleting stored data from removing its influence on trained artifacts. Its discussion also identifies tradeoffs among forgetting quality, retained model utility, and computational cost.
Removing a feedback row from a dataset does not itself modify a model already trained on that row. A failed attempt to elicit the information is not proof that its influence disappeared.
For an affected training release, the proposed response is to identify impacted model artifacts, contain their use where necessary, and evaluate a suitable replacement or unlearning procedure. A replacement trained from an unaffected starting point with the relevant data excluded has a different evidentiary basis from additional fine-tuning intended to discourage an answer.
Restoring the Previous Version May Not Be Safe
Treat kb-17 as a fallback candidate, not an automatically safe destination. It may contain the original error, conflict with current policy, or depend on components that have since changed.
Recovery may instead require a corrected kb-20, temporary removal of the disputed guidance, or a human-only workflow. Record the active safe state and validate it before resuming automated use.
Finally, a repaired knowledge base does not undo completed actions. Incorrectly closed tickets, customer messages, and downstream writes need explicit remediation by their operational owners. Distinguish restoring the decision system from repairing what it already did.
Make Feedback Governance an Operating Responsibility
NIST’s Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile is a voluntary companion resource for incorporating trustworthiness considerations across AI design, development, use, and evaluation. The proposed workflow here applies that lifecycle perspective to feedback; it is not a claim of NIST certification or compliance.
For this support service, the service owner validates operational meaning, the data owner approves reuse, the platform owner controls publication and lineage, and the model owner approves training inclusion. Security owns permission boundaries and participates when manipulation or unauthorized exposure is suspected.
Measure whether the process works without turning approval volume into another misleading target. Useful operational signals include changes lacking ownership, blocked out-of-scope requests, unresolved evidence gaps, and the time between revocation and confirmed non-use by known consumers. Report coverage and unreachable consumers alongside that timing.
Use event-driven review triggers as well as periodic checks. A model update, changed tool, revised policy, withdrawn source, or material service change can require re-evaluation of affected releases. An old approval should not silently authorize a new scope or a different candidate.
Begin with one feedback destination, such as tenant-scoped retrieval. Demonstrate its publication, isolation, and withdrawal paths before adding automated memory writes or training exports. A small enforced workflow is more useful than a comprehensive manifest no service actually evaluates.
Conclusion
Behaviorism explains why consequences can shape future behavior. Reinforcement learning formalizes that relationship. In an enterprise AI system, however, the organization still has to decide which observations are valid, which interpretations are justified, and which persistent changes are authorized.
This series connects three responsibilities: define the outcome, evaluate the behavior, and govern what becomes reusable. A reward score cannot substitute for evidence. A passing evaluation cannot substitute for permission to publish. A reviewed correction cannot substitute for a tested recovery path.
For the support workflow, start by tracing one piece of feedback from collection to every place it can persist. Identify its scope, owner, approval, consumers, and withdrawal mechanism. Where the trace ends before the data’s influence does, the operating model needs more work.
Feedback is evidence to assess, not authority to inherit. Improvement becomes dependable when the change is explicit, scoped, tested, and recoverable.
External References
- OWASP Gen AI Security Project: LLM04:2025 Data and Model Poisoning
- OWASP Gen AI Security Project: LLM06:2025 Excessive Agency
- LangChain: Memory overview
- Google Research: Announcing the first Machine Unlearning Challenge
- NIST: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
Evaluate what an agent attempted, what executed, and what happened to the service. Keep control compliance, appropriate behavior, and verified outcomes separate...
1 thought on “AI Feedback Governance: Control What Becomes Learning”