TL;DR
Enterprise AI incident response must address identity, execution, data, persistent state, and downstream business systems. Stopping an agent is only one containment action. Responders must also restrict tool and disclosure paths, revoke the relevant authority, account for queued and delegated work, preserve evidence, and reconcile actions already accepted by other systems.
This runbook connects those responsibilities to named owners and explicit exit conditions. Its central principle is straightforward: an agent is not contained because its process stopped. Containment requires evidence that the affected workflow can no longer continue the harmful activity.
Introduction
Consider a hypothetical procurement agent that reviews supplier documents, retrieves contract records, and creates purchasing requests.
A supplier document contains instructions that redirect the agent toward an unauthorized export. The agent retrieves information outside the intended task and submits an outbound operation through a connector. Security detects the activity, and operations stops the agent.
The dashboard shows no running instances.
The investigation, however, finds an outstanding connector job, a previously issued access token, and a workflow checkpoint containing the manipulated instructions. A separate worker has already accepted a delegated task.
Stopping the visible agent addressed only one part of the incident.
That is the problem this runbook addresses. The response must follow the workflow into the identities, tools, persistence mechanisms, and downstream systems through which work can continue.
My earlier article, How to Roll Back AI Agents: Incident Response, Circuit Breakers, and Recovery Patterns, examined how to restore a safe operating envelope through changes to versions, policies, tools, retrieval, memory, and autonomy. This article focuses on a narrower security question: what must responders do when they can no longer trust the execution path or the authority being exercised through it?
Rollback may become one recovery action. It is not the entire incident response.
Define the Incident at the Workflow Boundary
An agent compromise does not necessarily mean someone modified the model’s weights or installed malware. OWASP’s Top 10 for Agentic Applications for 2026 distinguishes goal hijacking, identity and privilege abuse, and memory or context poisoning because these failures affect different parts of an agentic system.
For this runbook, treat suspected compromise as credible evidence that an agent, a supporting component, or its credentials are being used outside the approved security or business boundary. This is an operational definition, not a conclusion about attacker intent.
A wrong answer alone does not establish compromise. Conversely, a successful API response does not establish that the business action was authorized. OWASP’s LLM06:2025 Excessive Agency guidance places authorization in downstream systems rather than leaving the model to decide whether an action is permitted.
Persistence also needs explicit treatment. LangGraph’s documentation separates thread-scoped checkpoints from stores that retain information across threads. Where durable storage is configured, restarting the runtime does not inherently remove the state that influenced its behavior.
NIST SP 800-61 Revision 3 integrates incident response into cybersecurity risk management. The stages and gates below are a proposed application of that foundation to enterprise agent workflows, not additional NIST requirements or a vendor-certified procedure.
Prerequisites and Safety Checks
Before giving an agent production authority, create a response record identifying its business owner, technical owner, executing identities, approved tools, data sources, persistent stores, and downstream systems. Include the people authorized to suspend work, revoke access, preserve evidence, and approve restoration.
Use the existing enterprise incident-command structure. Google’s Site Reliability Workbook separates incident command, operations, and communications so coordination does not depend on the engineer performing the repair.
Within that structure, assign identity, AI-platform, data, and application responsibilities explicitly. The identity team should not discover during an incident that the agent also holds a connector-specific API key outside the central identity provider.
Preauthorize bounded emergency actions. Define when responders may isolate one workflow, one tenant, one connector, or a shared platform. Record the business fallback and the impact of disabling shared identities before those controls are needed.
Make containment independent of the affected agent. A model instruction to stop using a tool is not an emergency authorization control.
The component under investigation should not control whether its own restrictions take effect.
The Enterprise AI Incident Response Workflow
Use the following flow to organize the response, not to impose a rigid sequence. Execution containment, credential revocation, evidence preservation, and communications should proceed in parallel when the situation requires it.
The important boundary is between taking a containment action and proving what that action accomplished. A stopped worker, a disabled key, and a canceled request are different observations. None should be used as shorthand for the state of the entire workflow.
Detect, Declare, and Establish the Initial Scope
Lead: Security incident commander, supported by the AI-platform on-call.
Begin with observable behavior and the boundary it threatens. Useful detection cases include an action without the required approval, unexpected access across tenant or data boundaries, unexplained credential creation, an unapproved destination, or execution continuing after a suspension request.
Implement and test those cases against the agent’s actual authority model. Do not make a prompt-injection classifier or a model-generated risk score the only route into incident response.
The initial incident record should establish what happened, which observations support that conclusion, which identities and systems are involved, and what harmful activity may still be possible. Assign an owner to the immediate containment decision.
Record the earliest known suspicious activity separately from the detection time. Distinguish attempted activity from accepted or completed activity.
A blocked export attempt, an unauthorized record change, and confirmed disclosure should not become one undifferentiated AI anomaly.
Apply the enterprise severity matrix using privilege, data sensitivity, business impact, ongoing activity, and uncertainty about scope. Root-cause certainty is not required before containing credible ongoing harm.
Exit condition: An incident owner is assigned, initial scope is documented, and responders have authority to apply the necessary restrictions.
Contain Execution and Disclosure Paths
Lead: AI-platform operations, with application and network owners.
Containment must reach beyond the agent runtime. Evaluate each boundary below and record both the action taken and the evidence that it worked.
| Boundary | Containment action | Verification evidence |
|---|---|---|
| New work | Suspend affected schedules, event triggers, and task admission. | New tasks are rejected or held in a controlled queue. |
| Running work | Stop affected workers and enumerate delegated executions. | Execution records identify what stopped and what remains active. |
| Tool access | Restrict access at the tool service, gateway, or target system. | Controlled prohibited requests are denied at an enforcement point. |
| Data and output | Restrict affected retrieval, export, and outbound communication paths. | Access and delivery records show the intended restrictions taking effect. |
| Pending work | Quarantine retries, delayed jobs, and outstanding approvals. | Each pending item has a known state and an assigned disposition. |
These are controls to design and test, not capabilities that every platform automatically provides.
Removing a tool from the model’s available-tool list should accompany enforcement at the service that executes the action. For Model Context Protocol integrations, the protocol’s Security Best Practices also emphasizes token audience validation and forbids token passthrough. A credential intended for one resource should not become an unexamined route into another.
Read-Only Is Not Automatically a Safe Fallback
A read-only workflow may still retrieve confidential information and expose it through its response or another output channel. Evaluate information access and information release separately.
A verified read-only fallback may be appropriate for a contained record-integrity problem. It is not the default response to suspected disclosure or an untrusted runtime. In those cases, keep the affected workflow quarantined until its access and output boundaries are understood.
Account for Work Already Accepted
An upstream cancellation request does not establish the state of a downstream operation.
Determine whether each consequential action was proposed, submitted, accepted, committed, canceled, or left in an unknown state. The Amazon Builders’ Library explains why a timeout can leave a caller uncertain whether an operation occurred and why blindly retrying can create additional side effects.
Revalidate authorization at the last practical execution boundary. Invalidate outstanding approvals that would otherwise permit work under the pre-incident authority state.
Where a target cannot cancel an accepted operation, document the remaining exposure and assign its reconciliation.
Exit condition: Restrictions are verified for the stated scope, and outstanding work is accounted for. Describe incomplete coverage as partial containment.
Revoke Credentials and Close Remaining Authority Paths
Lead: Identity and access management, supported by connector and application owners.
Build a credential and grant inventory for the affected execution path. Include workload identities, delegated user access, service-principal credentials, connector tokens, application sessions, signing certificates, and credentials or grants created during the suspicious window.
Record identifiers, issuers, intended audiences, scopes, and relevant timestamps. Do not copy credential values into the incident ticket.
The response has two separate objectives: prevent new credentials from being obtained and prevent already issued credentials from continuing to authorize harmful activity.
Microsoft Entra: Application Sessions Need Separate Attention
Microsoft’s Revoke user access in Microsoft Entra ID documentation explains that access-token validity and application-managed sessions affect how quickly access ends. Microsoft Entra cannot directly revoke a session token issued by an application.
Continuous Access Evaluation can help in supported scenarios, but it is not a universal guarantee for every application.
Treat this as delegated-user guidance. Do not apply a user-session procedure blindly to an app-only identity or managed identity. Identify the actual credential type and use the corresponding identity and resource controls.
AWS: Revocation Does Not Automatically Prevent Re-Assumption
AWS’s documented IAM role-session procedure attaches a time-conditioned deny policy. Sessions established after the policy’s cutoff are not covered by that older-session restriction.
Responders therefore need to address how new sessions could be obtained as well as revoke existing sessions.
AWS also documents exceptions: service-linked role sessions cannot be revoked through this procedure, and roles created through IAM Identity Center permission sets require the corresponding Identity Center process.
Evaluate the operational impact before applying role-wide restrictions to a shared role.
Google Cloud: Disabling the Key Is Not Revoking Its Tokens
Google Cloud’s Disable and enable service account keys documentation explicitly states that disabling a key does not revoke short-lived credentials already issued from it.
The documentation identifies disabling or deleting the represented service account as the mechanism for revoking those compromised short-lived credentials. Account-level action affects other workloads using that identity.
These examples are not interchangeable command recipes. They demonstrate why “rotate the secret” is an incomplete incident instruction.
Verify enforcement at the relevant resources through approved, non-destructive checks. Record the observed denial time and any remaining validity or enforcement gaps. Introduce replacement credentials only through a trusted recovery path.
Exit condition: Relevant issuance paths are closed, existing authority is denied or otherwise bounded using documented mechanisms, and unresolved access paths are explicitly recorded.
Preserve Evidence While Containment Proceeds
Lead: Security investigation and the evidence custodian.
Preserve evidence while other responders stop harmful activity. NIST SP 800-61 Revision 3 calls for protecting the integrity and provenance of incident records and retaining evidence under the organization’s procedures. Formal chain-of-custody handling depends on the incident and its requirements.
For agent incidents, organize collection around four questions.
| Investigation question | Evidence to preserve |
|---|---|
| What was running? | Agent build, model identifier where available, prompt and policy versions, tool definitions, connector configuration, and deployment changes. |
| What information reached it? | Relevant user input, retrieved source versions, tool responses, conversation state, memory, and checkpoints. |
| What authority was exercised? | Authenticated identities, grants, policy decisions, approvals, session metadata, and revocation actions. |
| What happened downstream? | Target-system request identifiers, job receipts, audit events, record changes, and delivery evidence. |
Capture relevant volatile state before destructive operations when that can be done safely. Do not allow the pursuit of perfect evidence to prolong active disclosure or destructive execution.
Preserve the relevant source content or version, not only a locator that may later point to a changed document. Protect collected artifacts with restricted access, integrity checks, collection timestamps, and an evidence manifest. Use redacted working copies where appropriate.
Do not turn routine telemetry into a repository of live secrets. The MCP authorization guide specifically warns against logging authorization headers, tokens, codes, and secrets.
Treat suspicious prompts, documents, and tool responses as untrusted input during investigation. Analyze them in an isolated environment without production action rights.
A model-generated explanation may assist an analyst. It should not replace independently collected execution and target-system records.
Exit condition: Responders can reconstruct the relevant execution path, identify evidence gaps, and show how the evidence was collected and protected.
Establish Business Impact and Communicate What Is Known
Lead: Incident commander and business-process owner, supported by communications.
Expand the investigation beyond the agent that produced the first alert. Search for workflows that used the same identity, connector, source document, memory store, deployment artifact, or delegated execution path.
Separate confirmed impact from potential exposure.
Build an action ledger connecting each consequential operation to its workflow, authenticated caller, target, approval, and observed result. Preserve an explicit unknown state when the available evidence does not establish the outcome.
Before replaying work, reconcile the original request with target-system records and the API’s idempotency behavior. An agent-side error is not sufficient evidence that the target did nothing.
Then identify the business consequences. Which records require correction? Which external messages were delivered? Which exports became accessible? Which commitments were made? Which users relied on the output?
Treat reversal and compensation separately. In the procurement scenario, withdrawing an unprocessed request is different from correcting a commitment that a supplier has already acted upon. A technical record change does not necessarily resolve the external consequence.
Assign an owner to every unresolved effect.
Keep Status Updates Bounded by the Evidence
Use a designated communications lead and an agreed update cadence. State what is confirmed, what remains uncertain, what has been restricted, the business impact, the fallback process, and the next update time.
For the hypothetical procurement incident, an update might read:
We have suspended the affected procurement-agent workflow and restricted its connector access. We are reviewing activity from the earliest known suspicious execution through the containment window. A downstream service accepted an outbound job; delivery status remains under investigation. Procurement requests are being handled through the approved manual process. The next update will be issued at 15:00 UTC.
Do not translate “we have not yet found evidence” into “nothing happened.”
Bring legal and privacy teams into notification decisions through the organization’s established response process. Notification requirements should be assessed for the actual incident, not inferred from the label “AI.”
Exit condition: Confirmed impact, potential exposure, and unresolved effects are distinguishable, with assigned owners and a communication plan.
Remove the Cause and Rebuild a Trusted Recovery Candidate
Lead: AI-platform engineering and security, with data and application owners.
Remediation should address the failed trust boundary, not merely the visible symptom. Determine whether the incident involved manipulated input, a vulnerable connector, an unauthorized deployment change, excessive permissions, credential theft, poisoned persistence, or several contributing conditions.
OWASP’s agentic guidance identifies memory and context poisoning as persistent corruption that can affect later sessions. Cleaning the original input therefore does not establish that every summary, checkpoint, or shared memory entry derived from it is trustworthy.
Quarantine affected source material and evaluate the indexes, summaries, checkpoints, caches, and peer workflows that consumed it. Prevent contaminated artifacts from being automatically reintroduced.
Rebuild an untrusted runtime from verified artifacts. Correct the relevant connector, authorization, source-admission, or workflow defect, and introduce replacement access through a trusted path.
NIST’s recovery guidance calls for checking restoration assets before use and verifying restored systems before production operation.
A previous release is a recovery candidate, not proof of a clean environment. It may predate the alert without predating the compromise.
Keep the candidate isolated from unrestricted production authority while validation proceeds.
Exit condition: The failed trust boundary has been addressed, affected persistent state has a documented disposition, and the recovery candidate is ready for independent validation.
Validate Explicit Return-to-Service Gates
Lead: Security, platform owner, and business owner jointly.
Do not restore normal operation because the agent produces a convincing answer to the original prompt. Test enforcement, workflow behavior, and business recovery separately.
| Gate | Required evidence | Accountable owner |
|---|---|---|
| Identity and access | Old authority is denied on relevant tested paths; replacement access matches the approved scope. | Identity and security |
| Runtime and state | Deployment artifacts and retained state have a verified recovery disposition. | Platform and data owners |
| Enforcement | Prohibited tool, data, and output operations are blocked, including attempts from resumed work. | Security and application owners |
| Incident regression | The observed failure and relevant variants are tested in isolation alongside legitimate workflows. | Engineering and security |
| Downstream recovery | Outstanding actions and business effects are reconciled or explicitly assigned and bounded. | Business-process owner |
| Operational readiness | Monitoring, containment, escalation, and fallback paths have been exercised. | Operations |
These are proposed gates. Define measurable acceptance criteria for the specific agent, its permissions, and its deployment.
Test the whole workflow, not only the model response. A prompt-level refusal does not establish that an independently callable tool endpoint rejects the same prohibited action.
Include delayed and resumed execution. Challenge outstanding approvals, retry queues, connector sessions, and access through alternate supported routes.
Record the approved model and deployment configuration, policy version, identity scope, permitted tools, data sources, test results, unresolved risks, signatories, and conditions that trigger another suspension.
A passing evaluation supports a release decision. It does not prove the absence of every future attack.
Exit condition: Security and business owners approve a specific operating state supported by evidence. Normal availability alone is not the criterion.
Restore Service Through a Controlled Fallback
Where practical, progress from isolated evaluation to controlled advisory operation, then narrowly scoped execution with external approval. Expand authority only when the evidence supports the next stage.
Advisory operation still requires appropriate data-access and output controls. It is not an automatic safe state.
For the procurement scenario, the approved manual process can continue while the agent remains quarantined. After validation, a restricted candidate might prepare drafts for review without permission to submit purchasing requests or initiate exports.
Do not automatically replay the backlog when service resumes. Reconcile pending work, confirm that it remains necessary, and obtain current authorization before releasing consequential actions.
Rollback remains available as a recovery technique, but only when the selected version, policy, retrieval configuration, and retained state meet the same trust requirements.
The goal is not to restore the previous level of autonomy as quickly as possible. It is to restore a justified level of service without reopening the incident.
Drill the Failure That Survives the Off Switch
Choose one consequential agent and rehearse this runbook in an authorized test environment. Use synthetic records and controlled destinations.
Construct a scenario containing a persistent checkpoint, a delegated task, a previously issued credential, and a downstream operation that accepts a request but withholds its response.
Then stop the visible agent.
Ask responders to establish which work remains possible, which authority remains usable, and which downstream effects are uncertain. Require evidence from enforcement points and target systems, not only an inactive agent dashboard.
Restore a clean runtime and test whether quarantined work, old approvals, or contaminated state can return.
Measure the time from detection to verified denial on the tested paths, the time required to account for outstanding actions, and the time required to establish a safe business fallback. Track how much of the action ledger is supported by authenticated execution and target-system evidence.
Use the findings to improve the design. A drill that exposes a shared credential or an uncancelable connector job has identified an architectural dependency, not merely an operator-training problem.
Conclusion
Enterprise AI incident response is not simply a procedure for stopping a model from answering. It is a coordinated process for stopping harmful use of authority, preserving evidence, accounting for business consequences, and establishing a defensible basis for restoring service.
That requires control over execution, identities, tools, persistence, and downstream systems. It also requires agreement about who may contain the incident, who owns unresolved effects, and who may approve the return to service.
The most useful question during the incident is not, “Did we shut down the agent?”
It is, “Which harmful actions remain possible, through which identities and execution paths, and what evidence shows that we have blocked them?”
Start with one production-relevant workflow and test that question end to end. The agent should return to service when the organization can answer it for the approved operating scope, not merely when its health checks turn green.
External References
- NIST: Incident Response Recommendations and Considerations for Cybersecurity Risk Management: A CSF 2.0 Community Profile
Canonical URL: https://csrc.nist.gov/pubs/sp/800/61/r3/final - OWASP: OWASP Top 10 for Agentic Applications for 2026
Canonical URL: https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/ - OWASP: LLM06:2025 Excessive Agency
Canonical URL: https://genai.owasp.org/llmrisk/llm062025-excessive-agency/ - Microsoft Learn: Revoke user access in Microsoft Entra ID
Canonical URL: https://learn.microsoft.com/en-us/entra/identity/users/users-revoke-access - AWS: Revoke IAM role temporary security credentials
Canonical URL: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_revoke-sessions.html - Google Cloud: Disable and enable service account keys
Canonical URL: https://docs.cloud.google.com/iam/docs/keys-disable-enable - LangChain: Persistence
Canonical URL: https://docs.langchain.com/oss/python/langgraph/persistence - Model Context Protocol: Security Best Practices
Canonical URL: https://modelcontextprotocol.io/docs/2026-07-28/tutorials/security/security_best_practices - Model Context Protocol: Understanding Authorization in MCP
Canonical URL: https://modelcontextprotocol.io/docs/2026-07-28/tutorials/security/authorization - Google SRE Workbook: Incident Response
Canonical URL: https://sre.google/workbook/incident-response/ - Amazon Builders’ Library: Making retries safe with idempotent APIs
Canonical URL: https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/
TL;DR AI context governance turns evaluation findings into controls over evidence, context assembly, and consequential actions. Preserve decision-critical facts through retrieval and...