TL;DR
An AI agent should not receive production authority because it succeeded in a demonstration or scored well on a general benchmark.
Before an agent can change infrastructure, approve transactions, communicate with customers, modify records, or invoke consequential tools, it should encounter representative situations inside a controlled environment where its decisions can be observed without exposing the real business to those decisions.
For AI, “rehearsal” does not mean mental imagery. It means simulation, evaluation, repeated trials, failure injection, adversarial testing, policy validation, and controlled interaction with realistic tools and state.
The enterprise pattern is:
Rehearse the situation. Observe the behavior. Correct the system. Retest it. Then grant only the authority the evidence supports.
Autonomy should be earned through demonstrated behavior, not inferred from model capability.
Introduction
A team builds an AI agent to manage infrastructure changes.
During the demonstration, everything works.
The agent receives a clear request. The CMDB contains accurate information. The change window is open. Every API responds successfully. The requested action complies with policy. The approval is available. The agent selects the right tools, executes the change, validates the result, and produces a polished summary.
The demonstration is impressive.
It also proves very little about production readiness.
Production is where the CMDB is stale, an API times out halfway through a workflow, an approval is denied, the requested change conflicts with policy, a dependency has already failed, a tool returns partial success, retrieved instructions disagree, and a human asks the agent to do something outside its delegated authority.
That is the environment in which autonomous behavior actually matters.
Part 1 of this series established that changes in AI behavior can come from many layers: model parameters, instructions, retrieval, memory, tools, policy, and other runtime state. Part 2 asks the next question.
How do we determine whether the resulting behavior is safe enough to act?
The answer should not be “we tried it a few times.”
It should be a repeatable rehearsal discipline.
AI Rehearsal Is Simulation, Not Imagination
The human-learning analogy needs a firm boundary here.
Humans can mentally rehearse future actions without physically performing them. An AI agent does not need an equivalent psychological mechanism for the architectural principle to remain useful.
For AI systems, rehearsal means giving the real or representative agent application a controlled environment in which it can attempt realistic work.
That environment may contain simulated users, synthetic business records, test APIs, controlled tool implementations, policy engines, test identities, representative memory, retrieval data, and intentionally introduced failures.
The objective is not to ask whether the model can produce a plausible answer.
The objective is to observe what the complete agentic system does when reality becomes inconvenient.
That includes the model, orchestration, instructions, tools, identities, memory, retrieval, policy, approvals, application state, and surrounding control system.
Rehearsal Is Different From Benchmarking
Benchmarks are useful for comparing capabilities. They are usually insufficient for deciding whether a particular enterprise agent should receive authority inside a particular environment.
A general benchmark might tell you whether a model is improving at reasoning, coding, tool use, or another capability.
It cannot tell you whether your change-management agent will correctly stop when your organization’s rollback policy is violated.
That requires your environment, your policy, your tools, and your expected outcomes.
The distinction looks like this:
| Evaluation question | Benchmarking | Enterprise rehearsal |
|---|---|---|
| What is being evaluated? | General model or agent capability | A specific production-oriented agent system |
| Environment | Standardized benchmark | Representative enterprise environment |
| Tools | Benchmark-defined | Production-like tools and schemas |
| Policy | Generic or benchmark-specific | Organization-specific rules |
| Success | Benchmark score | Business outcome plus control compliance |
| Failure handling | Often secondary | Explicitly tested |
| Authority | Usually simulated | Designed around intended production authority |
| Release decision | Comparative signal | Promotion or blocking evidence |
The purpose of rehearsal is therefore narrower and more operational.
Can this specific system behave acceptably under the conditions we reasonably expect it to encounter?
The Happy Path Is the Wrong Place to Stop
Most agent prototypes naturally begin with successful scenarios.
That makes sense during development. The team first needs to prove that the workflow is possible.
Production readiness starts when the team begins making the scenario worse.
A useful rehearsal suite should deliberately include conditions such as these:
| Scenario condition | What the rehearsal should reveal |
|---|---|
| Ambiguous request | Whether the agent asks for clarification instead of inventing missing intent |
| Stale source data | Whether the agent detects age or conflict before acting |
| Tool timeout | Whether it retries safely, changes strategy, or stops |
| Partial success | Whether it distinguishes incomplete execution from success |
| Policy conflict | Whether policy overrides user pressure |
| Approval denied | Whether the workflow stops before the side effect |
| Missing permission | Whether the agent escalates instead of finding an unauthorized workaround |
| Contradictory instructions | Which authority source wins |
| Untrusted retrieved content | Whether data can improperly steer tool behavior |
| Dependency failure | Whether the workflow degrades safely |
| Unexpected state | Whether the agent revalidates assumptions |
| Irreversible action | Whether stronger approval and validation controls are invoked |
The difficult cases are not edge decoration.
For agents with real authority, they are the test.
The AI Rehearsal Architecture
A production-oriented rehearsal environment should recreate enough of the operational system to expose meaningful behavior without allowing the test itself to damage production.
The following model separates the scenario, execution environment, observation layer, and promotion decision.
The critical point is the gate.
A successful final message is not sufficient evidence. The evaluation needs to inspect the outcome, the route taken to achieve it, the authority exercised, and the agent’s response when something went wrong.
Test Four Surfaces, Not One
A useful rehearsal model evaluates four different surfaces.
Outcome
Did the actual business or technical state reach the intended result?
If an agent says that a virtual machine was created, verify that the correct virtual machine exists with the expected configuration.
If it says a ticket was updated, inspect the ticketing system.
If it says a refund was processed, verify the transaction state.
The final sentence is evidence of what the agent said. It is not evidence of what happened.
Trajectory
How did the agent reach the outcome?
This includes tool selection, arguments, handoffs, intermediate state, retries, retrieval decisions, and workflow ordering.
An agent can produce the correct final state through an unacceptable path.
For example, it might query a system it was not supposed to access, expose data unnecessarily, skip a required validation, or make three unauthorized attempts before eventually finding a permitted route.
Outcome and trajectory are different controls.
Authority
Did the agent remain inside the authority it had actually been granted?
This is broader than safety filtering.
Authority testing should examine whether the agent respected identity boundaries, tool permissions, transaction limits, approval requirements, tenant isolation, data restrictions, and other enterprise controls.
A technically successful action can still be an authorization failure.
Recovery
What happened when the workflow stopped going according to plan?
Recovery testing should determine whether the agent can recognize partial completion, avoid duplicating actions, preserve diagnostic evidence, invoke rollback when appropriate, escalate to a human, or stop safely.
For consequential automation, failure handling is part of the feature.
Repeated Trials Matter Because Agents Are Not Deterministic Workflows
A traditional automation script normally follows a tightly specified control path.
An AI agent can select different approaches across repeated attempts at the same task.
That variability is one of its strengths. It is also one reason a single successful demonstration provides weak reliability evidence.
Anthropic’s current evaluation guidance explicitly distinguishes a task from a trial. A task defines the problem and success criteria. A trial is one attempt. Multiple trials are used because agent behavior can vary between attempts.
The architectural implication is important.
Do not record only:
This scenario passed.
Record:
This scenario passed 19 of 20 isolated trials, with one policy-compliant escalation and no unauthorized actions.
Those statements describe very different levels of evidence.
This is particularly important when an agent is allowed to modify state. A low-frequency dangerous path can be invisible in a small demonstration set.
Simulated Users Make the Test More Realistic
Many agent failures emerge only through interaction.
A customer does not always provide every required fact in the first sentence. An administrator may use the wrong terminology. A manager may pressure the agent to bypass procedure. A caller may change their request halfway through the workflow.
Static prompts can test some of these conditions, but simulated users can exercise the conversation dynamically.
Current agent evaluation systems increasingly support this approach. Google’s Agent Development Kit, for example, can use simulated users to drive multi-turn scenarios, including live voice evaluation. The simulator can be given a persona and goal while the agent must determine what questions to ask and how to progress the interaction.
The general pattern is vendor-neutral:
The value is not pretending the simulator is a real customer.
It is forcing the agent to work through missing information, misunderstanding, changing conditions, and conversational uncertainty before the same workflow reaches actual users.
The Environment Should Misbehave Too
Simulating the user is only half of the problem.
The infrastructure around the agent also needs to fail.
A rehearsal environment should be able to produce slow APIs, timeouts, stale records, rejected operations, missing resources, malformed responses, unavailable dependencies, changed state, and partial completion.
Google’s current enterprise agent evaluation tooling includes environment simulation specifically so teams can emulate conditions such as slow or failing backends without affecting production. Meta’s Agents Research Environments similarly emphasizes dynamic environments, ambiguity, noise, and temporal constraints as important evaluation conditions.
The broader architecture principle is more important than either implementation:
Do not evaluate an agent only against a cooperative environment.
The environment should occasionally disagree with the plan.
That is how the rehearsal discovers whether the agent understands the difference between an expected response and actual state.
Adversarial Rehearsal Tests the Boundary
Normal failure testing asks what happens when something breaks.
Adversarial testing asks what happens when something actively tries to make the agent behave incorrectly.
The attack may come from user input, retrieved content, a tool response, a document, a compromised source system, or another agent.
Microsoft Foundry’s current AI red-teaming capabilities, for example, include automated adversarial probing of model and application endpoints. The implementation is vendor-specific, but the control objective applies broadly.
Before an agent receives meaningful authority, rehearsal should determine whether hostile or misleading input can cause it to cross the authority boundary.
This is especially important for agents that can write to systems.
An agent that only produces advisory text has one risk profile. An agent that can invoke production APIs, spend money, change firewall policy, communicate externally, or modify customer records has another.
The authority under test should determine the severity of the adversarial scenarios.
Rehearsal Should Mirror the Intended Authority
An important mistake is evaluating an agent in one environment and deploying it with significantly more authority in another.
Suppose the test environment allows an agent to call three read-only tools.
The production deployment then receives ten tools, including write access.
The organization has not promoted the tested system.
It has created a new one.
The rehearsal environment should therefore reproduce the intended tool surface, identity model, approval boundaries, policy rules, and important workflow dependencies as closely as practical.
This leads to a useful rule:
Never grant production authority that was absent from the rehearsal envelope.
If additional authority is required, that authority should create new rehearsal requirements.
The Rehearsal-to-Authority Ladder
Autonomy should not be treated as a binary switch.
A safer model promotes the agent through increasingly consequential authority levels as evidence improves.
This is a proposed DTD operating model rather than a vendor framework.
| Level | Agent authority | Example | Promotion evidence |
|---|---|---|---|
| 0 | Observe | Read telemetry and produce analysis | Accurate interpretation and no action |
| 1 | Recommend | Suggest a remediation or business decision | Quality, policy compliance, uncertainty handling |
| 2 | Prepare | Build the proposed change or transaction without committing it | Correct tool selection, arguments, validation |
| 3 | Act with approval | Execute only after an authorized human approves | Approval enforcement, correct execution, recovery |
| 4 | Bounded autonomy | Execute preapproved actions within defined limits | High reliability across normal, failure, and adversarial scenarios |
The progression is important because it separates capability from delegated authority.
An agent may be capable of performing a task before the organization has sufficient evidence to permit autonomous execution.
That is not wasted capability. It is governance working correctly.
Rehearsal Becomes a Release Gate
Once rehearsal exists, it should stop being a special event performed before the initial launch.
It becomes part of change management.
An agent can change behavior when the model changes, but Part 1 established that many other changes matter too.
Rehearsal should be triggered when a material behavior-shaping dependency changes.
| Change | Why rehearsal should be reconsidered |
|---|---|
| Model release | Planning, tool use, refusals, and behavior may change |
| System instructions | Decision criteria and workflow behavior may change |
| Tool schema | Available actions or required arguments changed |
| Permissions | Authority envelope changed |
| Retrieval corpus | Evidence available to the agent changed |
| Memory policy | Cross-session behavior may change |
| Business policy | Correct behavior has been redefined |
| Workflow topology | Handoffs or sequencing changed |
| Backend API | Error handling or state semantics may differ |
| Agent composition | A new specialist or orchestrator changes the system |
The release pipeline does not necessarily need to rerun every evaluation for every minor change.
It does need an explicit mapping between the changed layer and the rehearsal suites that provide relevant evidence.
A Practical Rehearsal Profile
The following YAML illustrates how an organization could declare the important requirements around a specific agent rehearsal.
It is intentionally vendor-neutral. The value is the control model, not the syntax.
agent_rehearsal_profile:
service: infrastructure-change-agent
version: 2026-09-07.1
target_authority:
level: 3
mode: execute_with_human_approval
environment:
isolation: required
production_credentials: prohibited
production_side_effects: prohibited
reset_state_before_each_trial: true
scenario_suites:
normal:
minimum_trials_per_case: 10
ambiguous:
minimum_trials_per_case: 10
dependency_failure:
minimum_trials_per_case: 20
adversarial:
minimum_trials_per_case: 20
required_controls:
identity_validation: true
policy_check_before_action: true
human_approval_before_write: true
post_action_validation: true
forbidden_behavior:
- bypass_approval
- use_unapproved_tool
- report_success_without_state_validation
- persist_untrusted_content_as_policy
evaluation_surfaces:
- outcome
- trajectory
- authority
- recovery
release_gate:
critical_control_failures_allowed: 0
unauthorized_side_effects_allowed: 0
failed_cases_require_trace_review: true
approval_owner: ai-platform-governance
In a real implementation, the reader should replace the service name, authority level, scenario counts, controls, and release thresholds with values derived from the actual workload and risk model.
Success means more than the YAML parsing correctly. The organization should be able to run the referenced scenarios, capture the resulting traces and state changes, enforce the declared critical controls, and prevent promotion when the release gate fails.
A weak implementation is one in which the policy file exists but the evaluation infrastructure cannot actually prove whether the agent bypassed approval or modified prohibited state.
Evaluate the Evaluator
There is another important failure mode.
The rehearsal system can be wrong.
A grader may reward the wrong outcome. A simulated environment may be unrealistic. A scenario may be impossible. An LLM judge may interpret a rubric inconsistently. A hidden evaluation case may accidentally appear in the agent’s retrieval system. A test may claim success because the final response looked correct while the underlying transaction failed.
This means the evaluation system itself requires validation.
Anthropic’s current guidance recommends combining different grader types, including deterministic checks, model-based graders, and human evaluation. Microsoft Foundry similarly supports rubric-based evaluation alongside additional safety and quality evaluators. Google’s current tooling combines code-based and model-based metrics and can inspect tool use and trajectories.
The pattern is converging for a reason.
No single evaluation method captures everything.
Use deterministic checks when the state can be verified precisely. Use model-based judgment when quality or conversational behavior requires interpretation. Use human experts to calibrate important subjective criteria and review unusual failures.
Most importantly, inspect failed and surprising traces rather than relying only on an aggregate score.
The Tooling Ecosystem Is Moving Toward Rehearsal
The major agent platforms are increasingly turning these ideas into product capabilities.
As of September 2026, the direction is visible across the ecosystem:
| Source | Relevant capability or research direction |
|---|---|
| Anthropic | Multi-turn agent evaluations using tasks, trials, traces, outcomes, and mixed grader types |
| Microsoft Foundry | Agent-targeted rubric evaluation, built-in risk evaluators, trace-oriented workflows, and adversarial red teaming |
| Google Gemini Enterprise Agent Platform | Generally available agent and model evaluations covering quality, safety, grounding, tool use, trajectories, user simulation, and environment simulation |
| Google ADK | Simulated users for multi-turn text and live voice agent evaluation |
| Meta ARE | Research environments designed around dynamic state, ambiguity, noise, collaboration, and temporal constraints |
| tau-bench research | Simulated users, domain policies, tools, database end-state verification, and repeated-trial reliability |
The important architectural observation is not which vendor has the longest feature list.
It is that agent evaluation is moving away from asking only, “Was the final answer good?”
The system increasingly needs to know what happened during the task and what state existed afterward.
That is rehearsal becoming an engineering discipline.
Rehearsal Does Not Eliminate Production Risk
A controlled environment cannot perfectly reproduce reality.
This needs to remain explicit.
| Limitation | Operational implication |
|---|---|
| Simulated users differ from real users | Production conversations still require monitoring |
| Test backends simplify production systems | Hidden integration failures can remain |
| Scenario libraries are finite | Novel combinations will occur |
| Graders can be wrong | Important decisions require calibration and review |
| Agent behavior is variable | Passing trials do not guarantee every future run |
| Business policy changes | Previously correct behavior can become incorrect |
| Model capabilities change | Old evaluation evidence can become stale |
| Production scale differs from testing | Latency, concurrency, cost, and race conditions may emerge |
Rehearsal therefore does not prove that the agent cannot fail.
It creates evidence that the organization deliberately tested the behaviors it considers important before exposing real systems to them.
Production observability, incident management, rollback, human escalation, and ongoing evaluation remain necessary.
The Real Promotion Question
The wrong question is:
Is this model powerful enough to automate the workflow?
The better question is:
Has this complete agentic system demonstrated that it can operate inside the authority we plan to grant it?
That reframes the production discussion.
Capability tells us what might be possible.
Rehearsal tells us what the system has demonstrated under controlled conditions.
Authority determines what the organization will actually allow.
The distinction matters because the most capable agent is not automatically the agent that should receive the most authority.
Authority should follow evidence and recoverability.
Operational Implications for Enterprise AI Teams
Rehearsal changes several ownership boundaries.
The AI platform team can build the harness and runtime environment, but it cannot define business success alone. Domain owners need to define valid outcomes and important exceptions. Security must identify adversarial conditions and unacceptable authority crossings. Platform and application teams need to build realistic tool simulators and resettable test state. Governance needs to define promotion thresholds and exception processes.
The agent’s production owner then needs evidence that those parties agreed on what “ready” means before deployment.
This turns evaluation from a model-team activity into a shared operating discipline.
It also creates an important audit artifact.
Instead of saying, “We tested the agent,” the organization can show which version was tested, what authority was represented, which scenarios ran, what controls were evaluated, which failures occurred, which corrections were made, and why the resulting evidence justified the production authority granted.
That is a much stronger answer.
Conclusion
AI agents should not receive authority because a demonstration looked good.
They should receive authority because a repeatable rehearsal process has exposed them to representative normal conditions, ambiguity, dependency failure, policy conflicts, adversarial pressure, and recovery scenarios, and because the resulting evidence shows that the complete system behaves acceptably within a defined boundary.
For AI, rehearsal is not a psychological metaphor. It is an engineering control.
Scenario simulation creates the situation. Evaluation makes behavior visible. Failure injection tests resilience. Adversarial testing challenges the boundary. Repeated trials measure reliability. A promotion gate converts that evidence into bounded authority.
The resulting rule is simple:
Rehearse before you automate. Correct before you promote. Grant only the authority the evidence supports.
That still leaves one difficult problem.
Once the agent reaches production, it begins generating real outcomes, real traces, real user reactions, and real operational feedback. Some of that information should improve the system. Some of it absolutely should not.
Part 3 of the series addresses that boundary: Feedback Is the Control Plane: Preventing AI Systems From Repeating the Wrong Behavior.
External References
- Anthropic: Demystifying evals for AI agents
Canonical URL: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents - NIST: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
Canonical URL: https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence - Microsoft Learn: Evaluate your AI agents
Canonical URL: https://learn.microsoft.com/en-us/azure/foundry/observability/how-to/evaluate-agent - Microsoft Learn: Run AI Red Teaming Agent Locally (Azure AI Evaluation SDK)
Canonical URL: https://learn.microsoft.com/en-us/azure/foundry/how-to/develop/run-scans-ai-red-teaming-agent - Google Developers Blog: Agent and Model Evaluations in Gemini Enterprise Agent Platform are now GA
Canonical URL: https://developers.googleblog.com/agent-and-model-evaluations-in-gemini-enterprise-agent-platform-are-now-ga/ - Google Developers Blog: How to Evaluate Live & Voice Agents in ADK
Canonical URL: https://developers.googleblog.com/how-to-evaluate-live-voice-agents-in-adk/ - Meta AI: ARE: scaling up agent environments and evaluations
Canonical URL: https://ai.meta.com/research/publications/are-scaling-up-agent-environments-and-evaluations/ - ICLR: tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Canonical URL: https://proceedings.iclr.cc/paper_files/paper/2025/hash/1b126cc38b8638e07bef37e7b2bb72bf-Abstract-Conference.html
TL;DR AI systems generate enormous amounts of feedback: user ratings, evaluation scores, tool results, policy decisions, memory writes, retrieval outcomes, business metrics,...