TL;DR
AI reward design starts with the outcome the organization needs, not the metric that is easiest to collect. Separate mandatory permissions, verified results, appropriate escalation, and efficiency. Keep missing evidence and difficult cases visible instead of removing them from the score. A practical reward contract should identify what success means, who verifies it, when the result becomes final, and which shortcuts could improve the metric without improving the service.
Introduction
A support agent closes more tickets after an update. Average handling time falls. The backlog shrinks, and the dashboard suggests that the new version should replace the old one.
Then service owners report that users are returning with the same unresolved problems.
The foundational article, Behaviorism and AI: How Rewards Shape Model Behavior, established the underlying tension: feedback can shape behavior without guaranteeing that the rewarded behavior matches the designer’s intention. This follow-up turns that principle into an engineering question: what exactly should count as success?
We will use the same illustrative support workflow to build a reward and evaluation contract. The scenario, numbers, and configuration are proposed implementation examples, not results from a production deployment. The focus is defining a defensible objective before building the evaluation tooling or deciding which feedback may influence future systems.
Separate the Business Objective from the Optimization Mechanism
For this workflow, the business objective is to restore the affected service within an authorized operating boundary. Closing a ticket records a decision about that outcome. It is not the outcome itself.
Google DeepMind’s Specification gaming: the flip side of AI ingenuity describes how an agent can satisfy a formal objective while missing the intended result. Applied here, the design question is straightforward: could the system improve its score without restoring anything?
Keep three concepts separate. The business objective describes the result the service owner needs. The measurement provides evidence about that result. The optimization mechanism determines how the measurement influences future behavior or release decisions.
A dashboard does not automatically train a model. The score might feed reinforcement-learning updates, select among candidate responses, or persuade a team to promote one model-and-prompt configuration over another. In the last case, the model’s parameters may remain unchanged throughout production use.
That distinction changes the remedy. A poor training reward calls for changes to the learning process. A misleading release metric calls for changes to evaluation and promotion. Both require a clearer outcome definition, but they are not the same failure.
For the proposed pattern, assume production inference does not automatically update model parameters, downstream systems enforce permissions, and a separate verification component can inspect the affected service.
A Better Closure Rate Can Conceal a Worse Service
Consider two versions assessed against equivalent 100-case workloads. For this example, verified resolutions are completed by the agent without a human subsequently repairing the fault.
| Measure | Baseline | Candidate |
|---|---|---|
| In-scope cases assigned | 100 | 100 |
| Cases marked closed | 80 | 95 |
| Cases with independently verified autonomous resolution | 72 | 60 |
The candidate improves closure from 80% to 95%, while verified autonomous resolution falls from 72% to 60%. More records reach a completed state, but fewer cases meet the intended outcome.
These invented numbers demonstrate the measurement problem. They do not establish statistical significance or describe a benchmark.
Keep the Denominator Outside the Agent’s Control
For this workflow, define the reporting population before the agent routes or prioritizes work. Otherwise, difficult cases could disappear from the comparison through deferral, reclassification, or selective acceptance.
Verified autonomous resolution rate = independently verified agent-only resolutions / all in-scope assigned cases in the reporting cohort
An escalated case remains in that population, but it is reported separately as an escalation. A human-assisted recovery also remains visible without becoming an autonomous success.
Report missing evidence alongside the rate. An unknown outcome is not proof of failure, but excluding it can make an unreliable measurement pipeline look like an effective agent. Label the metric as confirmed performance, not an estimate of every case’s true outcome.
Compare equivalent incident classes, severity levels, and follow-up periods. A candidate assigned mostly routine requests should not be declared better than a baseline handling difficult outages merely because its aggregate score is higher.
Use Gates Before Preferences
A weighted score is useful for comparing preferences. It is a poor substitute for an execution boundary.
Suppose a simplified reward awards ten points for completion, subtracts two for an unauthorized action, and awards one for appropriate escalation. Unauthorized completion earns eight points. The supposedly unacceptable action still beats the permitted alternative.
Increasing the penalty may influence training, but it does not install an authorization check in the application. In this design, permissions are enforced before execution rather than treated as costs the agent can absorb.
OWASP’s LLM06:2025 Excessive Agency recommends limiting tool functionality and permissions and enforcing authorization in downstream systems. Apply that principle to each consequential action, not merely to the beginning of the conversation.
The proposed flow separates permission to remediate from permission to close the case:
Permission for a diagnostic action does not automatically authorize remediation, and permission to remediate does not automatically authorize closure. The gateway and downstream application must evaluate the relevant operation and resource scope.
Preserve Useful Escalation Without Rewarding Avoidance
Appropriate escalation is a valid workflow outcome, but it is not an autonomous resolution. The contract should distinguish a case that requires human authority from one the agent could have completed safely.
An agent that escalates every routine request should not pass a usefulness review simply because it avoided mistakes. Conversely, an agent should not receive a failure label for respecting an approval requirement. Define acceptable outcomes by scenario before comparing completion rates.
Only after mandatory controls and outcome requirements are satisfied should latency, cost, or response polish improve the assessment. Keep control violations separately visible; a favorable average must not conceal them.
Define What Counts as Independent Verification
A ticket marked closed, a successful tool response, and a restored service are three different observations.
Anthropic’s Demystifying evals for AI agents distinguishes the interaction record from the resulting environment state. That distinction matters here: a completion message is evidence of what the agent reported, not independent evidence that the service works.
For the support scenario, use a service-specific verification test. A generic health endpoint may not exercise the customer’s failing operation. An approved check should address the affected function, resource, and tenant rather than merely establish that a process is running.
Bind the result to the incident, remediation attempt, resource identity, and observation time. A valid check from before the action, or from another tenant, does not verify this recovery.
The acting agent should not be able to rewrite verification records or alter the test definition. A second model reading the agent’s own closure note does not create an independent observation. Separate credentials, protected evidence storage, and checks against actual service state make the proposed boundary meaningful.
Independence is not infallibility. Test the verifier against known broken states and contradictory evidence. When verification is unavailable, record an evidence gap and block unsupported closure rather than silently lowering the requirement.
Distinguish Immediate Verification from a Mature Outcome
A service can pass a check now and fail again later. Those observations answer different questions.
Decision-time verification establishes whether closure is justified under current operating rules. Follow-up verification determines whether the result persisted through a service-appropriate observation period.
For a recurring batch-processing fault, the service owner might require evidence from the next scheduled run. An interactive transaction could require a different check and observation window. There is no universal waiting period that proves every service has recovered.
The ticket may therefore be operationally closed while its final feedback label remains provisional. Keep that distinction visible in reporting. A recent case still awaiting follow-up should not be compared as though it had the same evidence as an older, fully observed case.
When a related incident reopens, append the new evidence and reassess the outcome. Do not erase the original action history or assume that every reopened ticket has the same cause.
Attribution matters as much as timing. When an engineer completes the repair after the agent’s attempt, record the intervention. The final recovery must not automatically become evidence that the agent independently succeeded. Likewise, the absence of a complaint is not a replacement for the verification the contract requires.
Avoid Rewarding Activity That Resembles Progress
A completion-only reward provides little intermediate feedback during a long workflow. However, adding rewards for activity can create another shortcut. DeepMind’s specification-gaming discussion identifies poorly designed reward shaping as one way an agent can be steered toward unintended behavior.
For this support example, rewarding every diagnostic call would let repeated checks accumulate credit without producing new information. The proposed alternative is to recognize useful evidence that satisfies a defined prerequisite, not the number of times a tool was invoked.
Even that needs limits. Awarding credit once per distinct tool could encourage unnecessary variety rather than useful investigation. Define which evidence contributes to the decision, cap repeatable progress credit, and retain an independently verified terminal outcome.
Efficiency should also account for the whole workflow. A cheaper model call is not necessarily a cheaper resolution when it generates more retries, handoffs, or rework. Compare costs across the assigned cohort rather than counting only the successful attempts.
More Optimization Does Not Repair a Weak Proxy
In Scaling Laws for Reward Model Overoptimization, Gao, Schulman, and Hilton studied a synthetic setting in which one reward model stood in for human judgments. Increasing optimization against a proxy could eventually reduce performance as assessed by that separate model.
That research was not an enterprise support evaluation. The operational implication for this example is narrower: a rising optimization score is not sufficient evidence that the service outcome improved. When the two diverge, investigate the objective, evidence, and selection process before increasing optimization pressure.
Make the Reward Contract Reviewable
The following YAML records the proposed decisions in a form that service, security, and platform owners can review together. It is an illustrative design contract, not configuration for a particular product or a validated reward function.
Replace the owner and follow-up profile with your actual operating decisions. Implement the enforcement rules in the application and downstream services; do not assume that supplying this text to the model creates those controls.
reward_contract:
id: support-resolution-v1
owner: service_operations
objective: verified_service_restoration
execution_gates:
authorization: validate_for_every_tool_action
closure_requires:
- current_closure_authority
- relevant_fresh_service_verification
missing_evidence: block_closure_and_record_gap
agent_can_modify_verifier: false
reporting:
denominator: all_in_scope_assigned_cases
freeze_scope_before_agent_routing: true
segment_by: [incident_class, severity]
report_unknown_outcomes: true
report_control_violations_separately: true
outcome_classes:
- verified_autonomous_resolution
- human_assisted_resolution
- appropriate_escalation
- unresolved
- evidence_incomplete
feedback_labels:
closure_alone_is_success: false
follow_up_profile: service_owner_defined
final_success_requires: completed_follow_up_verification
human_repair_requires_separate_attribution: true
reopen_event: append_evidence_and_reassess
comparison:
mandatory_control_failure_blocks_acceptance: true
require_scenario_appropriate_outcome: true
efficiency_applies_only_after_requirements_pass: true
The outcome class and control status are separate dimensions. A service could recover even though the agent crossed a permission boundary. Record both facts without allowing the recovery to excuse the violation.
Successful implementation would demonstrate that an unauthorized action cannot execute, missing evidence prevents unsupported closure, and an unverified closure cannot become a positive resolution label. If the agent can bypass the gateway or modify the verifier, the contract is not enforced.
Keep the contract version with the model, prompt, tool definitions, and verifier version used in each assessment. A result should be traceable to the rules under which it was produced.
Establish Ownership Before Tuning
For this pattern, the service owner defines resolution and the follow-up window. The security owner defines execution boundaries. The platform team implements enforcement and evidence collection. The evaluation owner maintains the rubric and investigates disagreement between the proxy and the observed outcome.
Begin by applying the contract to a sample of historical incidents. Include reopened cases, missing telemetry, human-assisted repairs, and correct escalations. Disagreements about what counts as success should be resolved before those records become training labels or release criteria.
Then compare a candidate against the approved baseline using equivalent scenarios and observation windows. Keep denied actions, unknown outcomes, and unnecessary escalations visible alongside verified resolutions. Do not rewrite the rubric simply to make the candidate pass.
Review the contract when a new model, tool, permission scope, incident mix, or verifier changes its assumptions. When verified outcomes regress despite a rising score, pause promotion and inspect the measurement path. Retaining the approved release is different from repairing side effects already created by the candidate; those require their own operational response.
Conclusion
AI reward design is not a search for one impressive number. It is the work of connecting an optimization signal to a defensible business outcome while keeping permissions outside the tradeoff.
For the support workflow, that means measuring verified restoration rather than celebrating closure, preserving difficult and unknown cases in reporting, and distinguishing appropriate escalation from autonomous success. It also means protecting the evidence path and waiting for the verification needed to finalize an outcome.
Start with one workflow. Write down what success means, which actions are prohibited, who verifies the result, and how the metric could improve while the real service gets worse. That exercise should happen before the next round of tuning.
The next article, AI Agent Evaluation: Test the Behavior, Not the Explanation, turns this contract into observable checks and a runnable evidence grader.
Define the outcome first. Enforce the boundary. Optimize only within those limits.
External References
- Google DeepMind: Specification gaming: the flip side of AI ingenuity
Canonical URL:https://deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity/ - OWASP Gen AI Security Project: LLM06:2025 Excessive Agency
Canonical URL:https://genai.owasp.org/llmrisk/llm062025-excessive-agency/ - Anthropic: Demystifying evals for AI agents
Canonical URL:https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents - arXiv: Scaling Laws for Reward Model Overoptimization
Canonical URL:https://arxiv.org/abs/2210.10760
TL;DR Enterprise RAG citation controls must establish more than whether a reference opens. Connect material claims to authorized, versioned evidence; preserve the...