TL;DR
AI agent stability depends on controlling the next intervention, not simply limiting model retries. Separate repeated delivery of the same operation from reconciliation and newly authorized recovery. Preserve action identity, execution state, and budgets across restarts. Wait for observable effects instead of relying on timers alone, and keep ownership of each operational decision explicit. A failed verification result should open a controlled recovery path, not give the agent permission to keep changing the environment until something looks healthy.
Introduction
The verifier correctly reports that the catalog-api rollout failed its service acceptance check. The agent requests an approved fallback release, but the request times out. It tries again, then proposes a restart because the dashboard still looks unhealthy.
The fallback was already accepted. Its replacement instances are still starting. The agent is preparing another change before it understands the first one.
Part 1, AI Agent Verification: Prove the Outcome, Not the Tool Call, established the evidence contract. The runtime distinguishes PASS, FAIL, PENDING, and UNKNOWN rather than treating every successful tool response as a successful outcome. This installment addresses what those observations are allowed to trigger.
We continue the hypothetical staging rollout, adding one separately approved recovery option: restore an explicitly identified, compatible release through the designated deployment owner. Database changes, arbitrary restarts, direct replica changes, and permission expansion remain outside the agent’s scope.
The execution policy and scenarios below are proposed design patterns, not a deployed system or a formal stability proof. They extend the foundation article’s treatment of delayed feedback, overcorrection, and competing controllers into runtime requirements.
Detecting a failed outcome does not make the next action safe. It makes the next decision necessary.
Separate Retries, Reconciliation, and Recovery
“Try again” conceals three different operations. Give them different meanings in the workflow before adding another tool.
| Operation | Purpose | What must remain controlled |
|---|---|---|
| Transport retry | Deliver the same logical request again after a delivery problem | Original intent, parameters, identity, and retry-safety contract |
| Reconciliation | Establish whether the original operation was accepted or applied | Authoritative readback, operation correlation, and observation limits |
| Recovery action | Change the target because the previous result is unacceptable | New action identity, current authorization, scope, and recovery criteria |
A timeout belongs first to the execution question: what happened to that request? A completed rollout that misses its error-rate target belongs to the outcome question: what intervention is justified now?
A genuine rejection needs its own classification. Repeating a request with invalid parameters or insufficient permission is not a recovery strategy. Changing those parameters is a new proposal, not an invisible repair to the original request.
For this design, Part 1’s verdicts retain their meanings. PASS completes the scoped task. PENDING permits bounded observation. UNKNOWN blocks dependent writes while the runtime reconciles or escalates. FAIL opens recovery review, but does not itself authorize rollback, restart, or scale-out.
Kubernetes provides a useful boundary: a Deployment revision changes when its Pod template changes. Resubmitting an unchanged template is not equivalent to inserting a fresh restart annotation into that template. The latter changes the desired state and can trigger another rollout.
An agent must not disguise a new intervention as a retry merely because its goal remains “make the service healthy.”
Put the Execution Gate Between Intent and Effect
The planner proposes an intent. The runtime decides whether that intent can become an executable request.
In this architecture, the gate checks current authority, target identity, competing work, remaining budget, and the state of the previous operation. It then hands the approved intent to the designated deployment owner. The agent does not receive an alternative credential path around that boundary.
The diagram shows why a retry should return to the existing action record rather than start a new planning cycle with an empty history.
These are logical responsibilities, not a requirement to deploy seven new services. However, their enforcement must survive a model session ending, a worker restarting, or an approval being withdrawn.
Idempotency Belongs to the Operation Contract
Amazon’s Making retries safe with idempotent APIs describes caller-provided identifiers that let a service recognize repeated requests expressing the same intent. Its discussion of changed parameters and late-arriving requests also establishes why the exact duplicate-handling contract matters.
For this runtime, persist an action identifier and payload digest before dispatch. Reuse the identifier only for delivery attempts with the same semantic payload. A fallback to another release receives a new identifier because it expresses different intent.
Check that duplicate-handling retention covers the permitted replay horizon, including worker restarts and delayed delivery. Expiration is not permission to mint a replacement ID.
None of this creates a guarantee merely by recording an identifier locally. The receiving service or adapter must enforce duplicate handling, including its downstream effects.
The custom action ID here is not a universal idempotency header that can simply be attached to Kubernetes requests. The adapter must use supported operation semantics, conditional writes, and readback. Where an ambiguous write cannot be reconciled or safely repeated, hold it for review.
Idempotency limits the consequences of repetition. It does not establish that the original action was correct, authorized, or harmless.
Bound Retry Pressure Across the Entire Call Path
AWS’s reliability guidance recommends bounded retries, exponential backoff, jitter, and a deliberate retry layer. It warns that independently retrying at multiple layers can compound attempts and consume additional resources during a failure.
Consider an illustrative call path where every layer permits three total attempts, including its initial attempt. If every attempt reaches the next layer and fails, the multiplication is:
Planner wrapper 3 attempts
Workflow wrapper x 3 attempts
Client adapter x 3 attempts
------------
27 downstream attempts
The planner may describe three attempts while the target receives 27. This is a worst-case arithmetic example, not a measured incident.
For the proposed workflow, designate one retry owner for each remote-call boundary and account for every attempt it can produce. Configure other wrappers so they do not independently multiply that allowance. Native platform reconciliation is a separate control loop, not another delivery retry to hide inside the same counter.
Backoff spaces eligible attempts. Jitter varies their timing so clients are less likely to retry together. The AWS Architecture Blog explains this distinction in Exponential Backoff And Jitter.
Neither mechanism answers whether an operation may be repeated. A delayed duplicate with unsafe semantics is still a duplicate. Keep retry eligibility, attempt limits, and timing as separate decisions.
Persist the Uncertainty, Not Just the Last Response
The execution ledger should distinguish an intent that is prepared, dispatched, acknowledged, applied, and verified. Store an explicit unresolved-execution state when the available evidence cannot establish what happened.
An acknowledged request can remain incomplete. A verified service failure can follow a successfully applied change. Collapsing those conditions into failed invites the wrong response.
For this design, create the action record and reserve its budget atomically before dispatch. Record the target, expected release, current authorization, payload digest, and operation deadline. A replacement worker resumes that record rather than asking the model to reconstruct events from conversation history.
The Crash Window Still Exists
Suppose the target accepts a change, but the worker crashes before recording the acknowledgment. The ledger proves intent, not nonexecution. The replacement worker must reconcile the target or use a verified safe-retry contract.
Do not reset the budget or allocate a new action ID just because the response was lost. Equally, do not assume the absence of a target-side log proves that no change occurred.
Duplicate workers require another control. Imagine an old worker resuming after its ownership lease expires while a replacement is already active. The design needs stale-owner rejection at the write boundary, not just a lock that cooperative workers are expected to respect.
That requirement is often called fencing. An ownership token written into a log does not implement it. The execution path must reject obsolete owners and account for requests already accepted downstream. Where that cannot be demonstrated, autonomous takeover is outside this pattern’s supported scope.
Wait for Effects, Not Merely for the Cooldown
Part 1 required a complete five-minute service window after verified convergence. A two-minute timer after request submission cannot replace that evidence.
Suppose a rollout request is dispatched at 10:00. Runtime convergence is verified at 10:03, so the required observation window ends at 10:08. A delivery retry delay measured in seconds, a mutation cooldown, and the service verification window serve different purposes.
For ordinary corrective decisions, require both timing and evidence conditions. The previous execution must be understood, its required observation must be available, and the next action must remain authorized. Expiration of a timer satisfies only the timing condition it was designed to enforce.
Damping Is Not the Same as Hiding Failure
For controllers that adjust quantities such as capacity, small permitted steps, rate limits, and distinct entry and exit conditions can reduce unnecessary reversals. Hysteresis means the condition that initiates a change differs from the condition that reverses it.
Kubernetes Horizontal Pod Autoscaling provides a concrete reference through scaling policies and stabilization windows. Those mechanisms regulate scaling behavior; they are not substitutes for the service-outcome contract in this series.
For a release workflow, the corresponding decision is not to keep alternating between a failed release and its fallback. Once the fallback is selected, this example prohibits automatic re-promotion of the failed release.
Keep critical stop conditions separate from ordinary waiting rules. A verified hazard should reach the approved incident path immediately. Do not wait for a completed latency window when an independently defined containment condition already applies.
Give Each Decision One Accountable Owner
“One controller” is too broad a rule for a layered platform. What matters is that independent actors do not compete to own the same desired-state decision.
For catalog-api, the release owner selects the approved artifact. The platform’s deployment machinery reconciles that desired state. An existing autoscaler may separately own replica count. The agent should not bypass those responsibilities by directly editing whatever setting appears convenient.
GitOps makes the conflict especially visible. Argo CD’s Automated Sync Policy documents synchronization between declared and live state, including optional self-healing. With that behavior enabled, a direct live rollback can conflict with the release still declared in Git.
For a GitOps implementation of this pattern, recovery must update the authoritative desired state through its approved path, or use a coordinated incident procedure that explicitly transfers control. A direct cluster edit is not a complete recovery plan while another controller still intends to restore the failed release.
Concurrent administrative changes also need detection. Kubernetes API documentation describes conditional updates using resourceVersion and conflict responses when the supplied version is stale.
Treat such a conflict as a reason to reread and reassess. Do not remove the precondition or force an overwrite merely to obtain a successful response. Version checks detect stale state; they do not independently prove current authorization or resolve controller ownership.
Limit Consequences, Not Just Request Counts
One API call can initiate a rollout across the entire service. An action limit therefore needs a permitted-impact definition as well as an integer.
For this example, bind the resource identity, approved release artifacts, allowable template changes, and rollout strategy before execution. Do not let the planner increase disruption limits, expand the namespace scope, or change unrelated configuration to make an action complete.
Define an episode as the controlled attempt to deliver one approved service change, including its permitted recovery. It is not a chat session. Repeated alerts for that same unresolved change join the episode rather than creating new action allowances.
Use separate limits for new change intents, delivery attempts, elapsed time, and concurrent affected services. Preserve them across restarts. At fleet scale, also bound simultaneous changes sharing a dependency; individually bounded agents should not collectively saturate it.
These controls limit particular failure mechanisms. They do not prove that the service will converge or remain available. If the permitted response set cannot resolve the condition, the correct terminal behavior is a controlled handoff, not self-expanded authority.
An Execution Policy for the Staging Rollout
The YAML below expresses the proposed operating contract. It is not a Kubernetes manifest, an Argo CD configuration, or a deployable vendor schema.
The example allows one approved deployment and at most one separately authorized restoration within the same episode. The numbers are illustrative. Replace them with limits supported by your deployment measurements, service objectives, and recovery tests.
policy_id: catalog-rollout-execution-v1
scope:
environment: staging
service: catalog-api
resource_uid: bind_before_execution
release_owner: approved-release-controller
rollout_profile: catalog-staging-rollout-v1
authority:
allowed_intents:
- deploy_approved_release
- restore_approved_release
bind_release_and_parameters: true
revalidate_before_each_dispatch: true
other_mutations: deny
episode:
key: bind_to_approved_service_change
max_change_intents: 2
max_deploy_intents: 1
max_restore_intents: 1
max_elapsed_seconds: 2400
restart_resets_budget: false
on_expiry: block_new_dispatches_and_handoff
execution:
durable_record_before_dispatch: true
atomic_budget_reservation: true
max_active_release_intents_per_service: 1
require_target_precondition: true
require_stale_owner_rejection: true
on_ambiguous_result: reconcile_then_hold_if_unknown
transport_retry:
owner: release_adapter
max_attempts_per_intent: 3 # Includes the initial attempt.
preserve_action_id_and_payload: true
require_verified_retry_safety: true
backoff_profile: bounded_jitter_v1
verification:
contract_id: catalog-rollout-verification-v1
on_pending: bounded_read_only_observation
on_unknown: hold_dependent_writes_and_escalate
on_fail: recovery_review
on_pass: close_current_intent_with_evidence
recovery:
target_release: bind_approved_compatible_fallback
require_new_intent_and_current_authorization: true
supersession: coordinated_by_release_owner
automatic_repromotion: deny
The rollout profile must resolve to validated constraints on the actual change. The backoff profile must define delay bounds, retryable conditions, and timeout handling. Neither may remain an unexplained label in the implementation.
The three-attempt allowance covers repeated delivery at the declared adapter boundary. Any internal service calls need their own reviewed limits. Read-only reconciliation also needs rate and duration limits even though it does not consume a new change intent.
The one-active-intent rule does not forbid an authorized recovery from superseding a known failed or stalled rollout. It requires the release owner to coordinate that transition, preserve the previous state, and prevent both intents from continuing to compete. Ambiguous execution is not silently cleared to admit the fallback.
Part 1’s verification contract applies separately to each accepted change, rebound to that change’s approved release. The original 20-minute verification deadline remains intact, subject to the remaining 40-minute episode budget. Do not shorten an acceptance window to fit the clock. When insufficient time remains, hand off through the incident process rather than manufacturing a pass.
Successful enforcement means an out-of-scope restart, stale worker, exhausted budget, or unauthorized fallback is rejected at execution. Parsing this YAML proves none of those controls.
Walk Through the Recovery Without Multiplying It
Return to Part 1’s worked failure: the intended release is active, but its measured error ratio exceeds the contract limit. That is a known service failure, not an ambiguous deployment request.
The service owner approves the compatible fallback. The runtime admits a new restoration intent, reserves the second change allowance, and dispatches it. The target accepts the request, but the response is lost and the worker restarts.
The replacement worker recovers the existing restoration record. It does not open a third intent. Authoritative reconciliation establishes that the fallback is already desired and still converging, so the runtime continues read-only observation.
After convergence, a new full verification window evaluates the fallback. A PASS closes the restoration as verified recovery while retaining the failed original deployment. It does not rewrite the release history to say that the initial change succeeded.
A FAIL or UNKNOWN after that recovery follows the escalation path. The agent cannot bounce back to the failed release, reset the episode, or substitute an arbitrary restart. Further intervention requires a new accountable decision about scope and risk.
Stopping and Rolling Back Are Different Operations
A stop control should block new agent-originated writes and prevent queued work from bypassing current authorization. It cannot be assumed to cancel operations already accepted by the target.
Kubernetes documents that ProgressDeadlineExceeded reports stalled progress rather than automatically performing a rollback. Likewise, ending the agent’s wait does not establish that the platform stopped reconciling its desired state.
The supervisor needs a record of accepted and unresolved operations, the current desired release, and the owner responsible for any continuing effects. Keep the stop path outside the model’s discretion, and test how it reaches queued workers and downstream credentials.
Rollback has a separate validity boundary. Kubernetes Deployment rollback restores the Pod template, not every effect of the application. Microsoft’s Compensating Transaction pattern similarly explains that compensation is application-specific, can fail, and may not restore the exact original state.
For catalog-api, the proposed fallback assumes compatible data and configuration. If a release changed a database schema or emitted irreversible external actions, restoring an older image is not sufficient evidence of recovery. That broader workflow needs its own containment or compensation design.
Test the Conditions That Would Create Another Incident
Test the execution boundary without relying on the planner to cooperate.
| Injected condition | Required result under this policy |
|---|---|
| Target accepts the request, response is lost | Reconcile or safely repeat the same intent; do not create another |
| Worker crashes after dispatch | Recover the record, deadline, and consumed budget |
| Old worker resumes after losing ownership | Reject its stale write path and reconcile any already accepted request |
| Another controller changes the desired release | Detect the conflict and reassess ownership before further action |
| Approval is revoked while a request waits in a queue | Prevent a newly unauthorized dispatch |
| Fallback fails verification | Preserve its result and escalate without automatic re-promotion |
These are proposed acceptance tests, not reported results. Run them across the actual gateway, persistence, credentials, adapter, and platform controller. A test confined to a model conversation cannot establish those properties.
Begin with read-only review of existing deployment episodes, then exercise the policy in staging with injected delivery failures and worker restarts. Include a successful recovery so the design is not judged only by how effectively it refuses work.
The service owner owns recovery criteria. The platform team owns the execution path and controller integration. The governance owner owns authority and exceptions. Track duplicate attempts, unresolved-operation duration, budget exhaustion, and service impact together. Fewer escalations are not an improvement if they come from hiding uncertainty or allowing more unreviewed changes.
Conclusion
AI agent stability requires more than a better retry prompt. The runtime must distinguish repeated delivery from a new intervention, retain uncertainty across failures, and enforce who owns the next change.
Start with one bounded workflow. Give it durable action identity, one accountable retry owner per call boundary, explicit recovery authority, and budgets that survive restarts. Require observable effects before ordinary correction, while preserving a separately authorized path for urgent containment.
Part 3, AI Agent Learning: Governing What Becomes Permanent, addresses the longer-lived loop: whether this recovery episode should change the knowledge, procedures, or policies used tomorrow.
A reliable agent does not keep acting until something works. It acts only while the system can justify the next step and contain its consequences.
External References
- Amazon Builders’ Library: Making retries safe with idempotent APIs
Canonical URL: https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/ - AWS: REL05-BP03 Control and limit retry calls
Canonical URL: https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_mitigate_interaction_failure_limit_retries.html - AWS Architecture Blog: Exponential Backoff And Jitter
Canonical URL: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ - Kubernetes: Deployments
Canonical URL: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ - Kubernetes: Kubernetes API Concepts
Canonical URL: https://kubernetes.io/docs/reference/using-api/api-concepts/ - Kubernetes: Horizontal Pod Autoscaling
Canonical URL: https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/ - Argo CD: Automated Sync Policy
Canonical URL: https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/ - Microsoft: Compensating Transaction pattern
Canonical URL: https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction
TL;DR AI agent verification must establish what happened in the target environment, not simply whether a tool returned successfully. Separate request acceptance,...