TL;DR
An execution ledger can select one worker without preventing that worker from acting after its authority has been superseded. Fencing stale AI workers requires an enforced boundary at the receiving system: an obsolete request must be rejected even when the worker resumes with previously valid credentials, approval data, and target preconditions.
This companion adds a local receiver-fencing lab to the preceding persistence exercise. Twenty-six local tests passed, and two demonstration runs rejected a paused worker before its replacement changed the target. The lab also preserves an important counterexample: restoring the receiver’s own obsolete database can restore obsolete authority. These are local protocol results, not production security or distributed-system validation.
A replacement worker is not safely in control until the previous worker’s requests can no longer produce the prohibited effect.
Introduction
Worker A claims an approved infrastructure change and prepares to execute it. Then the process pauses.
Its heartbeat expires. Operations starts Worker B. The dashboard now shows a healthy replacement and one current owner.
Worker A resumes.
It still has the request, a credential, and the information that made its action appear authorized before the pause. The target has not changed, so the original state preconditions may still hold. Unless the receiving path recognizes that A’s authority is obsolete, A can act before B makes its first change.
A durable claim record does not solve that problem by itself.
The previous companion, Building an AI Agent Execution Ledger That Survives Restarts, deliberately avoided automatic takeover. This installment examines what a stronger receiving boundary must establish before controlled resumption becomes defensible.
The reference design and lab are proposed engineering patterns. The executable example changes a local SQLite value, not a Kubernetes object or backup setting. Authentication, independent administration, and actual infrastructure integration remain outside its tested scope.
A Ledger Claim Is Not a Fence
A claim assigns work. A fence prevents an obsolete holder from exercising authority at a protected boundary.
Those responsibilities can cooperate without becoming interchangeable.
| Mechanism | Question it answers | What it does not establish alone |
|---|---|---|
| Execution claim | Which worker owns this recorded attempt? | Whether another process can still affect the target. |
| Lease or heartbeat | Has the coordinator recently heard from the worker? | Whether an unresponsive worker has stopped executing. |
| Leader election | Which participant should coordinate work? | Whether every other participant is prevented from acting. |
| Receiver fencing | Will the receiving boundary reject superseded execution authority? | Whether the current request is otherwise authorized or appropriate. |
Kubernetes’ client-go leader-election package explicitly documents that its implementation does not guarantee that only one client is acting as leader. That limitation matters when applications treat an elected leader as the sole protection around consequential writes.
The etcd documentation makes the external-resource boundary equally clear: an etcd lock does not automatically protect another system. The receiving resource needs the relevant validation and consistency mechanisms.
The trusted agent controller design remains the coordination foundation. Its ownership decision must reach an enforceable boundary rather than remain a fact known only to the controller.
An Old Worker Does Not Need to Be Malicious
A paused process, delayed request, interrupted network path, or restored queue can carry an earlier decision into a later operating state.
The design should reject that request because its authority is obsolete, not because a model identifies the worker as suspicious.
Treat agent reasoning as irrelevant to this particular control. A perfectly reasonable explanation from Worker A cannot make its superseded grant current again.
Define When the Fence Becomes Effective
For the proposed pattern, define a specific acceptance rule:
After the receiver commits a hold for a protected scope, requests using the superseded grant must not create new mutations within that scope.
This is stronger and more testable than “the old worker should stop.”
It also establishes a timing boundary. An operation that committed before the hold is not undone by the hold. An asynchronous operation already accepted elsewhere may still require cancellation or reconciliation.
The receiver needs a governed generation, sometimes called an epoch, that distinguishes successive authority states. The value must not be selected by the agent.
In this lab, the receiver requires an exact match to its currently installed generation and active grant. A larger number is not automatically better authority.
Do Not Wait for the Replacement’s First Write
One fencing design remembers the highest generation seen on accepted requests and rejects lower generations afterward. That can prevent old requests from following a newer accepted request.
It does not necessarily reject an old request that arrives before the replacement’s first request.
For an emergency stop or controlled takeover, install the new boundary explicitly. The receiver first advances its generation and enters a held state. Only after the required reconciliation does a separate activation make a new grant usable.
The old worker must be rejected during the held interval, not merely after its replacement has demonstrated activity.
A Generation Is Not a Credential
Knowing the current generation must not let the caller impersonate its current owner.
Bind the active grant to the authenticated executor, claim, logical action, exact request, protected scope, and applicable lifetime. An old worker that copies the new generation into its request should still lack the new grant and identity relationship.
The lab tests these bindings using trusted identity strings. It does not authenticate those strings. A production service must derive identity from its protected communication boundary rather than accept worker_ref as proof.
Advance to Hold Before Activating a Successor
The proposed recovery sequence separates containment from resumption.
First, establish the hold at the receiving boundary. Then reconcile affected work and decide whether a successor may act. Finally, install a separately authorized grant for that successor.
The diagram shows why these are different transitions. Advancing the generation does not automatically authorize new work.
Use the enterprise AI incident response runbook for the wider containment process. Fencing one receiver does not establish that every delegated tool, queue, or output path has stopped.
For workflows spanning several targets, track the result at each required boundary. Partial installation means partial containment. Do not open replacement work across the whole workflow because one receiver acknowledged the hold.
Preserve the Meaning of a Recovery Decision
A new grant should identify the reviewed successor action and its relationship to earlier work.
Do not obtain a current generation and attach every old queue item to it. That would make the recovery process an approval-renewal mechanism.
The original effect may already exist. The approval may have expired. The target may have changed. Each condition can change whether any further execution is justified.
Check the Fence Where the Effect Is Committed
A worker-side check leaves a familiar race:
Worker checks generation 1 Recovery installs generation 2 Worker sends the previously checked request Target accepts an ordinary write
Moving the same check into a gateway does not eliminate the race when the gateway subsequently calls an external target that does not enforce the condition.
The control must cover the relevant effect boundary.
In the new lab, the generation, active grant, synthetic target value, and receipt are stored in the same SQLite database. The receiver checks and changes them inside one write transaction. SQLite documents that only one write transaction can be active at a time, which allows the hold and mutation to be ordered in this local design.
The core checks resemble the following excerpt from the implementation:
if control["mode"] != "open":
return deny("receiver_held")
if envelope["epoch"] != control["epoch"]:
return deny("stale_or_unknown_epoch")
if envelope["grant_id"] != control["active_grant"]:
return deny("inactive_grant")
These checks run inside the transaction that also validates the bound grant and either returns a retained receipt or changes the local target.
They are not a standalone production authorization function. The complete implementation also validates request identity, owner and claim references, the test-time window, and target preconditions.
Do Not Move the Target Outside the Transaction Without Revisiting the Claim
Replacing the local update with a Kubernetes, NSX, or cloud API call changes the guarantee.
The database transaction cannot make an unrelated remote mutation atomic with its fence check. Keeping the database lock open longer does not enlist the external service in that transaction.
Where the actual target supports an appropriate conditional operation or target-enforced generation, use and test it. Otherwise, retain a more conservative operating mode: contain stale execution access, account for in-flight work, and establish an acceptable resumption point.
A receiver is the component that enforces the protected operation. Calling a proxy a receiver does not make it the final effect boundary.
Run the Receiver Fencing Lab
The Receiver Fencing Lab v0.1 contains a new local receiver and an unchanged copy of the preceding execution-ledger module. The request and patch examples are also preserved byte-for-byte.
The fixture remains the conditional thirty-to-forty-five change associated with the earlier unconsumed ConfigMap. The new receiver interprets that narrow fixture against its own local database. It does not send the JSON Patch to Kubernetes.
The exercise should leave the operator able to distinguish a stale-worker rejection, a restored-ledger claim, a receiver-side hold, and a recovery failure involving the receiver itself.
Use Python 3.10 or later, standard-library SQLite, and a trusted local filesystem. Local validation used Python 3.13.5 and SQLite 3.46.1 on Linux. Other environments were not tested.
After inspecting and extracting the package:
cd rtb-receiver-fencing-lab umask 077 python -m unittest discover -s tests -v python run_lab.py \ --output ../receiver-run-01
The destination must not exist, and its parent must exist. Preserve incomplete output and choose a new destination for a rerun.
The runner starts a bounded local child process for the pause-and-resume demonstration. It makes no model, network, or platform calls.
Know Which Responsibilities Are Simulated
The receiver begins at generation 1 in held mode.
admin_activate_demo installs a synthetic, reviewed grant and opens that generation. admin_hold advances the generation and removes active execution authority. Both represent privileged administrative operations whose authentication is not implemented.
A successful ledger claim does not invoke activation automatically.
Anyone who can edit the receiver database or invoke its administrative methods can bypass the intended separation. The lab demonstrates selected protocol behavior under trusted test administration, not resistance to a hostile process sharing its files.
Separate database files also do not establish separate administrative or infrastructure failure domains.
Read the Four Demonstrations Separately
Twenty-six local tests passed. Two complete demonstration runs produced the following outcomes.
| Demonstration | Observed local result | What it supports |
|---|---|---|
| Paused worker resumes after replacement activation. | Old request rejected before the replacement writes; target still has zero mutations. | Rejection depends on the installed fence, not a changed target value. |
| Receiver deliberately skips its fence checks. | Old request accepted under its previously valid grant. | The test detects the omitted enforcement condition. |
| Earlier ledger database is restored. | Restored ledger accepts another claim; current held receiver rejects the submitted work. | A restored claim does not automatically reopen this receiving boundary. |
| Receiver’s own pre-hold database is restored. | Obsolete receiver copy accepts the old grant. | The receiver cannot independently detect rollback of its own history. |
The last result is a deliberate counterexample. It is not a successful recovery-safety test.
The retained summary includes:
{
"stale_worker_accepted": false,
"mutations_before_current_worker": 0,
"current_worker_accepted": true,
"defective_receiver_accepted_stale_worker": true,
"restored_ledger_accepted_claim": true,
"held_receiver_accepted_restored_work": false,
"restored_receiver_copy_accepted_old_grant": true,
"production_authorization": "not_assessed"
}
The tests and expected results were authored alongside the implementation. They are not independent domain adjudication or exhaustive concurrency analysis.
Pause the Old Worker Before the New Worker Changes Anything
The child process holds its original execution envelope and waits.
The parent commits the generation change, installs the successor’s synthetic grant, and then releases the old process. The old request arrives while the target still has its original thirty-day value and revision.
It is rejected with stale_or_unknown_epoch.
Only then does the replacement perform the forty-five-day mutation. This ordering isolates the fencing condition from the target-state checks that would also reject some stale requests after a successful change.
The exercise tests one controlled schedule. It does not measure arbitrary process pauses, distributed partitions, or production takeover latency.
Keep the Known Defect in the Assessment
The skip_fence test configuration omits the current-mode, generation, and active-grant checks. It retains the stored grant, request binding, timing, and target checks.
The old request then succeeds.
That result demonstrates why a correctly issued historical grant is insufficient. The defect is in treating its previous validity as current authority.
A suite that remains green after removing its decisive control is not examining the claimed boundary.
Fencing Does Not Replace Duplicate Handling
Two requests can be current and still represent the same logical action.
The receiver therefore retains an action receipt alongside the mutation. Under a still-current grant, an identical request returns already_applied without changing the value again.
Amazon’s Making retries safe with idempotent APIs describes the importance of associating request identity with the service’s mutating work and rejecting inconsistent reuse. The local receiver applies that principle inside its own transaction; it does not transfer that guarantee to another service.
The order of checks matters. This lab checks current authority before returning a receipt through the mutation interface. An old grant remains rejected after a hold even when its historical action receipt exists.
The receipt can still be examined through the observation path. Reading historical evidence and authorizing a new mutation are different operations.
A Rejected Retry Does Not Prove the First Attempt Failed
Suppose the first operation committed and its reply was lost. Recovery then installs a hold. The repeated request is correctly rejected.
The rejection says the retry lacks current authority. It says nothing by itself about whether the earlier operation occurred.
Reconcile the original attempt through the target’s supported evidence. Keep its receipt, current state, and remaining uncertainty visible rather than interpreting a new denial as historical failure.
Do Not Treat a Kubernetes Lease as the Missing Target Control
The preceding Kubernetes kit still supplies useful protections: actual identities, resource-scoped permissions, admission validation, and conditional requests bound to object identity and state.
A Lease object used for coordination does not automatically make an unrelated ConfigMap mutation conditional on current leadership. The application needs an enforced relationship between those operations.
Likewise, Kubernetes documents resource-version conditions and conditional JSON Patch as mechanisms for detecting conflicting object state. A resource version does not inherently express which approval generation is current.
If ownership changes without changing the target object, the old request’s object preconditions may remain satisfied.
A protected generation associated with the target can be part of a stronger design, but the write path must require and validate it. Merely adding an annotation while allowing the executor to omit its test leaves a bypass.
Do not edit the earlier fixed policy and call the complete problem solved. Its tested value constraint and this proposed ownership boundary address different requirements.
Cover Every Usable Write Path
OWASP’s authorization guidance recommends validating permission on every request. Apply that to alternate tools, privileged deployment automation, emergency paths, and credentials that can bypass the intended receiver.
A fenced application endpoint cannot constrain a worker that can call the underlying administrative API directly.
The accepted threat model must also identify who can administer the fence itself. If the acting agent can install a new grant, change receiver code, or replace the database, it can redefine which generation is current.
Hold the Receiver Through Recovery
The restored-ledger demonstration preserves the preceding article’s counterexample: an old, internally valid ledger can accept authority already consumed elsewhere.
The new boundary does not repair that history. It prevents the receiver from treating the restored claim as sufficient to execute.
Keep the receiver held while the recovery team reconciles later approvals, revocations, claims, and external effects. Obtain any successor grant through a separately governed decision.
Do not let the restored workflow open its own receiver because its database passes an integrity check.
The Fence Has Recovery State Too
The receiver stores its generation and active grant. Restoring that state from before the hold can restore the exact conditions under which the old request was valid.
The lab demonstrates this failure without corrupting any file. The obsolete receiver copy is internally consistent.
Preventing that resurrection requires an accepted recovery decision outside the history being questioned. Depending on the architecture, that may involve separately protected control records, current identity restrictions, or a receiving boundary that remains disabled until reconstruction is approved.
No one mechanism is assumed to exist in this package.
If both the ledger and receiver have been restored, neither can establish its freshness merely by agreeing with the other. They may simply agree on the same obsolete history.
Preserve Safety When the Fence Cannot Be Installed
If the required receiver cannot acknowledge a hold, do not record full containment.
Use another independently authorized containment path where one exists. Account for what that path actually blocks and for requests already in progress.
Failover that restores throughput while leaving stale workers able to act is not a successful transfer of authority.
Capture Evidence at the Receiving Boundary
Use the infrastructure change evidence pattern to connect the recovery decision with its actual enforcement.
Record the previous and installed generations, protected scope, administrative decision, activation state, submitted grant, receiving identity, rejection reason, target observations, and outstanding operations.
The lab retains scenario reports, database state, receipts, events, and source and environment manifests. Its twenty-six tests also cover missing state, storage contention, changed request bytes, worker and claim mismatches, expiry, and rollback when required event or receipt insertion fails.
Those records share the local test administrator. They are not independently protected audit evidence.
Measure the Boundary’s Actual Reach
For integration testing, use a small set of explicit acceptance cases:
| Challenge | Required evidence |
|---|---|
| Old request arrives before the replacement’s first write. | Rejection at the intended fence, with no new protected mutation. |
| Old worker copies the current generation into its request. | Identity and grant binding still prevent unauthorized execution. |
| Fence installation fails or times out. | Workflow remains held; no unsupported claim that containment completed. |
| Operation was accepted before the hold. | Its later effects are tracked and reconciled rather than assumed canceled. |
| An alternate write path is attempted. | Applicable controls block it or the gap is documented as a release blocker. |
| Receiver state is restored from an old copy. | Independent recovery admission prevents the copy from reopening authority. |
Run permitted operations too. A broken route that rejects every request does not prove useful fencing.
Distinguish hold-request time, receiver acknowledgment, last observed stale rejection, and outstanding effects. Do not reduce them to one “worker stopped” timestamp.
Operate the Fence Without Creating a Permanent Outage
Fencing intentionally withdraws authority. Excessive generation changes can therefore create legitimate service disruption.
Choose the protected scope carefully. One service-wide generation may be simple but suspend unrelated workflows. Per-resource generations reduce that impact while creating more state, administration, and reconciliation work.
The scope must still include all operations whose ownership conflicts. Splitting generations too narrowly can allow two individually current grants to produce an unacceptable combined effect.
Assign owners for receiver availability, administrative access, activation decisions, and unresolved actions. Reserve the resources needed to reject requests and record evidence even when agent workloads are saturated.
Do not issue a new generation on every transport retry. A retry, a new approval, a takeover, and disaster recovery are different events.
Keep Failure Handling Conservative
A missing receiver database is not an empty initialization opportunity. The lab refuses to create operational state implicitly.
Storage contention and failed writes are errors, not permission to execute without checking. An unavailable observation path leaves relevant outcomes unresolved.
A generation approaching its representable limit needs a controlled transition, not numeric wraparound. Restoring a lower counter or changing its type without preserving the authority relationship creates another replay problem.
For the production design, rehearse these conditions alongside ordinary releases. Receiver code, policy, credentials, and recovery procedures are control-plane changes and need their own review.
Conclusion
The previous ledger established who claimed an action. Receiver fencing adds a different condition: whether that claimant’s authority remains usable when the request reaches the protected operation.
The local lab makes that distinction observable. A paused worker is rejected before its replacement changes the target. A restored ledger cannot reopen a held receiver. An obsolete receiver copy, however, still exposes the need for independently governed recovery admission.
That is the operating boundary to preserve. Coordination chooses the current owner. Enforcement makes the old path unusable. Evidence establishes what the transition actually accomplished.
The control plane must not grade itself. A recovered worker must not declare itself current.
Pause one worker before dispatch, establish its successor’s authority, and release the old request first. Which receiving control rejects it before the replacement performs any useful work?
External References
- Kubernetes client-go: leaderelection package
- etcd: etcd versus other key-value stores
- SQLite: Transaction
- Amazon Builders’ Library: Making retries safe with idempotent APIs
- Kubernetes: Kubernetes API Concepts
- OWASP: Authorization Cheat Sheet
Use a single-host SQLite lab to examine durable approval claims, worker restarts, and uncertain execution. Preserve ownership and reconcile effects before retrying...
1 thought on “Fencing Stale AI Workers: Enforcing Authority at the Receiver”