TL;DR
AI inference disaster recovery must restore an approved service, not merely a reachable model endpoint. That means recovering compatible model artifacts, retrieval context, authorization, application state, usable capacity, and safe request handling. Regional redundancy addresses regional failure; it does not automatically address a shared platform outage or a defective release deployed everywhere. Design complete recovery environments, define workload-specific recovery objectives, preapprove degraded modes, and test cold-cache performance, interrupted streams, and uncertain tool outcomes. A successful recovery preserves the service’s operating boundaries as well as its availability.
Introduction
Consider an illustrative incident. An enterprise assistant loses its primary region. The traffic manager redirects requests, the secondary endpoint answers, and the infrastructure dashboard turns green.
The application is not necessarily recovered. The secondary model might lack the approved adapter. Its retrieval index might contain yesterday’s access permissions. A conversation might be missing the tool result that explains whether an action already completed. The new endpoint can generate text while the business service remains incorrect, unsafe, or unusably slow.
Traditional disaster recovery already requires application and data recovery, not just infrastructure restoration. AI inference adds another set of dependencies: model behavior, inference runtime compatibility, context reconstruction, accelerator capacity, and generation-specific request state.
The architectural question is therefore broader than “Where is the second model server?”
What is the smallest independently operable AI service that can meet an explicitly approved recovery contract after the primary environment fails?
This article focuses on production inference, using large language models and retrieval-augmented generation, or RAG, as the main examples. It covers self-hosted serving and managed model APIs, not training checkpoint recovery. The designs and numerical targets below are illustrative, not measured performance claims.
Separate Regional Redundancy from Platform Independence
Start by naming the failure the design must survive. A second deployment is useful only when it removes the dependency involved in that failure.
| Failure scenario | Recovery requirement | What does not establish recovery |
|---|---|---|
| Worker, GPU, or serving replica failure | Another complete, healthy serving replica | Additional shards belonging to the same failed replica |
| Regional outage | A reachable service with the required dependencies outside that region | GPU capacity elsewhere that still calls the failed regional identity, retrieval, or state service |
| Shared platform or control-plane failure | A recovery path independent of the failed platform dependency | Two regions requiring the same unavailable deployment or administration service |
| Defective model, prompt, or policy release | A compatible, known-good release and controlled rollback | Rapidly replicating the same defect to every recovery target |
| Capacity exhaustion or severe latency | Admission control and enough qualified recovery capacity | A healthy endpoint that cannot complete the admitted workload |
Kubernetes documentation provides a useful boundary: PodDisruptionBudgets constrain certain voluntary disruptions, but they cannot prevent involuntary failures. They are not evidence that a regional recovery design works.
Likewise, active-active deployment describes traffic placement, not independence. Two regions can still share a model gateway, identity dependency, artifact registry, release pipeline, or provider service. A platform alternative needs to avoid the specific shared failure being addressed. A different logo alone is not sufficient.
Document those dependencies before selecting the recovery pattern. Otherwise, the architecture can accumulate replicas without materially improving recoverability.
Define Recovery at the Service Boundary
Recovery time objective, or RTO, should identify when the required service is usable again. Recovery point objective, or RPO, identifies acceptable data loss. For inference, both need more specificity than a single number attached to the cluster.
Specify Which Workload Must Return
A useful recovery contract states the supported task, eligible users, admitted demand, latency limits, quality checks, and operating restrictions.
For example, an organization might target restoration of priority, read-only assistance within two minutes, while allowing fifteen minutes for the full service, including actions. Those are different milestones. Reaching the first must not be reported as restoring the second.
Measure the recovery interval from the defined user-visible disruption, including detection and decision time, until the agreed workload passes its acceptance criteria for a specified observation window. A loaded model that handles one synthetic request does not establish sustained service recovery.
Assign Recovery Requirements to Different Types of State
A versioned model bundle has an identity and availability requirement: the approved artifacts must be present and compatible. It is not adequately described by “five minutes of model RPO.”
Conversation history, workflow state, audit records, and business transactions have their own durability requirements. A retrieval index has both recovery and freshness requirements relative to its authoritative sources. Permission revocations and document deletions need explicit treatment, not an assumption that ordinary content replication is sufficient.
Accepting limited conversation loss may be reasonable for a low-risk assistant. It is not permission to lose the record of a committed business action. A zero-loss objective for acknowledged actions requires a storage and acknowledgment design that actually supports it across the stated failure boundary.
Define Behavioral Acceptance, Not Textual Identity
An alternative runtime, quantization configuration, or model must pass the application’s task, schema, retrieval, and safety evaluations. API compatibility is only the starting point.
The vLLM reproducibility documentation explicitly limits its reproducibility guarantees to the same hardware and vLLM version. Do not promise byte-identical cross-platform answers merely because the model weights, seed, or temperature appear equivalent.
Where exact output is essential, use deterministic validation or a deterministic application path. For generative behavior, specify acceptable outcomes and failure thresholds rather than treating identical wording as the universal recovery criterion.
Recover a Complete Serving Environment
A practical design unit is a regional serving cell: a bounded environment containing the components required to serve its assigned workload. The term is an architectural grouping, not a claim that a particular product implements it automatically.
The diagram separates two regional cells on one platform from an optional alternate-platform cell. Each destination needs its own qualified dependency path.
The routing box is a logical function, not a single gateway instance. Its deployment, name resolution, credentials, certificates, and client entry points must survive the same event. Moving all backends while leaving the only reachable gateway in the failed region does not solve the problem.
AWS disaster recovery guidance recommends minimizing control-plane dependencies during recovery. For inference, prestage the cluster, approved artifacts, credentials, and application needed to restore minimum service.
Independence does not require duplicating every enterprise service. It requires demonstrating that each dependency needed by the recovery workload remains available, has a qualified alternative, or can be safely omitted in the approved degraded mode.
Replicate Complete Models, Not Just GPU Counts
vLLM supports tensor and pipeline parallelism to distribute a model across GPUs or nodes. Those devices can collectively form one serving replica rather than independent copies.
Eight GPUs hosting eight shards of one model are not eight failover targets. Losing a required shard can make that serving group unavailable until the runtime recovers it.
For a conventional regional DR design, keep each complete serving group within its qualified local accelerator and network topology. Route requests between regions instead of assuming that stretching a tightly coupled model across a wide-area network creates resilience. Validate model memory, KV-cache capacity, runtime configuration, and interconnect requirements together.
Preserve the Release, the Context, and the Authorization
Treat the recovery package as a versioned release bundle rather than a folder containing model weights.
For self-hosted inference, include the model and adapters, tokenizer, chat template, runtime image digest, quantization settings, supported accelerator configuration, and any platform-specific compiled artifacts. Add application prompts, tool schemas, retrieval configuration, policy definitions, and the evaluation results that approved that combination.
Different platforms may need different runtime builds. Record those as explicitly validated variants of the same service release, not as an assumption that one container runs equivalently everywhere. Stage both the current approved release and a known-good rollback candidate outside the primary failure domain.
For managed APIs, record the deployed model identity, relevant configuration, supported features, and approved alternatives. Do not assume that provider-hosted model artifacts can be exported into a self-hosted recovery environment.
Retrieval Must Recover as a Compatible System
Microsoft’s Azure AI Search guidance states that indexing and querying should use the same embedding model. This matters during recovery: replacing the query embedding model while retaining an incompatible vector index can break retrieval even when the vector dimensions match.
Keep the embedding model, preprocessing, index schema, source checkpoint, and retrieval configuration aligned. A fallback embedding model should have a separately validated index or another explicitly tested retrieval path, not an improvised substitution during an outage.
Recover authorization metadata alongside the content, including deletion markers and revocation state. Where freshness cannot be established, deny the affected retrieval or narrow the service to sources whose access can still be validated. Do not turn a data-plane outage into a broader permission grant.
Also preserve the evidence behind citations: source identity, document version, and the retrieved material supporting the response. An answer that cites unavailable or unauthorized evidence has not passed the same contract as the primary service.
KV Cache Is Not Durable Conversation State
For generative serving, key-value cache stores intermediate attention state used during processing. vLLM’s automatic prefix caching reuses this work to reduce prefill computation; it does not accelerate generation of new tokens during decode.
For recovery planning, treat cache as reconstructible unless a specific transfer or persistence mechanism has been implemented and tested across the required runtime and failure boundary. Do not assume that a cache feature provides general regional session recovery.
Persist the conversation and workflow information needed to reconstruct an authorized request. That is distinct from preserving a GPU’s execution state. A loaded model with an empty cache can be operationally healthy while facing substantially different startup demand from the normally warmed service.
Choose a Recovery Pattern Based on Readiness
AWS distinguishes a pilot light, which needs additional startup work, from a warm standby that is already functional at reduced capacity. Apply the distinction to the whole inference path, not only the virtual machines.
| Pattern | Inference-specific interpretation | Main tradeoff |
|---|---|---|
| Backup and restore | Rebuild from protected artifacts and state | Low standing compute commitment; longest preparation path |
| Pilot light | Partial environment; activate the remaining components | Lower standing capacity; more startup dependencies |
| Warm standby | Functional service at reduced capacity | Ongoing cost for an immediately usable service |
| Active-active | Multiple environments serve before failure | Continuous exercise; spare capacity and state coordination remain necessary |
A “warm” GPU pool that must first pull weights, compile an engine, obtain secrets, and start the application is not a warm inference service.
For an expensive model estate, a practical starting design is a warm recovery floor for critical workloads, with lower-priority processing paused or queued. Queueing is appropriate only when deadlines, retention, security, and duplicate handling support it.
Capacity Permission Is Not Capacity Readiness
An increased quota establishes permission to request resources. It does not establish that the required resources will be available at the recovery location.
AWS EC2 On-Demand Capacity Reservations illustrate the distinction: they reserve matching compute capacity in a specific Availability Zone and consume the associated quota. Even a suitable reservation still needs to become a working inference service. Validate the actual capacity product, accelerator type, topology, and startup path rather than treating a billing discount as a capacity commitment.
Active-active also requires arithmetic. In an illustrative deployment, each region might sustain 100 workload units while normally receiving 60. Losing one region sends 120 units toward a survivor qualified for 100. The topology remains active-active; the surviving capacity remains insufficient.
Use measured workload distributions, not GPU utilization alone. Prompt length, output length, concurrency, and cache state belong in the qualification. Specify which traffic will be rejected if the recovery floor is smaller than normal demand, and report that rejected demand rather than hiding it behind the success rate of admitted requests.
Distinguish Managed Inference Routing from Application Recovery
Managed cross-region inference can reduce part of the recovery burden, but its protection boundary must be understood.
Amazon Bedrock cross-Region inference uses profiles that identify a model and eligible processing regions. Geographic and global profiles have different routing scopes. The application still needs an approved data-processing destination and a reachable invocation path, along with its own retrieval, state, and tool dependencies.
Microsoft’s high-availability guidance for Foundry classic similarly distinguishes distributed model processing from resource-endpoint recovery. Its regional failover guidance uses separately deployed resources and caller-side routing when the primary endpoint cannot be reached.
The architectural implication is straightforward: provider-managed placement of inference work is not proof of complete application recovery or provider independence.
A platform alternative should be approved for the actual workload: context limits, input types, output schemas, tool-call behavior, identity, content controls, retention, and data-location requirements. It may offer a narrower service than the primary model. Read-only retrieval or a deterministic workflow can be a better fallback than an unqualified generative substitute.
Make Failover a Policy Decision
The router should not select any endpoint that returns a successful status. It should select a destination eligible for the request and the current recovery mode.
The following YAML is an illustrative application policy, not a Kubernetes resource or a built-in vLLM configuration. Its target names must resolve to a maintained catalog of approved deployments. The numbers are proposed objectives for an example service, not demonstrated recovery times.
service: enterprise-support-assistant
policy_version: 1
objectives:
priority_read_only_rto_seconds: 120
full_service_rto_seconds: 900
conversation_rpo_seconds: 60
retrieval_max_content_lag_seconds: 300
routing:
approved_targets:
- regional-secondary
- validated-platform-alternative
require_all:
- approved_release_and_task_evaluation
- permitted_data_location
- current_authorization_enforced
- compatible_retrieval_checkpoint
- recovery_capacity_canary_passed
- durable_audit_path_available
no_eligible_target: reject
degraded_mode:
tool_writes: disabled
oversized_input: reject
max_output_tokens: 512
request_recovery:
max_attempts_total: 2
total_deadline_seconds: 20
partial_stream: explicit_restart_required
unknown_tool_outcome: reconcile_before_retry
failback:
automatic: false
Enforcement belongs in the gateway, application, and policy services that actually control routing and tool execution. A YAML file alone enforces nothing. In particular, disabling writes must be a tool authorization decision, not merely a sentence added to the model prompt. The durable audit path can be an approved regional journal with later export; it need not depend on a central logging service that shares the outage.
The five-minute content-lag allowance does not allow five minutes of stale authorization. The recovery target must establish the access decision required by policy or refuse the affected request. Similarly, a one-minute conversation RPO cannot be used as the durability policy for action receipts.
Test the negative path as carefully as the positive path. When a target has insufficient capacity, an unapproved model, stale access information, or the wrong data location, the expected result is refusal or an approved narrower mode, not bypassing the gate.
Handle Interrupted Requests Without Inventing Continuity
New requests and in-flight work need different recovery behavior. The decision sequence should preserve that distinction.
A Second Generation Is a New Attempt
A pure generation request may be retried within its deadline when no response has been exposed and no side effect has occurred. Use a bounded retry budget across the whole call chain, not independently at every layer.
Once part of a response has reached the client, do not silently append another model’s generation as though the original execution continued. End the failed attempt and use an explicit restart or a specifically implemented resume protocol. A token offset alone is not proof that another runtime can continue the same execution correctly.
Record a stable logical request ID and distinct attempt IDs. Return or log the selected deployment, release identity, recovery mode, and completion status. Streaming success means a completed, valid response, not simply the HTTP status sent when the stream opened.
Tool Outcomes Need Authoritative Reconciliation
An agent may have submitted a business action before the connection failed. No response does not establish that the action did not happen.
AWS Builders’ Library explains how caller-supplied request identifiers support safe retries through idempotent APIs. For an agent workflow, carry a stable operation ID to the system that performs the effect and rely on its durable deduplication or authoritative outcome record. An orchestration log written separately from the action is not, by itself, an exactly-once guarantee.
If completion is unknown, reconcile before retrying. When promoting a new writer, enforce ownership or fencing so the old region cannot also commit conflicting work. Read-only inference can recover before this step; business actions should remain disabled until the write path is safe.
Prove Recovery with Failure Exercises
A deployment test establishes that the secondary environment can start. A recovery exercise must establish that it can take over under the conditions created by failure.
Use a small, deliberate exercise set:
| Exercise | Evidence required |
|---|---|
| Isolate the primary region | Clients reach the recovery path without relying on the isolated region |
| Remove access to deployment and artifact services | The minimum recovery service operates from staged, approved dependencies |
| Shift representative traffic with cold caches | Priority throughput and latency remain within the recovery contract |
| Present stale permissions or incompatible retrieval configuration | Affected requests are denied or placed into an approved narrower mode |
| Interrupt a stream and obscure a tool result | The client sees an explicit failure or restart, and no blind action replay occurs |
| Partition state, inject a bad release, then recover | Write ownership, known-good rollback, reconciliation, and failback remain controlled |
Measure from the client boundary and correlate with serving telemetry. vLLM exposes metrics including time to first token, end-to-end request latency, request waiting counts, and KV-cache usage. Add application-level measures for valid completion, schema compliance, retrieval correctness, authorization failures, rejected demand, and duplicated actions.
Record workload shape, cache conditions, model and runtime versions, actual replication lag, and the sustained observation period. A claim that recovery took ninety seconds is incomplete without identifying what capability returned and what remained unavailable.
Keep recoverable historical copies as well as replicas. AWS’s disaster recovery guidance notes that replication alone does not protect against logical corruption. Faster synchronization is not a substitute for a clean restore point.
Build the Capability in Controlled Stages
Establish the Contract and Dependency Map
Start with one consequential inference service. Name its recovery workloads, acceptable restrictions, state ownership, and prohibited destinations. Map the model path and the less visible dependencies, including identity, keys, name resolution, retrieval, outbound tools, and telemetry.
Assign an owner to each acceptance gate. The platform team owns runtime and capacity qualification; the data team owns recovery checkpoints and retrieval freshness; the application team owns request and action semantics; security owns the allowed authority and data boundaries.
Prestage the Minimum Independent Service
Deploy the critical path before an incident. Populate the approved release catalog, qualify platform-specific variants, establish state protection, and continuously exercise the recovery floor with representative requests.
Preserve a recovery candidate during production changes. Stagger releases and use explicit promotion gates so a single deployment does not eliminate every known-good option. Budget ongoing capacity and testing as operating costs, not one-time project deliverables.
Exercise Failover and Treat Failback as a Separate Change
Expand testing from replica loss to regional isolation, shared dependency failure, and logical corruption. Include false alarms and intermittent recovery so automation does not oscillate between targets.
Do not automatically return production traffic just because the primary endpoint becomes reachable. Reconcile authoritative state, resolve uncertain operations, validate the intended release, and confirm capacity before gradually restoring traffic. Where possible, let healthy in-flight generations finish on their existing target rather than moving them midstream.
Retest whenever the model, runtime, retrieval pipeline, access policy, or platform alternative materially changes. A recovery qualification applies to the tested service combination, not indefinitely to the architecture diagram.
Conclusion
AI inference disaster recovery is the recovery of a controlled capability. The endpoint, model, context, permissions, state, and capacity must work together under an explicit service contract.
A second region is valuable when it removes regional dependencies. A platform alternative is valuable when it removes the relevant shared failure without introducing an unapproved change in behavior or authority. Neither should be credited with protection that has not been demonstrated.
Begin with the minimum service the business genuinely needs during disruption. Make that service independently operable, qualify its degraded mode, and test the transitions into and out of recovery. Expand coverage only when the supporting state, capacity, and controls are ready.
The recovery test is not whether another system can generate an answer. It is whether the organization can safely depend on that answer, and on any action behind it, after the primary environment is gone.
External References
- AWS: Disaster recovery options in the cloud
Canonical URL: https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/disaster-recovery-options-in-the-cloud.html - Kubernetes: Disruptions
Canonical URL: https://kubernetes.io/docs/concepts/workloads/pods/disruptions/ - AWS: Reserve compute capacity with EC2 On-Demand Capacity Reservations
Canonical URL: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-capacity-reservations.html - AWS: Route model inference requests across AWS Regions with cross-Region inference
Canonical URL: https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html - Microsoft Learn: High availability and resiliency for Microsoft Foundry projects and Agent Services (classic)
Canonical URL: https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/high-availability-resiliency - Microsoft Learn: Generate embeddings for search queries and documents
Canonical URL: https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-generate-embeddings - AWS Builders’ Library: Making retries safe with idempotent APIs
Canonical URL: https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/ - vLLM: Parallelism and Scaling
Canonical URL: https://docs.vllm.ai/en/stable/serving/parallelism_scaling/ - vLLM: Reproducibility
Canonical URL: https://docs.vllm.ai/en/stable/usage/reproducibility/ - vLLM: Automatic Prefix Caching
Canonical URL: https://docs.vllm.ai/en/stable/features/automatic_prefix_caching/ - vLLM: Production Metrics
Canonical URL: https://docs.vllm.ai/en/stable/usage/metrics/
TL;DR Enterprise AI incident response must address identity, execution, data, persistent state, and downstream business systems. Stopping an agent is only one...