Animation: Workloads can remain available while management capabilities require recovery and verification before return to service.
TL;DR
I am releasing a new technical whitepaper, Beyond High Availability: Designing and Validating Management-Plane Recovery in VMware Cloud Foundation 9.1.x. The paper examines an operationally important problem: a private cloud can continue running applications while losing the management capabilities required to administer infrastructure, investigate incidents, recover services, and maintain protection.
The 38-page whitepaper introduces a capability-oriented assessment framework for VCF 9.1.x, using VCF 9.1.1 as its reference release. It connects recovery objectives, management dependencies, documented recovery models, proposed laboratory exercises, and evidence-based acceptance decisions. It also includes practical worksheets, qualification records, and bounded test scenarios that architects and operations teams can adapt to their own environments. The examples are planning assessments, not measured recovery results.
The central takeaway: High availability can preserve service execution. Recovery readiness must demonstrate that an organization can regain the capabilities required to operate the platform.
Read Beyond High Availability on Zenodo for the complete framework, planning example, qualification records, and reusable worksheets.
Introduction
Consider a VMware Cloud Foundation environment experiencing a serious management-plane failure.
Business-critical virtual machines remain powered on. Existing applications continue processing transactions. From the perspective of an application health dashboard, production may appear relatively stable.
But the infrastructure team has a different problem.
Administrators cannot establish new sessions through the normal identity path. Management interfaces are unavailable or incomplete. Infrastructure changes cannot be performed safely. Log ingestion may have stopped. Backup or recovery services may be inaccessible, and the team cannot establish whether a restored component is ready to resume its intended role.
The organization has preserved some workload availability, but it has lost essential operating capabilities.
What does recovery mean under those conditions?
Is recovery complete when an appliance restarts? When a management interface responds? When an administrator signs in? Or only when the required administrative, investigative, security, and protection functions have returned and been verified?
These questions motivated my new whitepaper.
Beyond High Availability examines management-plane recovery as a specific architecture and operational assurance problem within VMware Cloud Foundation 9.1.x.
Rather than proposing another universal disaster recovery checklist, the paper focuses on determining which capabilities must recover, which dependencies they require, and what evidence is necessary before declaring a recovery path ready.
Why I Wrote Beyond High Availability
I wrote this paper because “the platform is protected” leaves the failure boundary undefined. Component recovery, instance reconstruction, and fleet recovery require different surviving resources and procedures. The assessment starts by naming the capability to restore, checking that operators can reach its recovery tools, and qualifying the path for the installed versions and topology.
What the Whitepaper Covers
The paper follows the lifecycle of a management-plane recovery assessment, from defining the necessary capabilities to making an operational readiness decision.
Seven Capabilities That Define Useful Recovery
The framework begins with seven observable capabilities. Each requires a functional demonstration rather than reliance on an appliance status or completed task alone.
| Capability | What recovery must demonstrate |
|---|---|
| C1: Workload availability | A representative application transaction completes successfully |
| C2: Administrative access | An authorized operator establishes a fresh session and exercises the intended role |
| C3: Infrastructure management | An approved, reversible change completes and its effect is verified |
| C4: Self-service provisioning | A governed request creates the intended resource and completes cleanup |
| C5: Operational visibility | A fresh event becomes visible through the required investigation or alerting path |
| C6: Security and trust | Required permissions and trust relationships function, while prohibited actions remain denied |
| C7: Continued protection | New protection evidence is created and retrievable, with its recovery point identified |
These are assessment categories proposed in the paper, not vendor-defined service-level commitments.
The distinction matters because capabilities can recover at different times.
An administrator may regain emergency access while normal identity integration remains unavailable. A management interface may respond before infrastructure changes are safe. Logging services may restart without demonstrating that new events are being ingested.
One successful recovery milestone does not automatically satisfy the others.
Map the Dependencies That Recovery Actually Requires
The whitepaper examines how normal operating dependencies can differ from recovery dependencies.
A service that is not required to keep an existing workload running may become essential when that workload must be reconfigured, replaced, or recovered.
The assessment includes identity, Domain Name System (DNS), certificates, backup repositories, storage, network paths, software artifacts, management interfaces, credentials, and recovery-operator access.
It also examines dependency cycles where one component must be available to recover another, but the second component is needed to restore the first.
The central question is whether an authorized recovery path exists from the resources and information that survive the named failure. The supporting architecture discussions on VCF workload-domain boundaries and VCF fleet operations explain how lifecycle, ownership, and shared services affect those dependencies.
Compare the Documented VCF Recovery Models
The paper distinguishes three documented architectural recovery models:
| Recovery model | Primary boundary |
|---|---|
| Component Backup and Restore | Recovery of selected management components |
| VCF Instance Backup and Restore | Reconstruction of a failed VCF instance from backup under the documented topology and infrastructure-identity requirements. |
| VCF Fleet Disaster Recovery | Recovery of designated fleet functions into another qualified VCF instance within the same fleet under the documented disaster-recovery design. |
These models have different state requirements, supported mechanisms, destination conditions, and operational consequences. The documented architecture and procedures are linked under External References.
The paper examines how those differences affect management services, VCF Operations, VCF Automation, vCenter Server, NSX, and relevant external dependencies.
Importantly, an architectural model is not automatically an executable procedure for every installed VCF 9.1.x build.
A recovery plan still requires the applicable official procedure, an exact component inventory, release and topology qualification, and verification that the necessary artifacts and access paths are available.
From Recovery Procedures to Evidence-Based Readiness
A completed restore task answers only part of the readiness question. The paper tracks four separate evidence dimensions:
- Vendor documentation: Does an authoritative procedure apply to the selected recovery scenario, including its release scope and restrictions?
- Release and topology qualification: Do the actual component builds, deployment history, recovery destination, and environmental prerequisites satisfy that procedure?
- Observed behavior: Has the required capability been demonstrated through a controlled exercise with recorded results?
- Organizational acceptance: Has the responsible authority accepted the measured outcome, limitations, and operating conditions?
The following diagram illustrates the assessment logic. A successful restore task is an intermediate result, while readiness requires the complete evidence chain.
Figure: Proposed recovery-readiness assessment flow. Each stage must retain its own supporting evidence. The diagram is an assessment sequence, not a universal VCF component startup order.
The resulting operational decision is intentionally narrow: ready for the named scenario, ready with restrictions, or not ready for the named scenario.
An organization’s readiness assessment should identify the exact environment, failure boundary, required capabilities, and operating conditions. A successful component recovery test cannot be silently promoted into proof of complete management-domain or site recovery.
A Practical Example: Recovering VCF Log Management
The whitepaper includes a worked planning example focused on VMware Cloud Foundation Log Management.
The scenario assumes that the original Management Services cluster has become unavailable and that Log Management must be recovered onto a qualified new or rebuilt cluster.
The required outcome is straightforward: an investigator must be able to search for a newly generated event through the recovered logging service.
However, the recovery path involves more than deploying replacement infrastructure.
Broadcom KB 436904 documents a Log Management reconciliation scenario involving a new or rebuilt Management Services cluster. Where Network File System (NFS) storage was configured in the previous cluster, the guidance requires that path to be accessible from the new cluster before proceeding.
The whitepaper’s hypothetical assessment identifies several unresolved prerequisites:
- The precise source and target builds have not been qualified.
- Required authenticated recovery artifacts have not been acquired.
- The previously configured NFS path is unreachable from the target. Do not proceed with this recovery procedure until access is restored or Broadcom Support supplies applicable guidance.
- Backup metadata, destination conditions, and historical-data expectations require further review.
The assessment therefore reaches a specific conclusion: the proposed scenario is not yet ready for an authorized recovery exercise.
That conclusion is deliberate.
The existence of spare compute capacity does not establish a usable recovery path. A completed restore task would not, by itself, establish fresh logging functionality.
The example also separates several outcomes that might otherwise be reported together: new event ingestion, historical log availability, delivery of events generated during the failure, archive access, authorization, and renewed protection.
No exercise has been executed in the example, and no measured recovery result is claimed.
Practical Tools Included in the Whitepaper
I wanted this paper to be useful beyond an architecture discussion. It includes reusable assessment material that teams can adapt into their existing resilience and platform governance processes.
Recovery evidence record
A structured record for capturing the service, failure boundary, documented method, exact baseline, required dependencies, observations, acceptance decision, and outstanding work.
Laboratory planning and observation worksheets
Templates for defining authorized failure injection, prerequisites, stop conditions, recovery milestones, timing, functional validation, and final disposition.
Release and recovery qualification record
A method for documenting source and target versions, topology restrictions, vendor applicability, dependency requirements, and qualification gaps.
Bounded recovery test library
Ten proposed exercises covering identity, infrastructure control, monitoring, Management Services, log ingestion, provisioning, recovery repositories, destination qualification, network trust, and return to service.
The paper also includes a public source register and detailed component and fleet recovery qualification notes.
These resources are intended to reduce the amount of interpretation left to the person performing a recovery assessment. They do not replace release-specific product procedures, environment validation, or organizational authorization.
How I Recommend Using the Paper
For teams working with VMware Cloud Foundation 9.1.x, I recommend beginning with one critical management capability rather than immediately attempting to validate every possible disaster scenario.
For example, consider recovery of infrastructure management after a specified management-service failure.
Define which administrator must regain access, which management endpoint is required, which action demonstrates restored control, and what time objective the service owner has established.
Then map the dependencies of that exact action, qualify the relevant documented recovery path, and identify what infrastructure, identities, credentials, and recovery artifacts must survive the failure.
Only when those prerequisites are established should the team authorize the laboratory exercise.
The resulting evidence should show what succeeded, what failed, which dependencies were assumed to survive, which safeguards were restored, and what remains unresolved.
That process gives the platform owner a more useful deliverable than a generic disaster recovery test summary. It establishes whether a specific operating capability can be recovered under stated conditions.
Scope and Important Limitations
This first edition focuses on the availability and operational recoverability of the VCF management plane.
It is not a vendor-certified recovery design, a universal startup sequence, or a collection of laboratory benchmark results.
The whitepaper does not establish cyber-clean recovery, forensic preservation, regulatory compliance, or application-data recovery. Those outcomes require separate requirements, procedures, and evidence.
Its VCF 9.1.1 reference baseline also does not mean every 9.1.x source and target combination is supported. Environment-specific recovery procedures must be qualified against the installed products and actual deployment history.
The framework and hypothetical planning examples are proposed assessment guidance. They are not represented as customer results or executed tests.
Read the Complete Whitepaper
NEW PUBLICATION | FIRST EDITION
Beyond High Availability
Designing and Validating Management-Plane Recovery in VMware Cloud Foundation 9.1.x
Paul Bryant | Digital Thought Disruption
38 pages | Version 1.0 | October 20, 2026
Reference release: VCF 9.1.1
Read Beyond High Availability on Zenodo for the architecture discussion, evidence framework, worked planning assessment, recovery qualification material, and reusable worksheets.
The intended audience includes VMware Cloud Foundation architects, private cloud platform owners, infrastructure operations leaders, and enterprise recovery planners.
Conclusion
A private cloud can survive an infrastructure failure in one sense while remaining operationally impaired in another.
High availability helps preserve service execution when its surviving dependencies and configured protection mechanisms permit it. Management-plane recovery addresses the next question: can the organization regain the administrative, investigative, security, and protection capabilities required to operate the environment?
That distinction deserves explicit architecture decisions, documented dependencies, applicable recovery procedures, controlled exercises, and defensible acceptance evidence.
Beyond High Availability was written to provide a practical framework for performing that work in VMware Cloud Foundation 9.1.x.
If you own a VCF environment, I encourage you to begin with one recovery question:
Which management capability must your organization recover first, and what evidence would convince you that it can actually be recovered under the failure conditions that matter?
External References
- Paul Bryant: Beyond High Availability, Designing and Validating Management-Plane Recovery in VMware Cloud Foundation 9.1.x
- VMware Cloud Foundation Blog: Announcing General Availability of VMware Cloud Foundation 9.1.1
- Broadcom TechDocs: Component Backup and Restore of VMware Cloud Foundation
- Broadcom TechDocs: VCF Instance Backup and Restore
- Broadcom TechDocs: VCF Fleet Disaster Recovery
- Broadcom KB 436904: Log Management does not come back online after restoring it to a new management services cluster