Building a VCF 9.1 Upgrade Runbook: Testing, Fallback, and Readiness Gates

by

Scenario

The VCF 9.1 upgrade path has been selected. The planning tool has been reviewed. The target version is understood. The team has a maintenance window on the calendar.

That does not mean the environment is ready.

A VCF 9.1 upgrade runbook needs to prove more than task order. It needs to prove that the organization can execute the upgrade, validate the result, stop safely when needed, and hand the platform back to operations without ambiguity.

This is where many upgrade plans become too lightweight.

They contain the sequence, but not the gates.

They mention backups, but not restore confidence.

They include prechecks, but not application validation.

They describe rollback, but not what rollback means for each phase.

VCF 9.1 deserves a runbook that treats testing, fallback, and readiness as first-class workstreams.

Why This Matters Operationally

VCF 9.1 is not a single-appliance patch.

The upgrade path touches VCF Operations, SDDC Manager, VCF Management Services, licensing, NSX, vCenter, ESXi, NSX Edge, workload domains, and Day-N operational services.

That creates three practical risks:

  1. A phase can be technically complete but operationally incomplete.
  2. A component can upgrade successfully while observability, licensing, identity, or application acceptance remains unresolved.
  3. A failed phase may not have a clean “undo” path unless the fallback decision was defined before the work began.

The runbook should make those risks visible before the window starts.

Symptoms of a Weak VCF 9.1 Upgrade Runbook

A weak runbook usually sounds confident until you ask who owns the evidence.

Common warning signs include:

SymptomWhy It Is a Problem
“We will validate after the upgrade.”Validation must be designed before execution, not improvised after change.
“Backups are handled by the backup team.”The upgrade team still needs backup evidence and recovery ownership.
“Rollback is restore from backup.”That may be true for one component and wrong for another.
“Monitoring will be checked at the end.”VCF Operations is part of the sequencing model, not a final dashboard check.
“The platform team owns it.”Platform teams often do not own DNS, IPAM, PKI, identity, network policy, or application testing.
“Workload domains can be done later.”Later still needs an owner, sequence, acceptance model, and Day-N backlog.

If these answers show up in the planning meeting, the runbook is not ready.

The Runbook Model: Gates Before Tasks

A VCF 9.1 runbook should be organized around gates.

Each gate answers a simple question:

“Do we have enough evidence to proceed to the next phase?”

The task list still matters, but the gate controls the risk.

This model keeps the runbook honest.

The team cannot hide missing DNS records inside a task list. It cannot bury unclear rollback language under a generic change record. It cannot treat application testing as a courtesy check after the platform team leaves.

Every gate needs an owner, evidence, and a stop/go decision.

Prerequisites and Safety Checks

Before the runbook enters the execution window, the team should complete the following checks.

AreaRequired Evidence
Upgrade pathApproved path based on current Broadcom documentation, release notes, KBs, interoperability checks, and compatibility guidance.
Current-state inventorySDDC Manager, vCenter, ESXi, NSX, NSX Edge, Aria Operations, VCF Operations, workload domains, and adjacent management components documented.
VCF Operations readinessAria Operations version and patch level validated, VCF Operations path confirmed, Cloud Proxy requirements understood, dashboards and policies reviewed.
Management services readinessIP ranges reserved, DNS records planned, internal IP overlap checked, management network reachability confirmed.
Licensing readinessLicense server design, DNS requirements, and post-upgrade licensing validation documented.
Backup evidenceSDDC Manager, vCenter, NSX, VCF Operations, and other critical management appliances covered by known backup/recovery procedures.
Change controlWindow, owner, bridge, escalation path, support case strategy, communication plan, and stop/go criteria approved.
Application validationRepresentative workloads and application owner smoke tests identified before the window.

The point is not to create paperwork.

The point is to avoid discovering a blocking dependency while the environment is already in motion.

Runbook Stage 0: Source Validation

The first stage is source validation.

This is where the team confirms the runbook is based on current information, not stale assumptions.

Capture:

  • VCF 9.1 upgrade documentation.
  • VCF 9.1 release notes.
  • Broadcom KBs for known upgrade issues.
  • Interoperability Matrix results.
  • VCF compatibility and hardware guidance.
  • VCF Upgrade Planning Tool output.
  • Environment-specific assumptions.
  • Vendor or partner recommendations.

This stage should produce a source pack attached to the change record.

Do not rely on memory for version-sensitive claims. VCF 9.1 sequencing, Aria Operations patch-level behavior, licensing, management services, and supported upgrade paths can change.

Runbook Stage 1: Current-State Inventory

The inventory stage answers the question:

“What are we actually upgrading?”

Document the following:

Component AreaWhat to Capture
VCF baselineCurrent VCF version, SDDC Manager version, workload domain list.
vSpherevCenter versions, ESXi versions, cluster lifecycle model, vSAN state, distributed switch versions.
NSXNSX Manager version, transport nodes, TEP health, edge clusters, routing, federation if applicable.
OperationsAria Operations version, VCF Operations state, policies, dashboards, collectors, adapters, alerts.
Management servicesExisting fleet management state, identity services, licensing services, log management services.
External dependenciesDNS, IPAM, NTP, PKI, identity provider, backup platform, syslog, monitoring integrations.
Workload dependenciesCritical workloads, application owners, smoke tests, maintenance constraints.

This inventory should be reviewed by the owners, not only exported by the platform team.

A version report is useful. A validated dependency map is better.

Runbook Stage 2: VCF Operations Readiness

VCF Operations is the first major readiness gate.

Before core components move forward, the runbook should verify:

  • whether Aria Operations is already deployed,
  • the exact Aria Operations version and patch level,
  • whether the environment must first move to Aria Operations 8.18,
  • whether the exact 8.18.x patch level has known upgrade-path caveats,
  • whether VCF Operations 9.1 will be upgraded in place or deployed,
  • whether Cloud Proxy placement is required,
  • whether collectors and integrations are healthy,
  • whether dashboards, alerts, policies, and custom content are documented,
  • whether post-upgrade observability acceptance criteria are defined.

The output of this stage should be an operations readiness record.

That record should not simply say “monitoring healthy.”

It should say what was checked, who checked it, what changed, what remains at risk, and who accepts the operational state.

Runbook Stage 3: SDDC Manager and Management Services Readiness

After VCF Operations readiness is complete, the runbook moves into SDDC Manager and VCF Management Services.

This stage needs more than SDDC Manager prechecks.

It should validate:

DependencyReadiness Check
SDDC ManagerBackup evidence, precheck output, upgrade bundle availability, administrative access, certificate/password health.
VCF Management ServicesIP range plan, management network reachability, DNS requirements, internal network overlap review.
License servicesLicense server deployment plan, A/PTR records, connectivity, post-upgrade license validation.
DNS/IPAMReserved addresses, FQDNs, reverse lookup, change timing, owner availability.
SecurityFirewall paths, certificate trust, privileged access, audit requirements.
SupportSupport entitlement, case process, log bundle procedure, escalation owner.

This is the stage where many upgrade windows get derailed.

The platform workflow may be ready, but DNS, IPAM, routing, firewall, or licensing may not be. The runbook should expose those blockers before the window.

Runbook Stage 4: Core Platform Upgrade

The core platform stage includes the remaining management-domain components such as NSX, vCenter, ESXi, and NSX Edge.

The exact sequence must follow the supported Broadcom guidance for the environment’s starting version. The runbook should not flatten this into “upgrade the core stack.”

Break it into explicit work packages:

Work PackageRequired Validation
NSX upgradeNSX backup, manager health, edge health, transport node health, routing validation, federation checks if applicable.
vCenter upgradeBackup evidence, temporary IP requirements, RDU or in-place decision, service health, plugin compatibility.
ESXi remediationCluster lifecycle model, image readiness, evacuation capacity, vSAN health, host maintenance behavior.
NSX Edge finalizationEdge version alignment, routing checks, north/south connectivity, service impact validation.
Management-domain acceptanceSDDC Manager health, VCF Operations visibility, license state, monitoring state, backup resumption.

The key principle is simple:

A component upgrade is not complete until its control plane, data plane, and operational visibility are validated.

Runbook Stage 5: Validation

Validation should be layered.

A green platform dashboard is useful, but it is not enough.

Platform Validation

Confirm the infrastructure state:

  • SDDC Manager reports expected versions and health.
  • VCF Operations is reachable and collecting expected data.
  • VCF Management Services are healthy.
  • NSX managers are healthy.
  • NSX Edge nodes are healthy.
  • vCenter services are running.
  • ESXi hosts are connected and remediated.
  • vSAN and storage health are clean.
  • Backup jobs resume successfully.
  • License assignment is valid.

Operations Validation

Confirm the platform can still be operated:

  • Alerts are firing where expected.
  • Dashboards and policies are intact or intentionally replaced.
  • Logs are flowing to the expected destination.
  • Admin access works through the expected identity path.
  • Support bundle generation is understood.
  • Certificate and password workflows are documented.
  • Lifecycle views show the expected state.
  • Service desk routing is ready for post-upgrade noise.

Workload Validation

Confirm business impact is acceptable:

  • Representative VMs remain available.
  • Critical applications pass smoke tests.
  • North/south connectivity works.
  • East/west connectivity works.
  • Backup and restore confidence checks complete where applicable.
  • Monitoring accurately reflects workload state.
  • Application owners provide acceptance.

This is the difference between technical completion and operational acceptance.

Runbook Stage 6: Rollback and Fallback

The rollback plan should not pretend the whole platform has one undo button.

For VCF 9.1, use the term fallback deliberately.

In some phases, fallback may mean restoring a specific appliance. In other phases, it may mean stopping the sequence, stabilizing the environment, collecting logs, opening a support case, and waiting for a supported recovery path.

Define fallback by phase.

PhaseFallback Question
Before VCF Operations upgradeCan we restore Aria Operations state, content, and integrations if the upgrade fails?
Before SDDC Manager upgradeDo we have a valid SDDC Manager backup and documented recovery procedure?
Before VCF Management Services deploymentCan DNS, IPAM, firewall, or deployment changes be backed out cleanly if deployment fails?
Before NSX upgradeDo we have NSX backups, edge validation, routing state, and a supported recovery plan?
Before vCenter upgradeDo we have vCenter backup evidence, temporary IP planning, and a supported RDU or upgrade recovery path?
Before ESXi remediationCan workloads evacuate safely, and are cluster capacity/admission constraints understood?
Before workload-domain upgradesAre application owners available for validation and incident response?

The runbook should also define stop conditions.

Examples:

  • VCF Operations upgrade fails and observability cannot be trusted.
  • SDDC Manager precheck fails with unresolved blockers.
  • VCF Management Services deployment cannot complete due to DNS/IP/network issues.
  • NSX manager health is degraded before upgrade.
  • NSX Edge routing validation fails.
  • vCenter upgrade fails or services do not stabilize.
  • ESXi remediation creates cluster capacity or vSAN health risk.
  • Application validation fails for a critical service.

The important decision is not whether the team wants to continue.

The important decision is whether the team has evidence that continuing is safer than stopping.

Runbook Stage 7: Day-N Handoff

The upgrade is not finished when the first window closes.

The Day-N handoff should include:

Handoff ItemOwner
Remaining workload-domain upgradesVCF platform owner
VCF Operations cleanupOperations / monitoring owner
License validation and cleanupLicensing / platform owner
Identity transition tasksSecurity / identity owner
Log management planOperations / logging owner
Documentation updatesPlatform architecture owner
Application validation follow-upApplication owners
Known issues and support casesChange owner
Next maintenance windowsPlatform owner and change manager

This stage prevents the common problem where the project celebrates the management-domain upgrade but leaves the operating team with undocumented changes, half-finished cleanup, and unclear ownership.

VCF 9.1 Readiness Scorecard

Use a scorecard before the change advisory meeting.

Do not proceed with red items in:

  • VCF Operations readiness,
  • supported path validation,
  • DNS/IP/network readiness,
  • backup/recovery evidence,
  • or stop/go decision ownership.

Those are not cosmetic gaps. They are upgrade control failures.

Common Gotchas to Put in the Runbook

The following items should be explicit in the runbook, not left to memory.

Conclusion: A Good Runbook Controls the Upgrade, Not Just the Tasks

A VCF 9.1 upgrade runbook should do more than list steps.

It should control the upgrade.

That means every major phase has:

  • an owner,
  • a readiness gate,
  • evidence,
  • validation,
  • fallback criteria,
  • and a handoff path.

VCF 9.1 makes this especially important because VCF Operations, management services, licensing, NSX, vCenter, ESXi, and Day-N workload-domain planning are all connected.

A patch-window checklist may get the team into the upgrade.

A real runbook is what gets the platform back into a supportable operating state.

External References

Leave a Reply

Discover more from Digital Thought Disruption

Subscribe now to keep reading and get access to the full archive.

Continue reading