KB 393663: Fix Supervisor Deployment Stuck at “Installed and Started Kubernetes Node Agent on the ESXi Host”

by

TL;DR

When vSphere Supervisor deployment stalls at Installed and Started Kubernetes Node Agent on the ESXi Host, do not treat the message itself as the root cause. For the failure documented in Broadcom KB 393663, the important evidence is repeated Apply Solution activity plus vCenter WCP log entries stating that the solution specification in the desired image is incompatible with one or more ESXi hosts.

The supported remediation is to restore vSphere Lifecycle Manager compliance. Validate the cluster image, correct the ESXi base image and OEM vendor add-on as required, investigate incompatible components, verify that hosts are not booted with legacy BIOS, and remediate the hosts through vLCM. Once the hosts are compliant, vLCM can install the Spherelet VIB and Supervisor configuration can continue.

The same visible Supervisor stage can also fail because of content-library catalog problems, unreachable software depots, or ESXi DNS failures. Match the logs before applying a fix.

Introduction

A Supervisor deployment can look deceptively close to completion when it reaches the host configuration stage. The control-plane workflow has started, the cluster appears in Workload Management, and the interface reports that the Kubernetes node agent is being installed and started on the ESXi hosts.

Then nothing moves.

The Host Config Status remains Configuring, the same task stays pending, and repeated Apply Solution tasks appear in vCenter. Retrying the deployment does not change the result because the orchestration layer is waiting for a host-level prerequisite that vSphere Lifecycle Manager cannot satisfy.

Broadcom KB 393663 documents one specific version of this failure. The Spherelet VIB, which provides the Kubernetes node-agent function on each participating ESXi host, cannot be installed because the hosts are noncompliant with or incompatible with the vLCM desired image. Broadcom also identifies legacy BIOS boot mode as a condition that can leave hosts noncompliant and block the workflow.

The operational challenge is that the UI message is not unique to this root cause. Several different Spherelet delivery failures can stop at the same stage. A useful runbook must therefore separate symptom confirmation from remediation, preserve evidence, and avoid bypassing lifecycle controls before the actual failure domain is known.

Scenario and Scope

This runbook is written for administrators troubleshooting an initial vSphere Supervisor deployment with the following characteristics:

  • vCenter Server 8.x
  • ESXi 8.x hosts
  • vSphere Supervisor or vSphere Kubernetes Service
  • ESXi hosts managed with a vSphere Lifecycle Manager image
  • Host Config Status stuck at Configuring
  • The pending step reads Installed and Started Kubernetes Node Agent on the ESXi Host

The primary path follows KB 393663. Additional diagnostic branches are included because the same visible stage can be produced by other Spherelet, catalog, depot, or name-resolution failures.

This is not a generic Kubernetes troubleshooting article. The blocked component is the vSphere host lifecycle workflow that must place the Spherelet VIB onto each ESXi host and bring the node agent into the desired state.

Why the Deployment Stops at This Stage

Supervisor enablement crosses several control boundaries. vCenter coordinates the workflow, Workload Control Plane tracks Supervisor state, vSphere Lifecycle Manager evaluates and applies the host image, and each ESXi host must accept the Spherelet VIB.

The key dependency is easy to miss:

What to notice is that Supervisor is not directly copying a standalone agent onto an otherwise unmanaged host. The installation is reconciled through the host lifecycle system. If vLCM cannot prove that the desired image is valid for the hosts, the Spherelet installation is blocked with it.

That is why repeated retries are usually ineffective. The control plane is reissuing work against an unresolved lifecycle condition.

Symptoms That Match KB 393663

Do not classify the incident from the pending task alone. Confirm the complete signature.

vSphere Client Symptoms

The strongest UI indicators are:

  • Supervisor deployment is stuck at Configuring Host Nodes.
  • Host Config Status remains Configuring.
  • The pending task references installation and startup of the Kubernetes node agent.
  • A related message may report that the Kubernetes node agent stopped for maintenance mode.
  • Repeated Apply Solution tasks appear in recent tasks.
  • Cluster image compliance reports one or more hosts as noncompliant or incompatible.

WCP Log Signature

On the vCenter Server Appliance, review the Workload Control Plane log:

grep -Ei "Solution specification|incompatible with hosts|spherelet|vLCM|PMan" \
  /var/log/vmware/wcp/wcpsvc.log | tail -n 200

For the KB 393663 path, look for an error stating that the solution specification in the image is incompatible with named ESXi hosts. The surrounding operation identifier commonly includes a vLCM host-add or apply operation.

The exact timestamp, host identifiers, cluster identifiers, and attempt numbers will differ. The important part is the combination of Apply Solution, incompatible hosts, and the affected host list.

vSphere Lifecycle Manager Evidence

In the vSphere Client, navigate to the affected cluster and open:

  1. Updates
  2. Hosts
  3. Image
  4. Check Compliance

Capture the result before making changes. The compliance details should identify whether the conflict is tied to the base ESXi image, vendor add-on, firmware and driver add-on, additional component, solution component, hardware warning, or another image constraint.

Diagnostic Decision Tree

The visible message identifies the stalled stage. The logs identify the actual failure domain.

This distinction prevents a common troubleshooting mistake: using a valid fix from the wrong Spherelet failure scenario.

Prerequisites and Safety Checks

Before remediation, establish the operational boundaries.

Required Access

You need:

  • vCenter administrative access
  • permission to inspect and edit the cluster vLCM image
  • root access to the vCenter Server Appliance for log review
  • temporary ESXi Shell or SSH access for read-only host checks
  • out-of-band hardware management access if firmware mode may change
  • access to the hardware vendor support matrix and approved OEM image information
  • an approved maintenance window for host remediation or reboot

Environment Checks

Confirm the following before changing the desired image:

  • DRS and evacuation capacity are sufficient for maintenance mode.
  • vSAN health and resynchronization state are understood if the cluster uses vSAN.
  • NSX host transport-node health is stable if NSX is present.
  • Backups and recovery procedures are current for vCenter and critical management components.
  • The current desired image is documented, including base image, vendor add-on, firmware add-on, and extra components.
  • No unrelated lifecycle operation is already running.

Change-Control Warning

A vLCM image edit is not a cosmetic change. It can alter drivers, firmware coordination, host components, and reboot requirements across the cluster.

Do not remove an OEM add-on or third-party component merely to make the compliance screen turn green. Identify the owner, support requirement, and replacement path first.

Runbook Stage: Confirm the Exact Failure Signature

Start with the lowest-risk evidence collection.

Capture the Supervisor State

Record:

  • Supervisor name
  • cluster name
  • affected ESXi hosts
  • Host Config Status
  • exact pending task text
  • first observed failure time
  • recent deployment or upgrade changes
  • screenshots of repeated Apply Solution tasks

Review WCP and vLCM Logs

On the vCenter Server Appliance, collect the relevant WCP lines:

grep -Ei "Apply Solution|Solution specification|incompatible with hosts|spherelet|PMan|vLCM" \
  /var/log/vmware/wcp/wcpsvc.log | tail -n 300

If the WCP log points to an Apply Solution failure, correlate it with the Update Manager log:

grep -Ei "ClusterApplySolutionTask|VibDownloadError|incompatible|spherelet" \
  /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log | tail -n 300

Successful confirmation for the KB 393663 path means you can answer yes to all three questions:

  • Is Supervisor stalled at the Kubernetes node-agent stage?
  • Is vLCM reporting host noncompliance or incompatibility?
  • Do the WCP or Update Manager logs tie the failure to Apply Solution or the desired image?

If not, stop and use the differential diagnosis section later in this article.

Runbook Stage: Capture Host and Image State

Collect a baseline from every ESXi host before remediation. The commands below are read-only.

# Record the ESXi release and build
vmware -vl

# Record the installed image profile
esxcli software profile get

# Check whether the Spherelet VIB is already installed
esxcli software vib list | grep -i spherelet

# Check whether the host presents UEFI or legacy BIOS firmware
vsish -e get /hardware/firmwareType

For environments where normal management agents are impaired, Broadcom examples sometimes use localcli to list VIBs. Keep any localcli use read-only unless Broadcom Support explicitly directs otherwise. Changing host state with localcli bypasses normal management coordination and can create a second problem while you are investigating the first.

Create a small evidence table for the change record:

HostESXi buildImage profileSpherelet presentFirmware typevLCM status
esx01Record valueRecord valueYes or noUEFI or Legacy BIOSCompliant, noncompliant, or incompatible
esx02Record valueRecord valueYes or noUEFI or Legacy BIOSCompliant, noncompliant, or incompatible
esx03Record valueRecord valueYes or noUEFI or Legacy BIOSCompliant, noncompliant, or incompatible

This table often reveals whether the problem is cluster-wide or isolated to one host with a different firmware mode, OEM image, driver set, or build.

Runbook Stage: Interpret the vLCM Compliance Failure

The word noncompliant is not a diagnosis. Expand the details until the conflicting element is known.

Base ESXi Image Mismatch

The desired cluster image may specify a different ESXi release or patch level than one or more hosts can accept. Validate:

  • the exact target ESXi version and build
  • server hardware compatibility
  • vCenter and ESXi interoperability
  • Supervisor support for the selected release
  • boot-device capacity and firmware requirements

Do not select a newer base image solely because it appears in the depot. The target must be supported by the server platform and the surrounding VCF or vSphere stack.

OEM Vendor Add-On Mismatch

If the hosts were installed from an OEM-customized image, the desired image should usually include the correct vendor add-on for the server model and target ESXi release.

A missing, stale, or incompatible vendor add-on can create driver and provider differences that prevent compliance. Verify the add-on against the hardware vendor’s current support guidance rather than copying the configuration from another cluster with different servers.

Extra Component or Solution Conflict

Additional VIBs, async drivers, hardware-management providers, NSX components, storage plug-ins, or partner agents can make the desired image incompatible.

For each conflicting component, determine:

  • which product installed it
  • whether it is still required
  • which team owns it
  • whether a compatible version exists
  • whether the component must be upgraded before host remediation
  • whether removal is supported and reversible

The goal is not to produce a minimal image. The goal is to produce a supported image that represents the actual platform dependencies.

Depot or Repository Availability

A host can appear incompatible when vLCM cannot complete mandatory validation because the configured online or offline depot is unavailable. In that case, the image definition may be correct, but the lifecycle service cannot retrieve or validate its components.

If Update Manager logs show VibDownloadError, repository reachability, certificate trust, proxy configuration, or depot configuration becomes the first repair path. Do not rewrite the desired image to work around an unavailable source.

Runbook Stage: Resolve Legacy BIOS Boot Mode

KB 393663 specifically identifies legacy BIOS boot as a condition that can leave ESXi hosts noncompliant and prevent Spherelet installation.

Use the firmware check on every host:

vsish -e get /hardware/firmwareType

If the output reports Legacy BIOS, treat the correction as a hardware and boot-path change, not as a quick Supervisor setting.

Plan the Transition Before Changing Firmware

At minimum, document:

  • current boot device and boot order
  • disk partitioning and bootbank health
  • server model and firmware baseline
  • storage adapter and network adapter PCI identities
  • active vmnic names, MAC addresses, and uplink mappings
  • out-of-band console access
  • vendor support for UEFI on the exact server generation
  • recovery path if ESXi does not boot or networking changes

Switching firmware mode can affect how the platform presents boot devices and PCI hardware. A safe change includes console access, a host evacuation plan, post-change network validation, and a tested recovery procedure.

Do not toggle legacy BIOS to UEFI across every host at once. Use a pilot host, validate boot, storage, networking, vCenter connectivity, and vLCM compliance, then continue through the cluster according to the approved maintenance plan.

Runbook Stage: Correct the Desired Cluster Image

In the vSphere Client:

  1. Select the affected cluster.
  2. Open Updates.
  3. Select Hosts, then Image.
  4. Review the current desired image.
  5. Select the supported ESXi base version.
  6. Add or correct the OEM vendor add-on when required.
  7. Validate firmware and driver add-ons if a hardware support manager is integrated.
  8. Review additional components and solution dependencies.
  9. Save the image.
  10. Run Check Compliance again.

A useful review sequence is:

Do not proceed to remediation while the reason for incompatibility remains unexplained. A clean precheck is the change gate, not an optional report.

Runbook Stage: Remediate the Hosts Through vLCM

Once the desired image is correct and the compliance analysis is understood, remediate through the supported vLCM workflow.

Use Controlled Host Sequencing

Where cluster capacity and the platform support model permit it:

  • remediate one host first
  • confirm maintenance mode entry and workload evacuation
  • monitor image staging and installation
  • reboot only when required
  • validate management, storage, network, and cluster health
  • confirm the host returns compliant
  • then continue with the remaining hosts

For vSAN clusters, select an evacuation mode that matches the maintenance duration and available capacity. For NSX-backed clusters, verify transport-node and tunnel health after each host returns. For VCF-managed environments, ensure the action remains consistent with the supported VCF lifecycle path and does not create undocumented out-of-band drift.

Do Not Force Past Hardware Warnings

A command-line option that suppresses a warning does not resolve the underlying support condition. Legacy BIOS, unsupported hardware, incompatible drivers, or insufficient boot media should be corrected or explicitly accepted through the appropriate vendor and support process.

The objective is a compliant, supportable host image, not merely a completed task.

Validation After Remediation

Validation should prove both lifecycle recovery and Supervisor recovery.

Validate vLCM State

Confirm that:

  • every participating host is compliant with the desired image
  • no host remains incompatible
  • no Apply Solution task is repeating
  • recent Update Manager logs contain no new image or VIB installation failures

Validate Spherelet Installation

On each ESXi host:

esxcli software vib list | grep -i spherelet

Successful output should show a Spherelet package entry. Record the installed version for all hosts and confirm the versions are consistent with the target Supervisor and host image.

Validate Supervisor Host Configuration

In Workload Management, confirm that:

  • Host Config Status leaves Configuring
  • each host becomes ready for Supervisor use
  • the Kubernetes node-agent task completes
  • the overall Supervisor state is healthy
  • no new host configuration error appears

Recheck WCP Logs

Review recent WCP activity after remediation:

grep -Ei "spherelet|Apply Solution|incompatible with hosts|error wcp" \
  /var/log/vmware/wcp/wcpsvc.log | tail -n 200

The absence of the old incompatibility error is necessary, but it is not sufficient by itself. The operational pass condition is successful Spherelet presence plus a healthy Supervisor host state.

When the Same Message Is Not KB 393663

The UI status is a location in the workflow, not a unique error code. Use the evidence below to avoid applying the wrong repair.

EvidenceMore likely failure domainRecommended next action
Solution specification in the image are incompatible with hosts plus vLCM noncomplianceKB 393663 image or firmware pathCorrect desired image, resolve firmware issue, and remediate through vLCM
catalog is not ready, supervisor content is being processed, or malformed Supervisor OVF templateInvalid or unhealthy content libraryRepair or unassign the affected Supervisor content library according to the matching Broadcom KB
VibDownloadError for an online or custom depot componentvLCM depot, proxy, certificate, or repository reachabilityRestore depot access, rerun compliance, then allow orchestration to resume
Spherelet VIB download reports Name or service not knownESXi DNS or name-resolution failureCorrect ESXi DNS configuration and verify resolution of the vCenter artifact endpoint
Spherelet is absent, but none of the documented signatures appearUnclassified host configuration failurePreserve WCP, Update Manager, and ESXi logs; engage Broadcom Support before destructive recovery

A manual Spherelet VIB copy and install appears in separate Broadcom troubleshooting articles for other signatures and versions. That does not make it the preferred response to KB 393663. If vLCM reports the desired image as incompatible, bypassing vLCM can create image drift and leave the underlying lifecycle problem unresolved.

Common Mistakes That Make the Incident Harder

Reinstalling ESXi Before Reading the Logs

A host rebuild can remove the visible symptom while destroying the evidence that explains why the lifecycle workflow failed. It can also reintroduce the same problem if the rebuilt host uses the same incompatible image or legacy BIOS mode.

Deleting and Recreating the Supervisor Immediately

Recreating Supervisor repeats the workflow against unchanged hosts. Collect the WCP operation identifiers, vLCM compliance report, and host state first.

Manually Installing Spherelet as the First Fix

Manual installation can be valid only when the matching Broadcom procedure supports it for the exact failure. It should not replace correction of vLCM noncompliance.

Removing OEM or Third-Party Components Without Ownership Review

A component that blocks compliance may still provide a required NIC, storage, hardware-management, security, or NSX function. Removing it without a dependency review can trade a deployment failure for a host outage.

Changing BIOS Mode Without Console and Recovery Planning

Firmware changes can affect the boot path and device presentation. Out-of-band access and a pilot-host validation are mandatory operational controls.

Treating One Compliant Host as Proof of Cluster Readiness

Supervisor depends on every participating host reaching the required state. Compare image, build, firmware, and Spherelet evidence across the complete cluster.

Rollback and Fallback Guidance

Rollback depends on how far the change progressed.

If the Desired Image Was Edited but Not Remediated

Restore the previously documented desired image components, save the image, and rerun compliance. Because host state has not changed, this is the lowest-risk rollback point.

If a Pilot Host Was Remediated

Validate whether vLCM offers a supported rollback for the applied image and whether the previous image remains compatible with vCenter, Supervisor, hardware, and other cluster solutions. Do not assume that a host-level software rollback is always available or safe.

If rollback is not supported, the recovery path may require a vendor-approved image, host reinstallation, or Broadcom Support guidance. Preserve the remediation logs before taking that path.

If Firmware Mode Was Changed

Use out-of-band console access to evaluate boot, storage, and network state. Reverting firmware mode may be part of recovery, but only after confirming the boot-device implications and vendor guidance. Do not alternate repeatedly between legacy BIOS and UEFI without documenting each state.

If Supervisor Remains Stuck After Compliance Is Restored

Do not immediately repeat all remediation. Collect fresh evidence and compare the new error with the original one. The lifecycle incompatibility may be fixed while a second catalog, DNS, certificate, or connectivity issue is now visible.

Evidence to Collect Before Escalation

A strong support bundle shortens the path from generic symptom to actionable diagnosis.

Collect:

  • Supervisor name and deployment timestamp
  • affected cluster and host names
  • screenshots of Host Config Status and pending task details
  • recent task history showing repeated Apply Solution attempts
  • relevant wcpsvc.log excerpts with operation identifiers
  • relevant Update Manager log excerpts
  • vLCM desired image definition
  • vLCM compliance report for each host
  • ESXi version and build from each host
  • installed image profile from each host
  • Spherelet VIB presence and version
  • firmware type from each host
  • recent changes to ESXi images, vendor add-ons, firmware, DNS, proxy, content libraries, or depots
  • confirmation of hardware compatibility and OEM support
  • vCenter and ESXi support bundles when requested

The best escalation question is not, “Why is Supervisor stuck?” It is, “Why does vLCM consider this exact desired image incompatible with these exact hosts, and what supported change restores Spherelet installation?”

Conclusion

The pending message Installed and Started Kubernetes Node Agent on the ESXi Host is not the diagnosis. It is the point where Supervisor enablement is waiting for host lifecycle reconciliation to finish.

For the condition documented in KB 393663, the decisive evidence is vLCM noncompliance or incompatibility combined with Apply Solution failures in the WCP workflow. The supported response is to correct the desired cluster image, resolve legacy BIOS where applicable, and remediate the hosts through vSphere Lifecycle Manager. Once the hosts are compliant, the Spherelet VIB can be installed and Supervisor configuration can continue.

The operational discipline matters as much as the technical fix. Preserve the logs, validate every host, treat firmware conversion as a planned maintenance activity, and do not bypass vLCM merely because a manual VIB command exists in a different KB.

Fix the lifecycle state first. Then prove that Spherelet is installed, the host configuration is ready, and Supervisor has returned to a healthy operating state.

External References

Leave a Reply

Discover more from Digital Thought Disruption

Subscribe now to keep reading and get access to the full archive.

Continue reading