Site icon Digital Thought Disruption

How to Deploy VMware Private AI Foundation with NVIDIA on VCF 9.1

Paul Bryant

TL;DR

Deploying VMware Private AI Foundation with NVIDIA on VCF 9.1 is not a single-product installation. It is an integrated platform deployment spanning the VCF workload domain, GPU-enabled ESXi hosts, NVIDIA drivers and licensing, vSphere Supervisor, namespaces, Private AI Services, Harbor, identity, networking, certificates, and the AI consumption model.

The most important design decision happens before Private AI Services is installed: determine whether users need individually managed Deep Learning VMs, cloud-native VKS clusters, or both. That decision influences GPU allocation, networking, storage, automation, lifecycle management, and recovery.

For environments that require VCF Automation, multi-tenancy, self-service networking, and governed AI catalogs, design the Supervisor around VCF Networking with VPC from the beginning. Moving later from VDS-based Supervisor networking to VPC networking is a redeployment boundary, not a routine configuration change.

Introduction

VMware Private AI Foundation with NVIDIA brings enterprise AI workloads into the same operational framework used to manage VMware Cloud Foundation. That is valuable because most organizations do not need another isolated AI island. They need GPU infrastructure that fits existing practices for identity, networking, storage, lifecycle management, security, and support.

The challenge is that the solution crosses several administrative boundaries.

The VCF team owns the workload domain and Supervisor. The virtualization team owns ESXi hosts and GPU allocation. The platform team may own VKS and namespaces. The security team owns certificates, registry policy, and identity. Data science teams consume Deep Learning VMs, Kubernetes clusters, models, notebooks, and inference endpoints.

A successful deployment therefore depends less on completing an installation wizard and more on making those boundaries explicit.

This deep dive provides an implementation path for VMware Private AI Foundation with NVIDIA on VCF 9.1. It covers the infrastructure foundation, Supervisor architecture, Private AI Services, Harbor model distribution, Deep Learning VM and VKS consumption models, validation, lifecycle dependencies, and realistic recovery boundaries.

Deployment Scope and Assumptions

This design assumes the following baseline:

This is an architecture and implementation guide, not a replacement for the exact version-specific procedures in Broadcom and NVIDIA documentation. Names of service definitions, packages, images, and configuration objects can change between releases.

Understand the Platform Before Deploying It

VMware Private AI Foundation with NVIDIA is the overall solution. Private AI Services is one of the platform services operating within that solution.

The deployment can be understood as five cooperating layers:

  1. VCF infrastructure: Workload domains, clusters, ESXi hosts, networking, storage, lifecycle management, and identity integration.
  2. NVIDIA acceleration: Physical GPUs, host drivers, guest drivers, vGPU profiles or direct device assignment, licensing, and GPU telemetry.
  3. Supervisor services: Kubernetes control-plane functions, namespaces, resource policy, service operators, and workload APIs.
  4. AI platform services: Model Gallery, Model Runtime, data indexing, agent capabilities, API access, and related Private AI Services functions.
  5. AI consumption: Deep Learning VMs, VKS clusters, notebooks, model endpoints, applications, and automation pipelines.

Private AI Services does not replace the infrastructure underneath it. It automates and governs the creation of AI services on top of that infrastructure.

Reference Architecture and Ownership Boundaries

The following architecture shows the major components and where the most important ownership boundaries sit.

The diagram highlights three principles.

First, the Supervisor and Private AI Services are control layers. The actual GPU compute remains on ESXi hosts and inside workload VMs.

Second, Harbor is part of the production delivery path. It is not merely a convenient place to cache container images.

Third, enterprise services such as DNS, certificates, identity, and time synchronization are direct dependencies. Treating them as afterthoughts is one of the fastest ways to create an unreliable platform.

CapabilityAccountable ownerPrimary responsibilities
VCF workload domainVCF platform teamWorkload-domain health, cluster design, capacity, storage, networking, lifecycle
GPU-enabled ESXi hostsVirtualization and hardware teamsFirmware, BIOS, ESXi image, NVIDIA host driver, device health, maintenance
NVIDIA licensingPlatform or licensing teamLicense service, entitlements, client configuration, expiration monitoring
SupervisorVCF platform teamControl-plane sizing, networking mode, namespaces, service installation
VCF AutomationCloud platform teamOrganizations, projects, catalogs, policies, entitlements, self-service
Private AI ServicesAI platform teamService configuration, namespace activation, model services, platform updates
VKS clustersKubernetes platform teamCluster classes, GPU workers, GPU Operator, policies, observability
Deep Learning VMsAI platform and consumer teamsImages, VM classes, notebooks, frameworks, persistent user data
HarborPlatform engineering or DevSecOpsProjects, RBAC, certificates, retention, replication, backup, scanning
Model approvalAI governance and securityProvenance, licenses, validation, promotion, acceptable-use controls
Application endpointsApplication or MLOps teamDeployment, testing, scaling, monitoring, rollback, service ownership

The exact names of the teams matter less than ensuring every capability has an accountable owner.

Choose the AI Consumption Model First

Private AI Foundation supports more than one workload model. Do not deploy every capability simply because it is available.

Start with the way users need to consume the platform.

Deep Learning VMs

A Deep Learning VM is usually the fastest path for individual experimentation, notebook development, model evaluation, and framework-based work.

It is appropriate when the user needs:

The operational risk is that the VM can become a pet. Researchers may install packages manually, store notebooks locally, download models directly, and create an environment that cannot be reproduced.

For production use, the VM should be treated as rebuildable. Store notebooks, data, source code, environment definitions, and model artifacts outside the VM. Changes made only inside a deployed Deep Learning VM may disappear when the deployment is deleted or replaced.

VKS Clusters with GPU Workers

VKS is the stronger fit for repeatable, cloud-native AI services.

Use it when the workload requires:

VKS introduces more moving parts. Worker images, Kubernetes versions, GPU Operator components, container runtimes, network policies, storage classes, and workload manifests all become part of the lifecycle chain.

Deep Learning VM and VKS Comparison

Decision areaDeep Learning VMVKS cluster
Best fitIndividual development and experimentationShared or production AI services
User experienceFamiliar VM, shell, IDE, or notebookKubernetes API, pipelines, and manifests
Deployment unitVirtual machinePods, services, operators, and clusters
ScalingPrimarily vertical or additional VMsHorizontal and declarative
GPU deliveryvGPU profile or assigned deviceGPU-capable worker VMs and Kubernetes resources
ReproducibilityDepends on image and automation disciplineStrong when manifests and images are version controlled
Operational complexityLower initial complexityHigher platform complexity
Recovery modelRebuild VM and reattach external stateReconcile cluster and restore persistent state
Strongest use caseExploration, development, isolated testingInference, shared services, MLOps, production applications

A mature platform commonly provides both. The mistake is allowing both paths to evolve without common governance for models, identities, networking, registries, telemetry, and lifecycle management.

Prepare the VCF Workload Domain

The AI workload domain is the infrastructure boundary that protects both the management plane and the GPU environment.

A dedicated domain is not mandatory for every lab, but it is the preferred production pattern when GPU workloads have materially different capacity, maintenance, security, and lifecycle requirements.

Validate the VCF Bill of Materials

Before adding AI services, confirm that the workload domain is running a supported VCF 9.1 bill of materials.

Validate:

Do not treat “VCF 9.1” as sufficient compatibility evidence. Patch levels and driver releases can materially change the supported combinations.

Design the GPU Cluster

The GPU cluster should be designed around the AI workload profile, not around the number of GPUs that happened to be purchased.

Collect the following requirements:

At least one GPU host should remain available for maintenance or workload recovery where availability requirements justify it. A cluster with every GPU permanently allocated has no practical maintenance margin.

Validate the Hardware and Firmware

Confirm the entire server platform, not only the GPU model.

Review:

Firmware drift across otherwise identical GPU hosts can produce difficult placement and stability problems. Include firmware baselines in the same compliance process used for ESXi.

Manage the NVIDIA Host Driver as Part of the ESXi Image

Avoid treating the NVIDIA host driver as an informal post-installation package.

Where supported by the chosen deployment model, include the driver or component in the desired vSphere Lifecycle Manager image. This produces a repeatable host baseline and makes drift visible.

The following must remain compatible:

A platform upgrade is not complete when ESXi reports healthy. It is complete when a representative AI workload has successfully consumed the GPU and passed validation.

Select the GPU Allocation Method

The allocation model affects performance, mobility, scheduling, and operations.

vGPU profiles allow a physical GPU to be divided into supported profiles. This is useful for environments that need controlled sharing and predictable per-workload GPU memory.

Direct device assignment gives a workload more direct access to the GPU. It can be appropriate for workloads that need exclusive devices or the highest supported performance, but it can reduce mobility and create stricter placement constraints.

MIG-backed resources, where supported by the selected GPU and software stack, create hardware-partitioned GPU instances. MIG should be treated as a separate design decision because profile changes affect capacity planning, scheduling, and active workloads.

Do not mix allocation models casually across hosts in the same placement pool. The scheduler must be able to find compatible devices, profiles, drivers, and host capacity.

Design the Supervisor and Namespace Architecture

The vSphere Supervisor provides the control plane through which Private AI Services and AI workloads are managed.

This is not simply a Kubernetes feature toggle. Its networking model, control-plane size, namespace structure, and service dependencies influence almost every later platform decision.

Choose VPC or VDS Networking Deliberately

VCF 9.1 supports Supervisor architectures based on VCF Networking with VPC or vSphere Distributed Switch networking.

The decision should be driven by the required consumption model.

RequirementSupervisor with VCF Networking and VPCSupervisor with VDS
VCF Automation consumptionSupported and expected for the documented multi-tenant modelManual consumption path
Self-service networkingStrong fitLimited and administrator-driven
Multi-tenancyIntegrated organizations, projects, namespaces, and VPCsPrimarily namespace and infrastructure administration
Network policyVPC and distributed policy capabilitiesVLAN and external network controls
Load balancingFoundation Load Balancer or Avi options, depending on designExternal load-balancing integration required
Operational footprintMore platform components and network designSimpler initial topology
Long-term self-service platformPreferredBetter for constrained or manually managed environments

The documented VCF Automation consumption model depends on Supervisor networking with VPC. Without VCF Automation, administrators can still activate and operate Private AI Services using the VCF consumption CLI, Kubernetes manifests, and the Private AI Services interface, but they lose the same integrated self-service infrastructure experience.

This choice is a recovery boundary. Moving from VDS-based Supervisor networking to VPC networking in VCF 9.1 requires planned Supervisor redeployment. It should not be placed on a “we will fix it later” list.

Size the Supervisor for the Target Platform

Supervisor sizing should account for:

Plan conservatively because the control-plane size can be increased, but reducing an oversized control plane is not the same routine operation.

Create a Namespace Strategy

Namespaces should reflect ownership, security, quota, and lifecycle boundaries.

Useful patterns include:

Organization
  |
  +-- Development
  |     +-- Team A AI namespace
  |     +-- Team B AI namespace
  |
  +-- Test
  |     +-- Shared validation namespace
  |
  +-- Production
        +-- Application 1 inference namespace
        +-- Application 2 RAG namespace

Avoid placing unrelated development and production services in one namespace merely to reduce administrative work.

For each namespace, define:

Namespaces are governance objects, not folders.

Design Harbor as a Production Service

Harbor sits in the artifact path between software producers, model curators, the AI platform, and runtime workloads.

A registry outage can prevent new VKS workers, services, models, or inference endpoints from starting. A registry compromise can introduce unapproved code or models into the AI environment.

Treat Harbor as Tier 1 platform infrastructure when production workloads depend on it.

Separate Artifact Types and Trust Levels

A practical project design might include:

Harbor projectIntended content
platform-imagesApproved infrastructure and service images
gpu-componentsGPU Operator and related platform artifacts
dlvm-contentApproved Deep Learning VM supporting artifacts
model-quarantineNewly acquired or unvalidated models
model-approvedReviewed and approved production models
team-developmentTeam-owned development images and models
production-appsSigned production application images

Do not allow external models to move directly from an internet source into a production model gallery.

A safer promotion flow is:

Apply Registry Controls

Recommended controls include:

Traditional container vulnerability scanning alone does not validate a model. Model promotion should also consider licensing, origin, supported framework, serialization format, expected behavior, embedded code risk, and data-governance constraints.

Plan for Large Model Artifacts

Models can be much larger than standard application images.

Capacity planning must include:

A Harbor deployment sized only for container images can become a bottleneck once multiple large language models and embedding models are introduced.

Prepare Networking, DNS, Certificates, and Identity

Most failed deployments are not caused by the GPU. They are caused by dependencies around the GPU.

Build an Endpoint Inventory

Create an endpoint register before installation.

Endpoint or serviceConsumersRequired control
SDDC Manager and vCenterPlatform administrators and automationAdministrative DNS, certificates, restricted access
Supervisor APIAdministrators, automation, VKS operationsLoad balancing, DNS, trusted certificate
Private AI Services endpointsAI users and applicationsIngress, DNS, certificates, authentication
HarborESXi-adjacent services, VMs, VKS workers, administratorsTrusted CA, RBAC, adequate bandwidth
Identity providerVCF Automation, platform services, usersOIDC or supported federation, group mapping
NVIDIA licensingGPU-enabled guest workloadsReliable reachability and monitoring
External repositories or mirrorsPlatform deployment componentsControlled egress or internal mirrors
External PostgreSQL or vector servicesRAG and data-indexing servicesTLS, authentication, backup, latency monitoring
DNS and NTPAll componentsRedundancy and consistent configuration

The exact set of endpoints depends on whether the environment is connected, restricted, or fully disconnected.

Use Predictable DNS

Forward and reverse DNS should be validated for all critical platform endpoints before installation.

Check from more than one network location:

nslookup harbor.ai.example
nslookup supervisor-api.ai.example
nslookup private-ai.ai.example
nslookup identity.ai.example

A successful lookup from an administrator workstation does not prove that VKS workers, Supervisor components, or Deep Learning VMs can resolve the same records.

Establish Certificate Trust

Use certificates with complete chains and correct subject alternative names. Import the issuing CA chain into every component that must trust Harbor or another private endpoint.

A basic inspection can be performed with:

openssl s_client \
  -connect harbor.ai.example:443 \
  -servername harbor.ai.example \
  -showcerts

Verify:

Disabling certificate validation to complete an installation is not a deployment solution. It creates a hidden day-two dependency that usually reappears during upgrades or node replacement.

Map Enterprise Identity to Platform Roles

Define group-based access before onboarding users.

A practical model separates:

Avoid granting broad platform administration to AI consumers simply because finer-grained roles have not yet been designed.

Install Private AI Services

Private AI Services follows a two-level operating model.

At the Supervisor level, the service operator and supporting definitions are installed and managed by platform administrators.

At the namespace level, an organization administrator or authorized AI platform operator activates and configures the service for a specific tenant or project.

Supervisor-Level Installation

The infrastructure administrator should:

  1. Confirm the target Supervisor is healthy.
  2. Confirm required service artifacts are available from approved registries or internal mirrors.
  3. Configure registry authentication.
  4. Import the CA trust bundle required for Harbor and other private endpoints.
  5. Register or install the Private AI Services definition on the Supervisor.
  6. Verify that the service operator is running.
  7. Confirm that service resources can be made available to namespaces.
  8. Record the installed service version and artifact digests.

Do not continue to namespace activation while the Supervisor-level operator is degraded or unable to reach its required artifact sources.

Namespace-Level Activation

For each approved namespace:

  1. Apply the namespace storage, network, CPU, memory, and GPU policies.
  2. Assign the required identity groups and permissions.
  3. Configure Harbor endpoints and credentials.
  4. Configure NVIDIA credentials or secrets required by the selected services.
  5. Add trusted CA material.
  6. Configure external data services when required.
  7. Activate Private AI Services through the selected consumption path.
  8. Observe the service reconciliation process.
  9. Validate platform pods and endpoints.
  10. Test model acquisition, validation, and deployment.

In a VCF Automation deployment, much of this can be presented through organizations, projects, catalogs, policies, and guided provisioning. In a manually administered environment, operators use the VCF consumption CLI, Kubernetes manifests, and kubectl.

The interface is different, but the underlying requirements are the same.

Finalize Deep Learning VM and VKS Capabilities

Private AI Services alone does not make every workload type ready for use. The supporting images, drivers, packages, and blueprints must also be prepared.

Deep Learning VM Readiness

Validate:

Deploy a test VM and confirm:

nvidia-smi

Successful output should show the expected GPU or vGPU profile, driver version, memory, utilization, and licensing state where applicable.

Then validate the actual framework, not only the driver. Run a small supported CUDA, PyTorch, or TensorFlow workload that allocates GPU memory and completes a calculation.

VKS GPU Cluster Readiness

Validate:

After the cluster is created, inspect GPU resources:

kubectl get nodes \
  -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu

Inspect platform components and events:

kubectl get pods -A

kubectl get events -A \
  --sort-by=.lastTimestamp

The cluster is not ready merely because the control plane is healthy. At least one test pod must request a GPU, start successfully, execute a GPU workload, and release the resource cleanly.

Publish Governed AI Catalog Items

VCF Automation can present ready-to-use catalog items such as:

Catalog items should be versioned just like application artifacts.

For every published version, record:

Do not edit the only production blueprint in place. Create a draft version, test it in a non-production project, validate the resulting workload, and then promote it.

Follow a Controlled Deployment Sequence

The full dependency order should remain visible throughout implementation.

Compatibility Gate

Do not begin until the component matrix has been reviewed and signed off.

The matrix should include:

Infrastructure Gate

The workload domain must pass:

Platform Gate

The Supervisor and namespaces must pass:

AI Service Gate

Private AI Services must pass:

Each gate creates an evidence boundary. If model deployment fails, the team should know whether infrastructure, Supervisor, registry, service, or workload validation last succeeded.

Validate the Complete Deployment

Validation should proceed from the lowest layer to the user-facing service.

Infrastructure Validation

Confirm:

Deep Learning VM Validation

Confirm:

VKS Validation

Confirm:

Private AI Services Validation

Confirm:

Evidence to Retain

Capture:

This evidence becomes the baseline for upgrades and support escalations.

Operate the Platform After Deployment

Day-two operations must cover the entire dependency chain, not only Private AI Services.

Capacity and Placement

Monitor:

Low GPU utilization does not always mean excess capacity. The workload may be blocked by CPU preprocessing, storage throughput, network transfer, small batches, or application serialization.

Catalog and Blueprint Lifecycle

Maintain separate states for:

Before publishing a new version, deploy it into a validation project and test the complete outcome. That includes GPU access, package installation, registry trust, identity, storage, and application functionality.

Certificate and Secret Rotation

Maintain an inventory of:

Test rotation in a non-production namespace before production. A certificate replacement may require trust-bundle updates in existing Deep Learning VMs, VKS nodes, operators, and application containers.

Model Governance

Models should move through a controlled lifecycle:

Track the model digest rather than relying only on a mutable name or tag. Retain the evaluation results, approval record, model license, source, framework requirements, deployment configuration, and rollback target.

Upgrade Planning

Treat the stack as a compatibility set.

A useful planning order is:

  1. Identify the target VCF and ESXi baseline.
  2. Identify compatible GPU firmware and host driver.
  3. Identify compatible guest drivers and GPU Operator versions.
  4. Identify supported Deep Learning VM and VKS releases.
  5. Identify the matching Private AI Services release.
  6. Validate Harbor and external-service compatibility.
  7. Test catalog items and representative models.
  8. Execute the vendor-supported production sequence.
  9. Validate from the user-facing endpoint back down through the stack.

The supported upgrade order for a particular release must come from current release documentation. Do not assume that the planning hierarchy is automatically the execution order.

Define Recovery and Rollback Boundaries

Private AI Foundation does not have a single rollback button.

Each layer contains different state and must have its own recovery method.

BoundaryState to protectPreferred recovery approach
VCF management componentsInventory, workload-domain state, configurationSupported VCF backup and recovery procedures
ESXi GPU hostsDesired image, firmware, drivers, configurationReapply approved host baseline or remediate through lifecycle tooling
NVIDIA driver setHost and guest compatibilityRoll back as a validated compatibility set
SupervisorControl plane, networking model, servicesSupported Supervisor recovery or redeployment procedure
VCF AutomationOrganizations, projects, blueprints, policiesVersion-controlled definitions and platform backup
Private AI ServicesService version and namespace configurationReinstall operator and reapply declarative configuration
HarborDatabase, configuration, projects, artifactsDatabase and artifact backup, replication, tested restore
External PostgreSQL or vector serviceRAG state, metadata, indexesDatabase-native backup, replication, and recovery testing
Deep Learning VMOS image and user environmentRebuild from approved blueprint and restore externalized data
VKS clusterCluster configuration and workloadsReconcile from manifests and restore persistent application data
Model endpointModel digest, runtime configuration, traffic policyRedeploy previous approved version and restore routing
User notebooks and dataIntellectual property and work in progressExternal repositories, object storage, shared file services, or backup

Supervisor Networking Is a Major Boundary

Changing the Supervisor from VDS networking to VPC networking requires planned redeployment in VCF 9.1.

That means teams must preserve or reconstruct:

This is why the network model should be selected according to the target operating model before production workloads are onboarded.

Deep Learning VMs Should Be Rebuildable

A Deep Learning VM is a poor place to hold the only copy of:

The recovery target should be a newly provisioned VM created from a known blueprint, not an attempt to preserve years of undocumented manual customization.

Harbor Requires Independent Recovery Testing

Protect both the metadata database and artifact storage.

A successful database restore without the underlying model blobs is incomplete. Restoring blobs without consistent metadata can also leave the registry unusable.

Test:

Common Deployment Failure Patterns

GPU Is Not Visible in a Deep Learning VM

Likely causes include:

Start with the physical host and work upward. Rebuilding the VM will not fix an unsupported host-driver combination.

VKS Worker Nodes Are Ready but Expose No GPU

Check:

A healthy Kubernetes control plane does not prove that GPU enablement succeeded.

Private AI Services Components Remain Pending

Common causes include:

Review pod events and service reconciliation status before changing configuration.

Harbor Authentication or Model Pulls Fail

Check:

A successful browser login does not prove that a VKS worker or service account can pull the same artifact.

Model Deployment Starts but Never Becomes Ready

Investigate:

Separate “model cannot load” from “endpoint cannot be reached.” They require different evidence and owners.

VCF Automation Catalog Item Fails

Check:

Avoid editing the production blueprint until the failure has been reproduced in a test project.

Host Maintenance Cannot Evacuate GPU Workloads

This is usually a capacity or allocation-model problem rather than a maintenance-mode problem.

Review:

Maintenance behavior must be tested before the first production change window.

Practical Deployment Checklist

Infrastructure

Supervisor and Consumption

Platform Services

Workloads

Operations and Recovery

Conclusion

VMware Private AI Foundation with NVIDIA on VCF 9.1 provides a practical way to operate AI infrastructure through established private-cloud controls. The value is not limited to provisioning a GPU-enabled VM or Kubernetes cluster. The platform can connect infrastructure, namespaces, model distribution, identity, governance, and self-service consumption into one managed operating model.

That value depends on architectural discipline.

Choose the Supervisor networking model before installation. Treat VCF Automation requirements as architecture inputs. Build GPU drivers and licensing into lifecycle management. Separate Deep Learning VM and VKS use cases. Design Harbor as production infrastructure. Externalize user and application state. Validate the complete path from physical GPU to inference endpoint.

Most importantly, define recovery at each platform boundary. The workload domain, Supervisor, Harbor, Private AI Services, VKS clusters, Deep Learning VMs, databases, and model endpoints do not share one recovery mechanism.

When those boundaries are explicit, the platform becomes supportable. Without them, Private AI Foundation can turn into a collection of individually functional components that nobody can upgrade, recover, or operate confidently.

External References

Exit mobile version