Site icon Digital Thought Disruption

Can NVIDIA NIM Really Operate Disconnected? An Enterprise Guide to Air-Gapped Private AI

Paul Bryant

Introduction

NVIDIA NIM can operate without Internet access, but that statement is easy to oversimplify.

The container does not become air-gap ready merely because an administrator pulled it once. A production NIM service depends on a complete software and artifact chain: the OCI image, model weights, model profiles, runtime manifests, GPU drivers, container runtime integration, Kubernetes operators, storage classes, certificates, secrets, and a repeatable update process. If any one of those dependencies still points to a public registry or model repository, the deployment is disconnected only until the next cold start, node replacement, scaling event, or security patch.

NVIDIA’s current NIM for Large Language Models documentation describes a two-phase air-gap workflow: prepare the required assets on a connected system, then transfer and run them locally without outbound access or public-source API keys [1], [2]. As of the July 2026 documentation baseline, the current NIM LLM release is 2.0.8, and the 2.0 architecture differs materially from older 1.x examples that still appear in forum threads and internal runbooks [3], [4].

The practical question is therefore not whether the NIM process can start with the network disconnected. The real question is whether the enterprise has built a closed, supportable, and testable software supply chain around it.

TL;DR

Yes, NVIDIA NIM can run in a restrictive or fully air-gapped environment, provided every runtime dependency is available internally and has been validated under blocked-egress conditions.

A dependable design separates four artifact planes: container images, model artifacts and profiles, Kubernetes and GPU platform dependencies, and entitlement plus evidence records. Mirror NIM and operator images into an internal registry, preload the exact model profiles or local model store, preserve model-free runtime manifests when applicable, establish private-CA trust, and remove public credentials from the disconnected runtime. Then prove cold start, pod restart, node restart, scale-out, and rollback while outbound traffic is denied.

The most common failure is not the GPU. It is an incomplete import: a missing profile, an untrusted certificate, a registry pull failure, a stale 1.x configuration, or a hidden call to NGC, Hugging Face, an object-download endpoint, a Helm repository, or an operating-system package repository.

Define the Operating Model Before Designing the Mirror

Air-gapped is often used as a broad label for several materially different network models. The controls, evidence, and operational burden change depending on which model the organization actually requires.

Operating modelProduction Internet pathArtifact import methodRuntime source of truthOperational implication
ConnectedDirect or enterprise-proxied accessPull on demandPublic and internal sourcesSimplest lifecycle, largest external dependency surface
Restricted egressApproved proxy and allowlistAutomated through controlled egressPublic sources reached through policy controlsRequires proxy, DNS, firewall, and private-CA engineering
Dark siteNo direct production egressPeriodic import from a connected staging enclaveInternal registry, model store, and package repositoriesRequires a formal promotion and evidence process
Fully air-gappedNo network path to public or connected networksReviewed physical, one-way, or out-of-band transferOnly locally controlled services and mediaHighest assurance and highest lifecycle burden

These definitions should be written into the architecture decision record. A dark site with a scheduled import gateway is not the same as a fully air-gapped facility. A restrictive firewall that still allows NGC through a proxy is not disconnected. Neither model is inherently better, but treating them as interchangeable produces weak controls and misleading test results.

The design should also state whether the service must survive only loss of public connectivity or loss of an internal dependency. A NIM pod can be independent of NGC while still depending on Harbor, a shared model volume, DNS, the Kubernetes API, and an internal certificate authority. Air-gapped does not mean dependency-free.

Treat Disconnected NIM as a Four-Plane Supply Chain

The most useful mental model is to separate the assets that teams frequently mix together.

Plane 1: Container images. This includes the NIM image and, for Kubernetes, the NIM Operator, GPU Operator, driver, container toolkit, device plugin, GPU Feature Discovery, DCGM, validation, and other required images. An internal OCI registry such as Harbor belongs here.

Plane 2: Model artifacts. This includes model weights, tokenizer files, configuration files, profile metadata, checksums, and model-free runtime manifests. These artifacts belong in a NIM cache, model store, persistent volume, or supported internal object store. They are not automatically embedded in the NIM image.

Plane 3: Platform dependencies. This includes Helm charts, custom resource definitions, admission webhooks, storage provisioners, ingress components, certificate tooling, monitoring agents, operating-system packages, and Kubernetes base images. The GPU Operator documentation explicitly notes that disconnected installations may require both a local image registry and a local operating-system package repository unless precompiled driver containers remove that package-download dependency [10].

Plane 4: Entitlement and evidence. This includes accepted governing terms, product entitlement, approved model terms, NVIDIA AI Enterprise licensing records where applicable, image digests, model checksums, vulnerability reports, import approvals, and proof that the resulting service works without public access.

A deployment is not ready until all four planes are complete.

Choose the NIM Packaging Model Before Downloading Anything

The import workflow changes depending on whether the team is using a model-specific NIM or a model-free NIM.

Model-Specific NIM

A model-specific NIM image includes a manifest that describes the supported model and profiles, but it does not include the model weights. During connected preparation, the administrator uses the container to discover and download the required profile into the NIM cache. The cache can be transferred directly, or it can be converted into a properly formatted model store with create-model-store [2], [5].

This model is a strong fit when the organization wants a curated, certified deployment with a known profile and GPU combination. The critical design decision is which profile hashes to import. Downloading every profile simplifies fallback but consumes more transfer time and storage. Downloading only one profile reduces footprint but can leave the service unable to start after a GPU topology change.

Model-Free NIM

A model-free NIM can serve a supported model from NGC, Hugging Face, S3-compatible storage, Google Cloud Storage, ModelScope, or local storage [5], [6]. For a strict air gap, the cleanest pattern is usually a local model path or an internally hosted model repository.

When a remote model URI is used during the connected phase, current NIM 2.0 behavior can generate and persist a runtime manifest in NIM_CACHE_PATH. If the same persistent cache is mounted later, NIM can reuse that manifest during an air-gapped restart without making authentication or outbound calls [2], [6]. Losing that cache may force manifest regeneration and break restart in the disconnected zone.

Do Not Carry 1.x Runbooks Forward Unchanged

NIM LLM 2.0 uses a vLLM-based architecture, replaces nim-run with nim-serve, changes configuration behavior, and replaces NIM_MODEL_NAME with NIM_MODEL_PATH for relevant workflows [4]. Older forum posts are still useful as operational signals, but they should not be treated as current deployment instructions.

Pin the exact supported image and profile combination. Do not use a mutable latest tag in a disconnected production process. A reproducible import must identify the image by version and digest, the profile by hash, the model artifacts by checksum, and the target GPU topology by validated support data.

Build a Controlled Import Zone

The connected preparation system should be an enterprise service, not an engineer’s laptop that happens to have enough disk space.

It needs controlled outbound access, NGC credentials, access to any gated model source, sufficient storage for images and model artifacts, malware and vulnerability scanning, signature verification, and a transfer mechanism approved for the target security zone. It should also match the production CPU architecture and be able to exercise the same NIM packaging workflow.

Authenticate and Accept Terms Before the Change Window

NVIDIA requires NGC authentication to pull NIM images, using the special $oauthtoken username and an NGC API key. Governing terms for a specific NIM must also be accepted before the first download [1]. These actions belong in the connected preparation phase, not in the production rollout window.

A simple preparation sequence looks like this:

export NIM_IMAGE="nvcr.io/nim/meta/llama-3.1-8b-instruct:2.0.8"

printf '%s' "$NGC_API_KEY" | \
  docker login nvcr.io --username '$oauthtoken' --password-stdin

docker pull "$NIM_IMAGE"
docker inspect --format='{{index .RepoDigests 0}}' "$NIM_IMAGE"

The first command identifies the exact image tag. The login uses an API key without writing it into command history. The final command captures the immutable repository digest that should appear in the import manifest.

Create an Artifact Manifest

Every import batch should carry a machine-readable inventory. This is the bridge between security review, platform operations, and rollback.

release_id: nim-llama31-8b-2026-07
source_baseline:
  nim_llm_version: 2.0.8
  documentation_review_date: 2026-07-23
container:
  source_image: nvcr.io/nim/meta/llama-3.1-8b-instruct:2.0.8
  source_digest: sha256:REPLACE_WITH_CAPTURED_DIGEST
  internal_image: harbor.ai.corp/nim/llama-3.1-8b-instruct@sha256:REPLACE
model:
  deployment_mode: model-specific
  profile_hash: REPLACE_WITH_PROFILE_HASH
  artifact_checksum_manifest: checksums.sha256
platform:
  gpu_operator_bundle: REPLACE_WITH_APPROVED_VERSION
  nim_operator_bundle: REPLACE_WITH_APPROVED_VERSION
  driver_branch: REPLACE_WITH_SUPPORTED_BRANCH
controls:
  terms_accepted: true
  vulnerability_scan_id: SCAN-REPLACE
  import_approval_id: CHG-REPLACE
  rollback_release_id: nim-llama31-8b-previous

The values must be changed for the selected NIM, GPU, Kubernetes distribution, and internal registry. Successful execution means the manifest can be reconciled against the internal registry, model store, change record, and live workload.

Mirror Container Images into Harbor or an Equivalent Registry

Harbor is a reasonable fit because it provides projects, access controls, retention, vulnerability scanning, and content-trust capabilities. The same architecture principles apply to another enterprise OCI registry.

A production registry design should include:

Harbor’s current production guidance requires HTTPS and a resolvable FQDN, and its scanner can be operated with a manually supplied offline vulnerability database [14]. Content trust can add signature verification, but signature enforcement should be tested with the deployment toolchain before it becomes a production gate [15].

Mirror the Whole Dependency Graph

For Docker-only NIM, the primary OCI artifact may be one NIM image. For Kubernetes, the graph is larger:

A mirror that contains only the visible NIM service image is incomplete. The next cluster rebuild or operator upgrade will expose the missing dependencies.

Replace Public Pull Secrets with Internal Registry Credentials

NIM Operator examples use an NGC registry secret and a separate NGC API key secret for model downloads [9]. In a fully pre-staged disconnected runtime, those public credentials should not be required. Use an internal registry pull secret for images and local credentials only for an internal model store if the storage service requires them.

Keeping NGC or Hugging Face tokens in the disconnected namespace creates unnecessary secret-management risk and can hide an architectural defect. If the service starts only when those tokens are present, it is probably still trying to reach an external source.

Preload Model Artifacts and Runtime Profiles

Container-image mirroring and model-artifact staging are different workflows. NIM uses a manifest-driven model download process, and model assets are stored under NIM_CACHE_PATH, which defaults to /opt/nim/.cache and must be writable during download and cache construction [5].

Discover the Profiles on the Target GPU Class

Run profile discovery on hardware that represents the intended deployment. The selected profile encodes details such as parallelism, precision, and runtime compatibility.

export LOCAL_NIM_CACHE="/staging/nim-cache"
export PROFILE_HASH="REPLACE_WITH_APPROVED_PROFILE"

mkdir -p "$LOCAL_NIM_CACHE"

docker run --rm --gpus all \
  -e NGC_API_KEY \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  "$NIM_IMAGE" list-model-profiles

docker run --rm --gpus all \
  -e NGC_API_KEY \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  "$NIM_IMAGE" download-to-cache -p "$PROFILE_HASH"

The first container lists available profiles. The second downloads the approved profile. Success means the cache contains all referenced files and the profile can be started on the intended GPU topology before transfer.

For stronger portability, create a model store after the cache is populated:

export MODEL_STORE="/staging/model-store"
mkdir -p "$MODEL_STORE"

docker run --rm --gpus all \
  -e NGC_API_KEY \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -v "$MODEL_STORE:/model-store" \
  "$NIM_IMAGE" create-model-store \
    -p "$PROFILE_HASH" \
    -m /model-store

A model store produces an explicit local model directory. A transferred cache preserves the profile-oriented cache structure. Both can work, but the team should standardize one pattern and document the restart behavior.

Preserve Runtime Manifests for Model-Free NIM

If a model-free NIM first starts from a remote URI, retain the persistent cache containing the generated runtime manifest. Do not copy only the obvious weight files and assume the service will reconstruct the rest later. The restart test should prove that the manifest is found locally and that no authentication call occurs.

After a pre-staged workflow has been validated, NIM_DISABLE_MODEL_DOWNLOAD=true can provide an additional guard against startup downloads in the scenarios where that setting fits the deployment. It is not a substitute for a complete cache or model store [18].

A local-path model-free deployment avoids that dependency:

docker run --rm --gpus all \
  --name nim-airgap \
  --shm-size=16GB \
  -p 8000:8000 \
  -e NIM_MODEL_PATH=/model-repo \
  -e NIM_SERVED_MODEL_NAME=enterprise-llm \
  -v /srv/models/enterprise-llm:/model-repo:ro \
  harbor.ai.corp/nim/model-free-nim@sha256:REPLACE

The model directory is mounted read-only, and the container image is pinned by digest. If the selected release requires a writable cache for generated runtime data, provide a separate writable volume rather than making the governed model store mutable.

Separate the Image Registry from the Model Cache

The registry and model cache solve different problems and usually have different performance, retention, and security characteristics.

Design areaOCI image registryModel artifact cache or store
Primary contentContainer layers, manifests, signaturesWeights, tokenizer, profiles, runtime manifests
Typical accessLayer pull during scheduling or node replacementLarge sequential reads during NIM startup
Change unitImage version or digestModel version and profile
RetentionActive, rollback, security holdActive profiles, rollback models, import staging
IntegrityImage digest and signatureFile checksums, model manifest, provenance
Performance riskSlow image pull and node startupLong cold start and GPU starvation during load
Kubernetes patternRegistry pull secretPVC, ReadWriteMany storage, local model store, or internal object storage

NVIDIA recommends local model caching for NIM Operator deployments and network-backed storage for production so multiple pods can share the cache [8]. That does not mean any network filesystem is sufficient. A model cache is part of the serving path during startup and scale-out.

Size for Versions, Profiles, and Import Headroom

Use a capacity model rather than a single model-size estimate:

Required model capacity =
  Sum(selected profile artifact sizes)
  x retained model versions
  x required physical copies
  + import staging space
  + checksum and extraction workspace
  + rollback reserve
  + operational headroom

Measure the actual profile on disk after download. Quantization, tensor parallelism, tokenizer files, engine artifacts, and model-store formatting can make the required footprint differ from the headline parameter count.

Measure the Startup Path

Track at least:

A fast steady-state inference service can still have unacceptable recovery time if model loading from shared storage takes too long.

Close DNS, Proxy, Firewall, and Certificate Dependencies

Disconnected design fails most often at trust and name resolution boundaries.

DNS

The production zone needs resolvable internal names for the OCI registry, model store, Kubernetes API, storage endpoints, monitoring systems, identity services, and internal time sources. Do not depend on public DNS responses to resolve internal mirrors.

Test DNS after the public resolvers and default route are removed. A cached answer is not proof of a sustainable design.

Proxy and Firewall

Restricted-egress preparation systems may need HTTP_PROXY, HTTPS_PROXY, and a carefully scoped NO_PROXY. The no-proxy list should include internal registry, Kubernetes, storage, and service domains to prevent internal traffic from being sent through an outbound inspection layer.

Do not build a firewall rule set from one successful API call. Model downloads can involve authentication, catalog, API, and object-delivery endpoints. The connected staging test should capture actual destination names and flows for the exact NIM and model source, then compare them with the approved vendor documentation and enterprise proxy policy.

In a dark site or fully air-gapped runtime, public egress should be denied and treated as a control, not as a transient outage.

Private Certificate Authorities

A TLS-inspecting proxy or internal registry signed by a private CA requires trust at multiple layers:

NVIDIA’s current guidance requires a combined CA bundle when a proxy re-signs public HTTPS traffic. Replacing the default trust store with only the corporate CA can break trust for public endpoints during the connected phase [2], [5].

The two NVIDIA forum threads supplied with this assignment show why the visible error can be misleading. An apparent config.json download problem was associated with InvalidCertificate(UnknownIssuer) on an object-download path, and another on-premises case also surfaced certificate trust failures [16], [17]. The lesson is to trace the underlying TLS and network exception, not stop at the filename in the warning.

Make Kubernetes Offline Before Calling NIM Offline

A Kubernetes deployment adds dependencies that a single Docker host does not have. NVIDIA NIM Operator supports air-gapped patterns, but the operator still depends on preloaded or internally mirrored images and model assets [7].

The disconnected bill of materials should include:

The GPU Operator normally pulls both container images and operating-system packages. NVIDIA documents two viable offline patterns: mirror the OS packages into a local repository, or use a supported precompiled driver container that removes that download requirement [10].

Driver and CUDA Responsibilities

The NIM container carries the user-space libraries expected by that image. The host or Kubernetes nodes still need a compatible NVIDIA driver and container runtime integration. A full host CUDA toolkit is not automatically required merely to run a containerized NIM, but the driver, NVIDIA Container Toolkit, device exposure, and image compatibility must match the current support matrix.

GPU Operator can manage much of this stack, but it also expands the mirrored dependency graph. Pin the operator, driver branch, node operating system, kernel, container runtime, and NIM version as one tested platform baseline.

Image Pull and Model Pull Are Separate Secrets

In a connected NIM Operator deployment, an NGC registry secret pulls images and an API-key secret enables model downloads [9]. In a disconnected deployment:

A pod that reaches ImagePullBackOff has not reached NIM yet. A pod that starts the image but fails to find a profile has passed the registry layer and failed in the model layer. Keep those diagnostic domains separate.

Validate Licensing and Entitlement Before Import

Offline operation does not remove licensing, support, or governing-term obligations.

Some NIM microservices are freely available, while others require NVIDIA AI Enterprise. Production branches, long-term support branches, and enterprise support are tied to NVIDIA AI Enterprise offerings [11]. NVIDIA AI Enterprise is licensed per GPU for covered software, with specific inclusions and activation conditions for selected hardware [12].

The import workflow should therefore record:

Do not assume every NIM performs a live license-server check at container startup. Also do not assume that successful offline startup proves the organization is licensed for production use. Entitlement validation is an administrative control that must be completed before artifact promotion.

Prove Cold Start, Restart, and Recovery

A disconnected-readiness test must recreate the events that invalidate warm-cache assumptions.

Test from a Clean Node

Use a node that does not already contain the NIM image layers. Confirm that it pulls from the internal registry, mounts the internal model storage, selects the approved profile, and becomes ready while public egress is blocked.

This proves that the environment can recover after node replacement. Starting a pod on a node with cached image layers proves less.

Test a True Model Cold Start

Use a new NIM service instance that points to the approved internal model source. Capture:

Test Restarts After Outages

At minimum, test:

Failure eventExpected resultEvidence
NIM pod deletionReplacement pod becomes ready from internal sourcesPod events, image digest, readiness time
GPU node rebootDriver and device plugin recover, NIM reloadsGPU Operator state, device discovery, NIM health
Loss of public DNS and default routeService restarts normallyNetwork policy and firewall logs
Internal registry outage after image is cachedExisting pod remains up; reschedule behavior matches designNode cache evidence and documented limitation
Model storage interruptionStartup fails clearly, then recovers after storage restorationStorage events and NIM logs
Certificate rotationRegistry and model-store clients trust the new chainPull test, mount test, TLS verification
Upgrade rollbackPrevious image and model cache restore serviceChange record, rollback timing, health checks

A fully air-gapped service is still vulnerable to simultaneous loss of an internal registry and a clean-node replacement. Document that dependency instead of claiming the service is self-contained.

Collect a Restart Evidence Pack

The acceptance record should contain:

A useful service check can remain simple:

curl --fail --silent localhost:8000/v1/health/live
curl --fail --silent localhost:8000/v1/health/ready
curl --fail --silent localhost:8000/v1/models | jq .

Success means all three calls complete locally, the expected model identifier is returned, and the service logs show that the approved local profile or model path was loaded.

Design Upgrades as New Imports, Not In-Place Mutations

Disconnected systems need a deliberate release train.

The current NIM Operator upgrade model requires a new NIMCache for a newer model or profile version, followed by an update to the NIMService or NIMPipeline. Cached model profile versions are not updated in place, and multiple replicas are required to avoid availability impact during a rolling change [13].

A practical release process is:

The organization should define separate response targets for critical vulnerabilities, routine patches, model updates, certificate changes, and platform changes. A disconnected site that cannot import a critical fix within its risk deadline is not operationally ready, even if the first installation succeeded.

The vulnerability process must also update the scanner’s offline database. Otherwise, a registry may report that an image is clean only because the local scanner has stale intelligence.

Assign Operational Ownership

Disconnected AI introduces an import function that often does not exist in a connected platform team.

CapabilityAccountable ownerResponsible teamRequired evidence
NGC entitlement and termsAI platform ownerVendor management or AI platformAccepted terms, entitlement record
Connected artifact retrievalSoftware supply-chain ownerImport teamSource, time, digest, checksum
Vulnerability and malware reviewSecuritySecurity engineeringScan report and exception record
Registry promotionPlatform engineeringRegistry administratorsApproved digest and audit log
Model-store promotionAI platform ownerMLOps or model operationsProfile, checksum, provenance
GPU stack compatibilityInfrastructure ownerKubernetes and GPU platform teamDriver, operator, GPU validation
Disconnected deploymentService ownerPlatform operationsManifest, health, restart evidence
Patch and rollback decisionChange authorityJoint service teamRisk assessment and rollback plan

The key control is separation of duties. The person who downloads an artifact should not be the only person who approves it for production. The application team should not own private-CA distribution. The registry team should not decide whether a model’s terms permit the intended use.

Disconnected-Readiness Checklist

Operating Model and Scope

NIM and Model Selection

Container and Platform Artifacts

Model Artifacts

Registry, Storage, and Trust

Network and Secrets

Licensing, Lifecycle, and Operations

Validation

Troubleshooting Decision Tree

Use the first observable failure to identify the layer. Do not jump directly to GPU diagnostics when the image or model was never delivered.

Common Failure Patterns

config.json or generation_config.json Fetch Failures

A filename-oriented warning does not identify the root cause. Check the nested exception and destination category.

Likely causes include:

The two community threads demonstrate that UnknownIssuer can surface beneath a config.json download symptom [16], [17]. Validate the full certificate chain from inside the same container and through the same proxy path used by the download command.

Missing Model Profile

If NIM reports that no compatible profile is available:

Do not solve a missing profile by enabling public egress in production. Repair the import.

TLS Errors

For UnknownIssuer, hostname mismatch, or certificate-chain errors:

Registry Failures

For ImagePullBackOff, unauthorized pulls, or manifest errors:

Blocked Outbound Calls

A blocked outbound call during final testing is useful evidence. Identify what requested it.

The correct fix is to internalize or remove the dependency, not create an undocumented firewall exception.

Conclusion

NVIDIA NIM can operate disconnected, but only when the enterprise treats disconnection as a software supply-chain and operational-readiness problem.

The OCI image is one dependency. The model profile, model weights, runtime manifest, GPU stack, Kubernetes operators, storage, certificate chain, license evidence, and update process are equally important. A mirror that handles only containers creates a deployment that appears successful during a warm start and fails during the event that matters most: a cold restart after an outage.

The strongest design uses a controlled import zone, immutable artifacts, separate image and model stores, explicit ownership, and a repeatable validation pack. Production acceptance should require a clean-node pull, a true model cold start, pod and node recovery, blocked-egress evidence, and a tested rollback.

That is the standard that turns “the container started once” into a defensible air-gapped private AI service.

External References

Exit mobile version