How to Deploy NVIDIA NIM Microservices on Kubernetes with the NIM Operator

TL;DR

NVIDIA NIM can be deployed on Kubernetes through Helm or managed declaratively through the NVIDIA NIM Operator. The operator-based path is the better fit when you want Kubernetes-native lifecycle management for model caching, GPU scheduling, health probes, service exposure, scaling, and upgrades.

The practical sequence is straightforward, but the dependencies matter. Build a supported Kubernetes and GPU foundation, install the GPU Operator, install NIM Operator 3.1.1, create NGC secrets in the workload namespace, populate a persistent NIMCache, and then deploy a NIMService that requests the correct GPU resources and mounts the cache.

The most important production distinction is that NGC credentials authorize image and model downloads. They do not automatically protect the inference endpoint. Client authentication, TLS, rate limiting, and authorization should be enforced at an ingress controller, Kubernetes Gateway API implementation, service mesh, or enterprise API gateway.

Introduction

Running an inference container is easy. Operating a reliable inference service on Kubernetes is not.

The difficult parts are usually outside the container itself: installing and validating GPU drivers, exposing GPU resources to the scheduler, downloading large model artifacts, preserving those artifacts across restarts, matching a model profile to the available GPU, waiting long enough for model initialization, protecting the endpoint, and upgrading without turning every model change into a long outage.

The NVIDIA NIM Operator addresses that operational layer. It extends Kubernetes with custom resources for model caching and NIM service lifecycle management. Instead of manually maintaining a Deployment, Service, persistent volume, health probes, and model-download job, the platform team declares the desired NIM state and allows the operator to reconcile the supporting resources.

This tutorial uses the Automation Tutorial content family because the goal is not simply to explain NIM architecture. The goal is to leave you with an end-to-end implementation path, validation commands, production controls, rollback boundaries, and troubleshooting guidance.

What You Will Accomplish

By the end of this tutorial, you will have:

validated a Kubernetes cluster with schedulable NVIDIA GPU resources
installed the NVIDIA GPU Operator and NVIDIA NIM Operator with pinned versions
created NGC credentials for container and model access
created a persistent model cache with a NIMCache custom resource
deployed a GPU-backed NIMService
tested the first inference endpoint through local port forwarding
understood how to expose the endpoint through Ingress or Gateway API
configured a practical path for health checks, logs, metrics, and horizontal scaling
established upgrade and rollback procedures that preserve the previous model cache

The example uses an NVIDIA-documented LLM-specific NIM image and a single-GPU deployment. Confirm that the selected image tag, model profile, GPU type, memory capacity, and NVIDIA AI Enterprise entitlement are valid for your environment before applying the manifests.

How the NIM Operator Fits into the Request Flow

The application does not send inference traffic through the NIM Operator. The operator is a control-plane component. It watches NIMCache and NIMService resources, then creates and maintains the Kubernetes objects that serve requests.

The runtime request path goes through the Kubernetes Service or external gateway directly to the NIM pod and GPU.

The separation matters operationally. If the operator restarts, an already running NIM pod can continue serving traffic. If the NIM pod fails, Kubernetes and the operator reconcile the declared service state. If the model cache is missing or incompatible, the serving pod may start but never become ready.

Prerequisites and Support Boundaries

The commands in this article use a version baseline validated on July 22, 2026. NVIDIA NIM, NIM Operator, GPU Operator, model images, and supported Kubernetes releases are version-sensitive. Recheck the NVIDIA support matrix and release notes before production deployment.

Requirement	Practical baseline	Why it matters
Kubernetes	Version 1.26 or later for the documented NIM LLM operator path	Older API behavior and unsupported platforms can block deployment or supportability
Administrative access	Cluster-admin for operator installation	The operators install CRDs, controllers, RBAC, and node-level GPU components
Helm	Helm 3	Used to install and upgrade both operators
NVIDIA GPU nodes	GPU type and count supported by the selected NIM model profile	A pod can request a GPU and still fail if the model does not fit or the profile is incompatible
GPU Operator	Version 24.3.0 or later, with the article pin set to v26.3.3	Provides drivers, container runtime integration, device plugin behavior, runtime classes, validation, and node labels
NIM Operator	Version 3.1.1	Current documented operator release at the research baseline
NGC access	NVIDIA AI Enterprise subscription or eligible NVIDIA Developer Program access, plus an NGC API key	NIM images and model artifacts are access controlled
Persistent storage	A working storage class sized for the selected model profiles	Model caches can be tens or hundreds of gigabytes
Network access	Outbound access to NVIDIA registries and model services, or a validated air-gapped mirror	Cache jobs and image pulls fail without the required egress path
Endpoint control	Ingress, Gateway API, service mesh, or API gateway for production exposure	Kubernetes Service exposure alone does not provide enterprise authentication and authorization

Confirm the GPU and Model Fit Before Installation

Do not choose a NIM image solely because the model name looks appropriate. Confirm:

supported GPU families
required GPU count
minimum GPU memory
supported inference engine
expected model cache size
whether the model uses an optimized profile, a community engine, or a buildable profile
whether your NGC organization is entitled to the image and model artifacts

For a first deployment, start with one model, one replica, one supported GPU, and persistent storage. Prove the complete path before adding autoscaling, multiple GPU types, multi-node inference, or shared platform tenancy.

Prepare the Deployment Variables

The NVIDIA Helm repository address is intentionally stored in an environment variable so the article body remains free of publication links. Copy the repository endpoint from the NVIDIA NIM Operator installation guide listed in the External References section.

export NVIDIA_HELM_REPOSITORY="<NVIDIA_NGC_HELM_REPOSITORY>"
export GPU_OPERATOR_VERSION="v26.3.3"
export NIM_OPERATOR_VERSION="3.1.1"
export NIM_NAMESPACE="nim-service"
export NIM_NAME="meta-llama-3-2-1b-instruct"

Change the storage class, NIM image, and NIM tag in the manifests before deployment. The image and tag shown in this tutorial come from NVIDIA’s current NIM Operator examples, but they should be treated as a documented implementation example, not a guarantee that the image is entitled or appropriate for every GPU.

Validate the Kubernetes and GPU Foundation

Start by confirming that the cluster, Helm client, and GPU nodes are visible.

kubectl version
helm version
kubectl get nodes -o wide
kubectl get storageclass

If the cluster already has the GPU Operator, inspect the existing installation before creating another release.

helm list -A | grep -i gpu-operator || true
kubectl get pods -A | grep -E 'nvidia|gpu-operator' || true
kubectl get nodes -L nvidia.com/gpu.product,nvidia.com/gpu.count

The nvidia.com/gpu extended resource must appear as allocatable on at least one worker node. This command displays allocatable GPU counts without requiring a test workload:

kubectl get nodes \
  -o custom-columns='NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu,GPU_PRODUCT:.metadata.labels.nvidia\.com/gpu\.product'

A blank GPU column means the Kubernetes scheduler does not currently see an allocatable NVIDIA GPU on that node. Fix the GPU foundation before installing NIM.

Install the NVIDIA GPU Operator

Add the NVIDIA Helm repository and install the GPU Operator with a pinned version.

helm repo add nvidia "${NVIDIA_HELM_REPOSITORY}"
helm repo update

helm upgrade --install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --version "${GPU_OPERATOR_VERSION}" \
  --wait

This default path is appropriate when the GPU Operator should manage the driver and container toolkit. Managed Kubernetes services, DGX systems, preinstalled drivers, vGPU environments, Secure Boot systems, and cloud-specific node images can require different Helm values. Do not apply the default command blindly to a cluster where drivers or the NVIDIA Container Toolkit are already managed by the platform.

Validate the GPU Operator

kubectl get pods -n gpu-operator
kubectl get pods -n gpu-operator -l app=nvidia-cuda-validator
kubectl get runtimeclass
kubectl get nodes -L nvidia.com/gpu.product,nvidia.com/gpu.count

Expected results include:

the GPU Operator controller is running
driver, toolkit, device plugin, and validation components are healthy where applicable
the CUDA validator completes successfully
an NVIDIA runtime class is present when required by the platform
GPU nodes have NVIDIA feature and product labels
nvidia.com/gpu appears in node allocatable resources

Do not proceed while the CUDA validator is failing or GPU resources are missing from node capacity.

Install the NVIDIA NIM Operator

Create the operator namespace and install the pinned NIM Operator release.

kubectl create namespace nim-operator --dry-run=client -o yaml | kubectl apply -f -

helm upgrade --install nim-operator nvidia/k8s-nim-operator \
  --namespace nim-operator \
  --version "${NIM_OPERATOR_VERSION}" \
  --wait

Validate the controller and CRDs.

kubectl get pods -n nim-operator
kubectl get crd | grep apps.nvidia.com
kubectl api-resources | grep -E 'NIMCache|NIMService|NIMPipeline'

At minimum, the cluster should recognize the NIMCache and NIMService resource types before you continue.

Configure NGC Authentication and Image Access

Create a dedicated namespace for the NIM workload.

kubectl create namespace "${NIM_NAMESPACE}" --dry-run=client -o yaml | kubectl apply -f -

Export the NGC API key in your shell without writing it into a manifest or source repository.

read -s -p "NGC API key: " NGC_API_KEY
export NGC_API_KEY
printf '\n'

Create two secrets in the workload namespace:

ngc-secret is a Docker registry secret used to pull NIM container images
ngc-api-secret is a generic secret used by model puller containers to download model artifacts

kubectl create secret docker-registry ngc-secret \
  --namespace "${NIM_NAMESPACE}" \
  --docker-server=nvcr.io \
  --docker-username='$oauthtoken' \
  --docker-password="${NGC_API_KEY}" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic ngc-api-secret \
  --namespace "${NIM_NAMESPACE}" \
  --from-literal=NGC_API_KEY="${NGC_API_KEY}" \
  --dry-run=client -o yaml | kubectl apply -f -

unset NGC_API_KEY

Validate only metadata. Do not print or decode the secret value in routine troubleshooting output.

kubectl get secret -n "${NIM_NAMESPACE}" ngc-secret ngc-api-secret

NGC Authentication Is Not Endpoint Authentication

The two secrets above authenticate image and model downloads. The authSecret field in a NIMService points to the secret containing NGC_API_KEY. It should not be interpreted as client authentication for application requests.

For production, protect the inference endpoint with controls such as:

TLS termination
OIDC or JWT validation
mTLS between trusted workloads
API keys issued and validated by an API gateway
namespace and network policy restrictions
rate limiting and request-size controls
audit logging tied to application or workload identity

Keep the NIM Service as ClusterIP unless there is a specific reason to expose it directly.

Design the Model Cache Before Deploying the Service

Model caching is not a cosmetic optimization. It controls startup time, network demand, repeatability, scaling behavior, and rollback safety.

The NIMCache resource starts a Kubernetes Job that downloads compatible model profiles to persistent storage. A later NIMService mounts that cache instead of downloading the complete model every time a serving pod starts.

Choose the Access Mode Deliberately

For a single-replica lab, ReadWriteOnce is often sufficient. For production scaling across multiple GPU nodes, evaluate whether the storage system and access mode allow every replica to mount the cache when scheduled on different nodes.

Use these rules:

use persistent storage for production
avoid emptyDir for production model persistence because data disappears with the pod
use hostPath only for tightly controlled single-node or proof-of-concept deployments
size storage for the filtered model profiles plus upgrade headroom
use profile filtering so the cache job does not download unnecessary engines or GPU variants
keep the old cache until the upgraded service is fully validated
understand the deletion behavior before allowing the operator to create and own the PVC

When spec.storage.pvc.create is true, deleting the NIMCache can delete the operator-created PVC and cached models. That behavior is convenient for cleanup and dangerous during a rushed rollback.

Create the Persistent NIM Cache

Create a file named nim-cache.yaml.

apiVersion: apps.nvidia.com/v1alpha1
kind: NIMCache
metadata:
  name: meta-llama-3-2-1b-instruct
  namespace: nim-service
spec:
  source:
    ngc:
      modelPuller: nvcr.io/nim/meta/llama-3.2-1b-instruct:1.12.0
      pullSecret: ngc-secret
      authSecret: ngc-api-secret
      model:
        engine: tensorrt_llm
        tensorParallelism: "1"
  storage:
    pvc:
      create: true
      storageClass: "<YOUR_STORAGE_CLASS>"
      size: "80Gi"
      volumeAccessMode: ReadWriteOnce

Change the storage class and validate that the engine and tensor parallelism match a profile supported by your GPU. If the model supports several engines and you do not filter correctly, the cache can consume significantly more storage than expected.

Apply the cache resource.

kubectl apply -f nim-cache.yaml
kubectl get nimcache -n "${NIM_NAMESPACE}" -w

The cache must reach Ready before the serving workload is deployed.

Inspect the supporting PVC, Job, events, and logs when progress stalls.

kubectl get pvc -n "${NIM_NAMESPACE}"
kubectl get jobs -n "${NIM_NAMESPACE}"
kubectl describe nimcache -n "${NIM_NAMESPACE}" "${NIM_NAME}"
kubectl logs -n "${NIM_NAMESPACE}" -l job-name="${NIM_NAME}-job" --tail=200

A successful cache stage proves four things at once: NGC authentication works, the image is entitled, the cluster can reach the required registries, and persistent storage can hold the selected model profile.

Deploy the First NIM Endpoint

Create a file named nim-service.yaml.

apiVersion: apps.nvidia.com/v1alpha1
kind: NIMService
metadata:
  name: meta-llama-3-2-1b-instruct
  namespace: nim-service
  labels:
    app.kubernetes.io/name: meta-llama-3-2-1b-instruct
    app.kubernetes.io/part-of: nvidia-nim
spec:
  labels:
    app.kubernetes.io/name: meta-llama-3-2-1b-instruct
    app.kubernetes.io/part-of: nvidia-nim
  image:
    repository: nvcr.io/nim/meta/llama-3.2-1b-instruct
    tag: "1.12.0"
    pullPolicy: IfNotPresent
    pullSecrets:
      - ngc-secret
  authSecret: ngc-api-secret
  storage:
    nimCache:
      name: meta-llama-3-2-1b-instruct
      profile: ""
  replicas: 1
  resources:
    limits:
      nvidia.com/gpu: 1
  expose:
    service:
      type: ClusterIP
      port: 8000

The blank profile allows NIM to select a compatible cached profile. Pin a specific profile only after you have inspected the cached profile list and validated it against the target GPU.

Apply the service.

kubectl apply -f nim-service.yaml
kubectl get nimservice -n "${NIM_NAMESPACE}" -w

Inspect the generated Kubernetes objects.

kubectl get deployment,pod,service -n "${NIM_NAMESPACE}"
kubectl describe nimservice -n "${NIM_NAMESPACE}" "${NIM_NAME}"
kubectl get events -n "${NIM_NAMESPACE}" --sort-by=.lastTimestamp | tail -n 30

A large model can take several minutes to initialize even when the cache is ready. Do not treat a temporarily failing readiness check as a container crash without checking startup progress and logs.

Test the NIM Endpoint

Use port forwarding for the first validation. This keeps the service internal and removes ingress, DNS, certificates, and gateway policies from the initial test.

kubectl port-forward -n "${NIM_NAMESPACE}" \
  service/"${NIM_NAME}" 8000:8000

In another terminal, check the health endpoints.

curl -s localhost:8000/v1/health/live
curl -s localhost:8000/v1/health/ready

Discover the model name exposed by the service.

curl -s localhost:8000/v1/models | jq

Send a small chat completion request using the model identifier returned by the models endpoint.

curl -s localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "meta/llama-3.2-1b-instruct",
    "messages": [
      {
        "role": "user",
        "content": "Explain why persistent model caching matters in Kubernetes."
      }
    ],
    "max_tokens": 128
  }' | jq

Successful execution should produce a valid JSON response and a generated assistant message. A successful liveness check with a failing readiness check usually means the container is running but the model has not finished loading.

Expose the Service Without Bypassing Security

The safest default is ClusterIP. Add an external path only after the internal service is healthy.

Exposure method	Best fit	Primary caveat
Port forwarding	Administrator validation and troubleshooting	Not an application access pattern
ClusterIP	Internal applications in the same cluster or connected service network	Requires internal identity and network policy design
Ingress	Existing HTTP ingress standard with TLS and authentication integrations	Controller-specific annotations and behavior must be governed
Gateway API	Platform teams standardizing routes, gateways, policy attachment, and cross-namespace ownership	Requires a Gateway API implementation and pre-created Gateway
LoadBalancer	Simple external connectivity in a controlled environment	Can create a direct entry point that bypasses shared gateway controls

Example Ingress-Based Exposure

The NIM Operator can generate an Ingress resource through the spec.expose.router.ingress field. Add the following structure to the NIMService specification after your ingress controller, DNS, and TLS secret are ready.

spec:
  expose:
    service:
      type: ClusterIP
      port: 8000
    router:
      ingress:
        ingressClass: nginx
        tlsSecretName: nim-api-tls
      hostDomainName: ai.example.com

The generated hostname follows the operator naming pattern using the NIM service name, namespace, and host domain.

Authentication is still a gateway responsibility. Configure the ingress controller or upstream API gateway to validate client identity before traffic reaches the NIM service. Do not place a public LoadBalancer in front of the service and assume the ngc-api-secret protects it.

Restrict East-West Access with NetworkPolicy

This example permits traffic only from namespaces explicitly labeled for NIM access.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-approved-nim-clients
  namespace: nim-service
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/part-of: nvidia-nim
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              access.nim/enabled: "true"
      ports:
        - protocol: TCP
          port: 8000

Label only approved client namespaces.

kubectl label namespace <CLIENT_NAMESPACE> access.nim/enabled=true

NetworkPolicy support depends on the cluster network plugin. Validate enforcement rather than assuming the policy is active.

Understand the Default Health Checks

The NIM Operator configures three health behaviors by default:

Probe	Default endpoint	Operational meaning
Liveness	`/v1/health/live`	Confirms the container and proxy are alive; repeated failure restarts the pod
Readiness	`/v1/health/ready`	Controls whether the service sends traffic to the pod
Startup	`/v1/health/ready`	Protects slow model initialization from premature liveness and readiness actions

The documented startup probe permits up to approximately 20 minutes by default through a 10-second period and failure threshold of 120. Large models, slow storage, engine construction, or cold caches can require a longer window.

Do not disable probes simply to make a deployment look healthy. Increase the startup window only when logs show legitimate model initialization that needs more time.

spec:
  startupProbe:
    enabled: true
    probe:
      httpGet:
        path: /v1/health/ready
        port: 8000
      initialDelaySeconds: 30
      periodSeconds: 10
      failureThreshold: 240

A forty-minute startup allowance can hide genuine failures if it is applied without monitoring. Pair longer startup thresholds with cache validation, log inspection, and deployment time objectives.

Configure Logging and Operational Visibility

Start with Kubernetes-native inspection.

kubectl logs -n "${NIM_NAMESPACE}" \
  -l app="${NIM_NAME}" \
  --all-containers=true \
  --tail=200

kubectl logs -n nim-operator \
  -l app.kubernetes.io/name=k8s-nim-operator \
  --tail=200

The operator-managed NIM LLM path uses JSONL logging and an INFO log level by default. Increase verbosity temporarily when diagnosing a deployment, then return to the normal level to avoid excessive volume and sensitive request context.

spec:
  env:
    - name: NIM_LOG_LEVEL
      value: DEBUG

Production logging should separate:

operator reconciliation logs
model cache Job logs
NIM application logs
ingress or gateway access logs
GPU telemetry
Kubernetes events
authentication and authorization decisions

Do not rely on pod logs alone for a service-level view. Correlate requests, queue depth, latency, errors, GPU utilization, pod readiness, cache state, and gateway identity.

Enable Metrics and Horizontal Scaling

NIM LLM exposes inference metrics, and the operator can create a Prometheus ServiceMonitor. The cluster must already have the Prometheus Operator CRDs and a matching discovery label.

spec:
  metrics:
    enabled: true
    serviceMonitor:
      additionalLabels:
        release: kube-prometheus-stack
      interval: 30s
      scrapeTimeout: 10s

Horizontal scaling should use inference-specific pressure rather than generic CPU utilization. GPU inference pods can have high GPU saturation while CPU remains moderate, or queue requests while memory and CPU metrics appear acceptable.

The following example scales on a custom metric representing waiting requests.

spec:
  scale:
    enabled: true
    hpa:
      minReplicas: 1
      maxReplicas: 4
      metrics:
        - type: Pods
          pods:
            metric:
              name: vllm:num_requests_waiting
            target:
              type: AverageValue
              averageValue: "10"

When spec.scale.enabled is true, remove spec.replicas from the NIMService. The HPA owns the replica count.

This configuration also requires a Prometheus Adapter or equivalent custom metrics bridge. Verify that the metric appears in the Kubernetes custom metrics API before enabling production autoscaling.

Scaling Is Also a Capacity and Storage Decision

Before increasing maxReplicas, confirm:

enough schedulable GPUs exist for the maximum replica count
the model cache can be mounted by replicas across the intended nodes
the gateway can distribute requests correctly
startup time is compatible with the expected traffic ramp
the cluster autoscaler can add compatible GPU nodes when required
GPU quotas and priority classes prevent lower-value workloads from consuming serving capacity
the model license and NVIDIA AI Enterprise subscription cover the deployment

An HPA that requests four replicas does not create four GPUs. It only creates pending pods when capacity is unavailable.

Upgrade the NIM Operator Safely

Upgrade the operator separately from the model-serving workload. The operator can be upgraded while NIM custom resources are running, but CRD handling requires attention because Helm does not automatically upgrade existing CRDs in the normal chart lifecycle.

Export the current values and manifests before changing the operator.

helm get values nim-operator -n nim-operator -o yaml > nim-operator-values-before.yaml
kubectl get nimcache,nimservice -A -o yaml > nim-resources-before.yaml

Update the chart repository and inspect the target values.

helm repo update nvidia
helm show values nvidia/k8s-nim-operator \
  --version "${NIM_OPERATOR_VERSION}" \
  > nim-operator-values-target.yaml

Perform the upgrade with your reviewed values file.

helm upgrade nim-operator nvidia/k8s-nim-operator \
  --namespace nim-operator \
  --version "${NIM_OPERATOR_VERSION}" \
  -f nim-operator-values-target.yaml

Validate the controller, CRDs, and existing custom resources after the upgrade.

kubectl rollout status deployment -n nim-operator --timeout=5m
kubectl get pods -n nim-operator
kubectl get nimcache,nimservice -A

If the chart upgrade fails, inspect the pre-upgrade hook and image pull events before retrying. A failed hook can leave the Helm release in a failed state even while existing NIM services continue running.

Upgrade a NIM Model Without Destroying the Rollback Point

A model image upgrade is not only a Deployment image change. New NIM versions can introduce different model profiles, cache artifacts, engine behavior, and storage requirements.

Use this sequence:

Create a new NIMCache with a new name and the target NIM image tag.
Wait for the new cache to reach Ready.
Export the current NIMService manifest.
Update the NIM image tag and cache reference together.
Keep at least two replicas when availability during a rolling update is required and supported by available GPU capacity.
Validate health, model discovery, inference output, logs, latency, and error rate.
Retain the previous cache and manifest until the rollback window closes.

A practical naming pattern is:

meta-llama-3-2-1b-instruct-cache-1-12-0
meta-llama-3-2-1b-instruct-cache-<NEXT_VERSION>

The serving resource can retain a stable service name while the cache name changes between releases.

Roll Back Predictably

Rollback should restore a known combination of image tag, cache, environment variables, probe settings, and routing configuration.

NIM Workload Rollback

kubectl apply -f nim-service-previous.yaml
kubectl get nimservice -n "${NIM_NAMESPACE}" -w
kubectl rollout status deployment -n "${NIM_NAMESPACE}" --timeout=30m

Validate the old endpoint before deleting the failed cache.

NIM Operator Rollback

List Helm revisions.

helm history nim-operator -n nim-operator

Roll back to a known release revision.

helm rollback nim-operator <REVISION> -n nim-operator --wait

A Helm rollback does not guarantee that CRDs are restored to an earlier schema. Review CRD compatibility before downgrading the operator. CRD changes are one reason operator rollback should be tested in a non-production cluster.

Common Deployment Failures

Symptom	Likely cause	What to check
`ImagePullBackOff`	Invalid NGC key, missing entitlement, wrong image tag, registry egress blocked	Pod events, secret namespace, image name, NGC organization access, proxy and firewall rules
`NIMCache` remains `Pending`	PVC cannot bind or cache Job cannot schedule	StorageClass, PVC events, quota, node selector, access mode
`NIMCache` becomes `NotReady`	Cache Job failed, disk full, profile filter mismatch, authentication or network failure	Cache Job logs, NIMCache status conditions, PVC capacity, NGC secrets
NIM pod remains `Pending`	No compatible GPU is allocatable, taints are not tolerated, node selector is wrong, RWO volume cannot attach	Pod events, GPU node labels, quotas, taints, storage topology
NIM pod enters `CrashLoopBackOff`	Incompatible model profile, driver or runtime issue, missing cache content, GPU memory exhaustion	Pod logs, GPU Operator validation, cache readiness, model support matrix
Liveness passes but readiness fails	Model is still loading or initialization failed	Startup logs, readiness endpoint, startup probe window, cache mount
Endpoint works through port forwarding but not externally	Ingress or Gateway route, DNS, TLS, NetworkPolicy, or authentication configuration is wrong	Generated route, gateway listener, certificate, service port, policy logs
HPA shows unknown metrics	ServiceMonitor not discovered or custom metrics adapter is missing	Prometheus targets, ServiceMonitor labels, custom metrics API, metric name
NIM resources are not reconciled	Operator pod, RBAC, webhook, or CRD issue	Operator logs, CRDs, events, Helm release status
New replica cannot mount the cache	Storage access mode or topology does not support multi-node mounts	PVC access mode, CSI driver behavior, pod node placement, RWX availability

Fast Diagnostic Command Set

kubectl get nimcache,nimservice -n "${NIM_NAMESPACE}"
kubectl get pod,pvc,job,service -n "${NIM_NAMESPACE}"
kubectl get events -n "${NIM_NAMESPACE}" --sort-by=.lastTimestamp | tail -n 50
kubectl describe nimcache -n "${NIM_NAMESPACE}" "${NIM_NAME}"
kubectl describe nimservice -n "${NIM_NAMESPACE}" "${NIM_NAME}"
kubectl logs -n "${NIM_NAMESPACE}" -l job-name="${NIM_NAME}-job" --tail=200
kubectl logs -n "${NIM_NAMESPACE}" -l app="${NIM_NAME}" --all-containers=true --tail=200
kubectl logs -n nim-operator -l app.kubernetes.io/name=k8s-nim-operator --tail=200

The order matters. Start with custom resource status, then inspect the Kubernetes objects created by the operator. Avoid changing three layers at once because that destroys the evidence needed to identify the original failure.

Production Readiness Checklist

Before directing production traffic to the endpoint, confirm:

the model image, tag, profile, GPU type, and GPU count are documented
the GPU Operator and NIM Operator versions are pinned
operator values and custom resources are stored in version control
NGC secrets are managed through the organization’s secret-management process
Kubernetes secrets at rest are protected according to platform policy
the model cache is persistent, sized, monitored, and backed by a defined lifecycle policy
the external route terminates TLS and validates client identity
NetworkPolicy or equivalent segmentation limits east-west access
request and token limits are enforced at the appropriate layer
startup, readiness, and liveness behavior are tested under cold-start conditions
logs, metrics, GPU telemetry, gateway identity, and Kubernetes events are centralized
HPA metrics are visible before autoscaling is enabled
maximum replicas align with available GPU capacity and storage access mode
the previous model cache and manifest remain available during the rollback window
an owner exists for platform operations, model lifecycle, endpoint security, and application integration

Conclusion

The NIM Operator turns NVIDIA NIM deployment into a Kubernetes-native operating model rather than a collection of manually maintained resources.

The clean implementation pattern is to establish the GPU foundation first, install the operator control plane second, populate persistent model storage third, and deploy the inference service only after the cache is ready. That sequence makes failures easier to isolate because each stage proves a specific dependency before the next layer is introduced.

The most important production decisions sit around the NIM pod, not inside it. Storage topology determines whether replicas can scale across nodes. Gateway design determines whether the endpoint is authenticated and governed. Probe settings determine whether slow model initialization is tolerated without hiding genuine failures. Upgrade discipline determines whether a model change is reversible.

For a first deployment, keep the design intentionally small: one documented NIM image, one compatible GPU, one persistent cache, one ClusterIP service, and one controlled test client. Once that path is repeatable, add external routing, observability, autoscaling, higher availability, and multi-model operations as deliberate platform capabilities.

External References

NVIDIA: Installing NVIDIA NIM Operator
Canonical URL: https://docs.nvidia.com/nim-operator/latest/install.html
NVIDIA: Release Notes
Canonical URL: https://docs.nvidia.com/nim-operator/latest/release-notes.html
NVIDIA: Platform Support
Canonical URL: https://docs.nvidia.com/nim-operator/latest/platform-support.html
NVIDIA: Image Pull Secrets
Canonical URL: https://docs.nvidia.com/nim-operator/latest/image-pull-secrets.html
NVIDIA: Caching LLM NIM
Canonical URL: https://docs.nvidia.com/nim-operator/latest/cache-llm.html
NVIDIA: Managing NIM Services
Canonical URL: https://docs.nvidia.com/nim-operator/latest/service.html
NVIDIA: Upgrading NVIDIA NIM Operator
Canonical URL: https://docs.nvidia.com/nim-operator/latest/upgrade.html
NVIDIA: Upgrade NVIDIA NIM Microservices
Canonical URL: https://docs.nvidia.com/nim-operator/latest/nim-upgrade.html
NVIDIA: NIM Operator Deployment
Canonical URL: https://docs.nvidia.com/nim/large-language-models/2.0.3/deployment/kubernetes-deployment/nim-operator-deployment.html
NVIDIA: Installing the NVIDIA GPU Operator
Canonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html

Keep exploring

Choose your next step

Continue with the path that best matches the architecture or operating challenge in front of you.

AI strategy & delivery Enterprise AI Strategy, governance, AI platforms, data, and accelerated infrastructure. Explore Enterprise AI Architecture & integration Hybrid Platforms Architectures that connect VCF, Azure, public cloud, Kubernetes, and edge. Explore Hybrid Platforms Day-2 execution Operations & Resilience Security, recovery, lifecycle, observability, capacity, and FinOps. Explore Operations

New articles

Get practical guidance by email

Receive new enterprise AI and hybrid platform articles when they are published.

The Agent Blast Radius Model: Matching AI Agent Autonomy to Access, Risk, and Reversibility

Role-based access control tells you whether an identity can access a system. That still matters. It is no longer enough. When AI...

How to Run Coding Agents Safely Inside CI/CD Pipelines

TL;DR A coding agent should enter a CI/CD pipeline as an untrusted change producer, not as a privileged developer account. Give it an ephemeral sandbox, read-only repository access, a writable…