How to Deploy the NVIDIA RAG Blueprint on Kubernetes with Helm

by

TL;DR

The NVIDIA RAG Blueprint is not one application pod. It is a coordinated retrieval platform that combines an ingestion service, a RAG server, NVIDIA NIM microservices, NV-Ingest, a vector database, object storage, model caches, and supporting Kubernetes operators.

For the current 2.6.0 release, Elasticsearch is the default vector database and SeaweedFS is the default object store. Milvus remains a supported alternative, but changing backends requires document reingestion because the Blueprint does not migrate existing vector data automatically.

A production-minded deployment should begin with supported GPU and Kubernetes checks, explicit storage classes, pre-created secrets, pinned operator and chart versions, and a small Helm override file. Validate ingestion and retrieval separately, add hybrid search and reranking only after the baseline works, and treat MIG as a capacity design decision rather than a simple Helm switch.

Introduction

Retrieval-augmented generation often looks simple in an architecture slide: ingest documents, create embeddings, retrieve context, and send that context to a language model. The operational reality is more involved. Each stage has different GPU, storage, scaling, persistence, security, and failure characteristics.

The NVIDIA RAG Blueprint packages those stages into an opinionated reference implementation that can run on Kubernetes. Helm makes the deployment repeatable, while the NVIDIA GPU Operator and NIM Operator handle much of the accelerator and model-serving lifecycle. That does not remove the need for design decisions. It makes those decisions visible in values files, storage classes, resource requests, secrets, vector database choices, and node placement.

This tutorial uses NVIDIA RAG Blueprint 2.6.0 as the implementation baseline. It assumes self-hosted NIM microservices on a standard Kubernetes cluster, Elasticsearch as the starting vector database, and a dedicated rag namespace. The commands are designed as a practical deployment pattern, but operators should still validate every version, model profile, and GPU combination against their own supported platform before production rollout.

What You Will Build

By the end of this walkthrough, you will have a Kubernetes deployment with:

  • NVIDIA GPU Operator providing the driver, container runtime integration, device plugin, GPU discovery, and optional MIG management.
  • NVIDIA NIM Operator managing NIM caches and NIM services required by the Blueprint.
  • The NVIDIA RAG Blueprint installed through a pinned Helm chart.
  • A RAG server for query orchestration and answer generation.
  • An ingestor server for document upload, parsing, metadata handling, chunking, and indexing.
  • Elasticsearch as the default vector backend, with a documented path to Milvus.
  • Persistent storage for model caches, vector data, object data, and ingestion state.
  • Optional hybrid dense and sparse retrieval followed by reranking.
  • A validation workflow that separates ingestion health from query health.
  • An operational baseline for evaluation, tracing, metrics, troubleshooting, rollback, and MIG.

Reference Architecture

The most important architectural distinction is between the document ingestion path and the user query path. They share models and data stores, but they have different triggers, bottlenecks, scaling patterns, and recovery procedures.

The ingestion path is asynchronous and storage-heavy. It parses source files, extracts text and structured elements, generates embeddings, writes object data, and updates the vector index. The query path is latency-sensitive. It embeds a question, retrieves candidate chunks, optionally reranks them, and sends selected context to the generation model.

What matters operationally is that a healthy RAG server does not prove that ingestion is healthy, and a completed upload does not prove that retrieval quality is acceptable. Treat the two paths as separate services with separate tests, dashboards, scaling policies, and incident runbooks.

Prerequisites and Design Assumptions

Supported Platform Baseline

NVIDIA documents a specific Kubernetes deployment baseline for the Blueprint. At publication time, the Helm guide identifies Kubernetes 1.34.2 on Ubuntu 22.04 or 24.04 and Helm 3. The broader support matrix also calls for a GPU driver at version 560 or later and CUDA 12.9 or later.

Do not interpret those values as a reason to upgrade a production cluster in place. First confirm that your Kubernetes distribution, container runtime, GPU Operator version, CNI, CSI driver, and chosen NVIDIA driver are supported together. Managed Kubernetes services can also require provider-specific GPU Operator settings, especially when drivers or the NVIDIA Container Toolkit are preinstalled.

GPU Capacity

The default Kubernetes deployment is substantial. The current support matrix lists these default options:

Deployment profileDocumented accelerator footprint
Full GPU deployment8 x H100 80GB
Full GPU deployment8 x B200
Full GPU deployment8 x RTX PRO 6000
MIG deployment5 x H100 80GB

Optional services such as VLM generation, image captioning, VLM reranking, Nemotron Parse, and audio processing can add accelerator demand. Do not size from GPU count alone. Verify framebuffer requirements, tensor-parallel placement, supported NIM profiles, NVLink dependencies, and whether the workload is primarily ingestion-heavy or query-heavy.

Storage Capacity

Plan for at least 200 GB of available disk on every node that can host NIM workloads. NVIDIA attributes most of this to model downloads and caches, with additional capacity required for container images, vector indices, object data, logs, and temporary files.

That 200 GB figure is a deployment floor, not a production capacity model. A durable design normally separates:

  • Fast node-local image and runtime storage.
  • Persistent NIM model caches.
  • Vector database volumes.
  • Object storage volumes.
  • Ingestor working data.
  • Observability retention.
  • Backup or snapshot capacity.

Kubernetes Services

Before continuing, confirm:

  • Cluster-admin access for operator installation.
  • A working default StorageClass, or explicit StorageClasses in your values overlay.
  • Dynamic PVC provisioning.
  • DNS and outbound access to required registries and model repositories.
  • Adequate CPU and memory on non-GPU nodes for Elasticsearch, SeaweedFS, Redis, operators, and observability.
  • A load-balancing or ingress pattern that does not expose internal NIM, vector database, or ingestion services directly.
  • A secret-management approach for NGC credentials, API keys, database credentials, and TLS material.
  • A rollback window that accounts for model download and cache rebuild time.

Run a basic readiness check before installing anything:

kubectl version
helm version

kubectl get nodes -o wide
kubectl get storageclass
kubectl get pvc --all-namespaces

kubectl get nodes \
  -L nvidia.com/gpu.present,nvidia.com/gpu.product,nvidia.com/gpu.count

Successful output should show schedulable GPU nodes, a usable StorageClass, and no unresolved PVC problems. If GPU labels are absent before the GPU Operator is installed, record the node names and hardware inventory so you can validate discovery afterward.

Install the GPU and NIM Dependencies

Register the Required Helm Repositories

Register the NVIDIA and Elastic Helm repositories by using the current commands in their official installation documentation, then update local repository metadata.

Keep repository registration separate from release installation in production automation. This lets you approve a repository source once, mirror charts internally where required, and deploy only pinned chart versions from a controlled location.

Install the NVIDIA GPU Operator

The following example pins the GPU Operator to the current patch baseline used for this tutorial. Change the version only after checking the operator platform-support matrix and your node operating system.

export GPU_OPERATOR_VERSION="v26.3.3"

helm upgrade --install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --version "${GPU_OPERATOR_VERSION}" \
  --wait \
  --timeout 20m

If your platform already manages the NVIDIA driver or container toolkit, do not blindly use the default installation. Set the appropriate operator values to disable components that are already owned by the cloud provider, base image, or infrastructure team.

Validate the operator before installing NIM services:

kubectl get pods -n gpu-operator
kubectl get clusterpolicy
kubectl get nodes \
  -L nvidia.com/gpu.present,nvidia.com/gpu.product,nvidia.com/gpu.count

All required operands should be running, ClusterPolicy should report a ready state, and GPU nodes should advertise the expected product and count.

Install the NVIDIA NIM Operator

The NIM Operator depends on the GPU Operator. It manages the NIM cache and NIM service custom resources used by the Blueprint.

export NIM_OPERATOR_VERSION="3.1.1"

helm upgrade --install nim-operator nvidia/k8s-nim-operator \
  --namespace nim-operator \
  --create-namespace \
  --version "${NIM_OPERATOR_VERSION}" \
  --wait \
  --timeout 10m

Validate the controller and its custom resources:

kubectl get pods -n nim-operator
kubectl get crd | grep -E 'nimcache|nimservice'

Do not continue until the controller is running and the expected CRDs exist. A missing CRD can leave the Blueprint chart partially installed while NIM resources remain unrecognized.

Install the ECK Operator for Elasticsearch

Elasticsearch is the default vector database for RAG Blueprint 2.6.0. The Helm deployment uses Elastic Cloud on Kubernetes to manage it.

helm upgrade --install elastic-operator elastic/eck-operator \
  --namespace elastic-system \
  --create-namespace \
  --wait \
  --timeout 10m

Validate the operator:

kubectl get pods -n elastic-system
kubectl get crd | grep elasticsearch

You can omit ECK only when your final values disable chart-managed Elasticsearch and point both the RAG and ingestion services at a supported alternative backend.

Prepare the Namespace and Secrets

Create the workload namespace first:

export RAG_NAMESPACE="rag"

kubectl create namespace "${RAG_NAMESPACE}" \
  --dry-run=client \
  -o yaml | kubectl apply -f -

Create the NGC Registry Secret

Keep the NGC API key in your shell environment, CI secret store, or enterprise secret manager. Do not place it in a Git repository or commit it inside a Helm values file.

test -n "${NGC_API_KEY}" || {
  echo "NGC_API_KEY is not set"
  exit 1
}

kubectl create secret docker-registry ngc-secret \
  --namespace "${RAG_NAMESPACE}" \
  --docker-server=nvcr.io \
  --docker-username='$oauthtoken' \
  --docker-password="${NGC_API_KEY}" \
  --dry-run=client \
  -o yaml | kubectl apply -f -

Create the NGC API Secret

The chart and NIM resources also require access to the NGC API key as a generic Kubernetes secret.

kubectl create secret generic ngc-api \
  --namespace "${RAG_NAMESPACE}" \
  --from-literal=NGC_API_KEY="${NGC_API_KEY}" \
  --dry-run=client \
  -o yaml | kubectl apply -f -

Manage Additional API Keys

If you use hosted inference services, external embedding services, an external Elasticsearch cluster, or another managed vector database, create a separate secret for those credentials. Prefer External Secrets Operator, a CSI secrets-store driver, or your platform’s native secret integration so Kubernetes Secret objects are reconciled from an authoritative vault.

The deployment should reference existing secret names. Helm should not be the system that owns long-lived production credentials.

Build a Production Override File

Start with the upstream chart defaults and create a small override file containing only deliberate changes. Copying the entire default values file into your repository makes future upgrades harder because every upstream change becomes hidden behind stale local values.

This example keeps Elasticsearch enabled, references pre-created secrets, makes the frontend internal, and assigns explicit persistence:

imagePullSecret:
  create: false
  name: ngc-secret

ngcApiSecret:
  create: false
  name: ngc-api

frontend:
  service:
    type: ClusterIP

ingestor-server:
  persistence:
    enabled: true
    storageClass: fast-rwo
    size: 100Gi

seaweedfs:
  allInOne:
    data:
      storageClass: capacity-rwo
      size: 200Gi

eck-elasticsearch:
  enabled: true
  volumeClaimDeletePolicy: DeleteOnScaledownOnly

Save the file as values-enterprise.yaml.

Change the storage class names, capacity, access modes, reclaim behavior, and secret references for your platform. The example deliberately does not expose any service through a public load balancer. Add ingress only after you have identity, TLS, network policy, and request-size controls defined.

This overlay is a deployment starting point, not a complete security baseline. Review the version-matched chart defaults for object-store credentials, Elasticsearch security, TLS, pod security, and network exposure. Replace defaults before production and keep the resulting credentials in a managed secret system.

Before applying the chart, render it locally and inspect the result:

export RAG_RELEASE="rag"
export RAG_CHART="./nvidia-blueprint-rag-v2.6.0.tgz"

helm template "${RAG_RELEASE}" "${RAG_CHART}" \
  --namespace "${RAG_NAMESPACE}" \
  --values values-enterprise.yaml \
  > rendered-rag.yaml

kubectl apply \
  --namespace "${RAG_NAMESPACE}" \
  --server-side \
  --dry-run=server \
  --filename rendered-rag.yaml

The server-side dry run catches missing CRDs, invalid fields, admission-policy failures, and some namespace-level policy conflicts before the installation begins.

Deploy the RAG Blueprint with Helm

Install the pinned chart archive with a timeout long enough for first-time model downloads. The --atomic flag rolls the release back when the deployment fails, while the extended timeout prevents Helm from treating normal cache creation as a premature failure.

helm upgrade --install "${RAG_RELEASE}" "${RAG_CHART}" \
  --namespace "${RAG_NAMESPACE}" \
  --values values-enterprise.yaml \
  --atomic \
  --timeout 90m

A first deployment can take 60 to 70 minutes. Most of that time can be spent downloading and preparing model caches, and progress may not be obvious from the pod list alone.

Monitor the release from a second terminal:

watch kubectl get pods -n "${RAG_NAMESPACE}"

Inspect NIM cache and service resources separately:

kubectl get nimcache -n "${RAG_NAMESPACE}"
kubectl get nimservice -n "${RAG_NAMESPACE}"

kubectl describe nimcache -n "${RAG_NAMESPACE}"
kubectl describe nimservice -n "${RAG_NAMESPACE}"

Then review events in time order:

kubectl get events -n "${RAG_NAMESPACE}" \
  --sort-by='.lastTimestamp'

Do not restart downloading pods simply because they remain in an initialization state for several minutes. First check cache resources, PVC status, image-pull events, disk pressure, and NGC authentication.

Validate the Platform in Layers

A reliable validation sequence moves from infrastructure to model serving, then to ingestion, retrieval, and generation.

Validate Workloads and Storage

kubectl get pods -n "${RAG_NAMESPACE}" -o wide
kubectl get pvc -n "${RAG_NAMESPACE}"
kubectl get svc -n "${RAG_NAMESPACE}"
kubectl get endpoints -n "${RAG_NAMESPACE}"

Expected results:

  • All required PVCs are Bound.
  • NIM cache resources complete successfully.
  • NIM services become ready.
  • RAG server and ingestor server pods are running.
  • Elasticsearch, SeaweedFS, Redis, and supporting services have endpoints.
  • No pod is repeatedly restarting or remaining unschedulable.

Validate the RAG Server

Forward the RAG server locally:

kubectl port-forward \
  --namespace "${RAG_NAMESPACE}" \
  service/rag-server 8081:8081

From another terminal, call the dependency-aware health endpoint:

curl --fail --silent --show-error \
  --proto-default http \
  'localhost:8081/v1/health?check_dependencies=true'

A successful response should indicate that the RAG server can reach its dependencies. If the basic health endpoint works but dependency checking fails, investigate vector database, embedding, reranking, and LLM service connectivity.

Validate the Ingestor Server

Forward the ingestor service:

kubectl port-forward \
  --namespace "${RAG_NAMESPACE}" \
  service/ingestor-server 8082:8082

Use the Blueprint user interface, ingestion notebook, or API schema to create a test collection and upload a small representative file. Start with one clean PDF or text document, not a large mixed corpus.

The test is complete only when:

  • The upload request succeeds.
  • Ingestion status reaches a completed state.
  • Chunks appear in the selected vector database.
  • Object data is present in the configured object store.
  • A query retrieves the uploaded content.
  • The answer is grounded in the retrieved text.

Understand the RAG and Ingestion Services

RAG Server Responsibilities

The RAG server owns the latency-sensitive query workflow. It accepts questions, optionally rewrites or decomposes them, generates a query embedding, retrieves candidates, applies metadata filters, reranks results, assembles context, invokes the generation model, and streams the answer.

Scale the RAG server based on concurrent query demand, end-to-end latency, time to first token, and downstream NIM capacity. Adding RAG server replicas does not help when the bottleneck is a single saturated reranker, vector database, or LLM service.

Ingestor Server Responsibilities

The ingestor server is the control surface for document processing. It accepts files and metadata, tracks ingestion status, calls NV-Ingest, coordinates extraction services, generates embeddings, and writes data to the vector and object stores.

Scale ingestion independently from queries. Bulk ingestion creates bursty CPU, GPU, memory, storage, and network demand. Production environments often need admission control, queueing, file-size limits, and scheduled ingestion windows so large indexing jobs do not destabilize interactive query traffic.

Choose Between Elasticsearch and Milvus

RAG Blueprint 2.6.0 uses Elasticsearch by default. Milvus remains an optional backend. The choice should follow your search requirements and operating model, not a generic claim that one vector database is always faster.

CriterionElasticsearchMilvus
Blueprint statusDefault Helm backendOptional backend
Best fitTeams combining vector, keyword, filtering, and mature search operationsTeams centered on vector-native indexing and Milvus operations
Hybrid searchDense and sparse search with rank fusionDense and sparse search with rank fusion
Metadata filteringElasticsearch Query DSL and generated filtersMilvus expression filtering
Kubernetes dependencyECK when chart-managedMilvus services and dependencies
Default accelerationCPU in the standard chartCan use GPU acceleration where supported
Operational familiarityStrong fit for existing Elastic teamsStrong fit for existing Milvus teams
Backend migrationReingestion requiredReingestion required

Use Elasticsearch When

Choose Elasticsearch when the organization already operates Elastic, expects substantial keyword and metadata filtering, needs a familiar search API, or wants the Blueprint’s default path with fewer chart changes.

The standard chart deploys Elasticsearch on CPU. GPU-accelerated Elasticsearch requires additional enterprise access and specific images, and GPU search is not the default Helm behavior. Treat that as a separate design and licensing review.

Use Milvus When

Choose Milvus when vector-native operations are already standardized, the platform team has Milvus expertise, or the workload benefits from a Milvus-specific indexing and scaling design.

Switching the chart to Milvus requires consistent settings in both the RAG server and ingestor server. It also requires disabling chart-managed Elasticsearch when it is no longer used.

A minimal override pattern looks like this:

envVars:
  APP_VECTORSTORE_NAME: "milvus"
  APP_VECTORSTORE_URL: "<MILVUS_SERVICE_ENDPOINT>"

ingestor-server:
  envVars:
    APP_VECTORSTORE_NAME: "milvus"
    APP_VECTORSTORE_URL: "<MILVUS_SERVICE_ENDPOINT>"

eck-elasticsearch:
  enabled: false

nv-ingest:
  milvusDeployed: true

Replace the endpoint placeholder with the in-cluster service endpoint and connection scheme required by your Milvus deployment. Reingest documents after switching. Existing Elasticsearch indices are not migrated into Milvus by the Blueprint.

Configure Hybrid Search and Reranking

Dense retrieval is useful when the query and source express the same meaning with different words. Sparse or keyword retrieval is useful when exact terms, identifiers, product names, error codes, and uncommon phrases matter. Enterprise corpora usually need both.

Hybrid search runs both retrieval methods and merges their result sets. The current Blueprint supports reciprocal rank fusion and weighted ranking. When using an Elasticsearch edition that does not provide reciprocal rank fusion, select weighted ranking explicitly.

The following overlay enables hybrid retrieval with a modest dense-search bias:

envVars:
  APP_VECTORSTORE_SEARCHTYPE: "hybrid"
  APP_VECTORSTORE_RANKER_TYPE: "weighted"
  APP_VECTORSTORE_DENSE_WEIGHT: "0.6"
  APP_VECTORSTORE_SPARSE_WEIGHT: "0.4"

ingestor-server:
  envVars:
    APP_VECTORSTORE_SEARCHTYPE: "hybrid"
    APP_VECTORSTORE_RANKER_TYPE: "weighted"
    APP_VECTORSTORE_DENSE_WEIGHT: "0.6"
    APP_VECTORSTORE_SPARSE_WEIGHT: "0.4"

Apply the same vector-store settings to both services. Configuration drift between ingestion and retrieval can create indices that exist but do not behave as expected.

Reranking should operate on a larger candidate set than the number of chunks finally passed to the LLM. A practical starting point is to retrieve 40 to 50 candidates, rerank them, and send the best 8 to 12 chunks forward. The correct values depend on chunk size, corpus redundancy, query complexity, context-window budget, and latency objectives.

Do not tune weights or top-k values against anecdotal questions. Use a representative evaluation set and record retrieval recall, context relevancy, groundedness, answer accuracy, latency, and token consumption.

Plan Persistent Storage and Model Caches

A Helm release can be recreated. Your model caches, vector indices, object data, and ingestion state cannot be treated as disposable unless the environment is explicitly ephemeral.

NIM Model Caches

NIM caches reduce repeat deployment time and make pod startup more predictable. They also consume significant capacity and can become tightly coupled to model versions and GPU profiles.

For production:

  • Use a StorageClass with predictable throughput.
  • Protect caches from accidental release deletion.
  • Monitor cache PVC usage and node disk pressure.
  • Pre-warm new model versions before a maintenance window.
  • Keep enough free capacity for the old and new model during upgrades.
  • Document when a cache can be safely rebuilt rather than restored.

Vector Database Data

Elasticsearch or Milvus persistence contains the searchable representation of the corpus. Back it up according to the database’s supported mechanism, not by copying a live filesystem blindly.

Your recovery plan must answer:

  • Can the vector index be restored?
  • Can it be rebuilt from source documents and metadata?
  • How long will full reingestion take?
  • Where are source documents retained?
  • How are collection schemas and embedding-model versions recorded?
  • What happens when the embedding model changes?

Object Data and Ingestion State

SeaweedFS stores object data used by the current Blueprint release. The ingestor also needs persistent working space. Size these volumes for the source corpus, extracted artifacts, temporary processing, and growth.

Use separate storage classes when the access patterns differ. Vector search may need low-latency SSD-backed storage, while object data may favor capacity and durability.

Ingest Documents with Useful Metadata

A RAG platform becomes operationally valuable when documents are more than anonymous chunks. Metadata enables authorization filtering, lifecycle management, business ownership, targeted retrieval, deletion, and audit.

A useful collection schema can include:

  • business_domain
  • document_type
  • owner
  • classification
  • region
  • effective_date
  • expiration_date
  • product
  • version
  • status

The Blueprint can enforce custom metadata schemas. Strict validation is valuable because unknown or malformed fields should fail ingestion rather than silently corrupt retrieval behavior. Build schema changes into version control and test them before admitting new document producers.

The following Python pattern creates a collection through an ingestor endpoint supplied at runtime:

import os
from typing import Any

import requests

endpoint = os.environ["INGESTOR_ENDPOINT"].rstrip("/")

payload: dict[str, Any] = {
    "collection_name": "platform-runbooks",
    "description": "Approved infrastructure and AI platform runbooks",
    "metadata_schema": [
        {"name": "business_domain", "type": "string", "required": True},
        {"name": "classification", "type": "string", "required": True},
        {"name": "owner", "type": "string", "required": True},
        {"name": "effective_date", "type": "date", "required": False},
        {"name": "status", "type": "string", "required": True},
    ],
}

response = requests.post(
    f"{endpoint}/v1/collection",
    json=payload,
    timeout=30,
)
response.raise_for_status()

print(response.json())

Change the collection name, schema, and endpoint variable for your environment. Confirm the exact API request schema in the version-matched ingestor API documentation before execution.

The Blueprint data catalog can also track collection-level descriptions, tags, owners, creators, business domains, status, file counts, content-type indicators, and ingestion status. Use those fields to support stewardship and operations, not merely UI decoration.

Metadata Governance

Metadata values should come from controlled systems where possible. For example, source classification should come from a document-management policy, not a free-text field entered by an uploader.

Define:

  • Who can create collections.
  • Who owns each collection.
  • Which fields are mandatory.
  • Which values come from controlled vocabularies.
  • How document updates replace or supersede old versions.
  • How deletion propagates through object and vector stores.
  • Which metadata fields drive authorization filters.
  • How schema changes are tested and rolled back.

Secure Authentication, Secrets, and Service Exposure

The chart can deploy services quickly, but a production security boundary must be designed around it.

Keep Internal Services Internal

Do not expose NIM services, Redis, Elasticsearch, SeaweedFS, or the ingestor directly to untrusted networks. Use ClusterIP services, namespace-scoped network policy, controlled ingress, and an API gateway or application tier for user-facing access.

Separate ingestion identities from query identities. A client that can ask questions should not automatically have permission to create collections, upload documents, or delete data.

Centralize Secrets

Use a managed secret store and short-lived credentials where supported. At minimum:

  • Rotate the NGC API key according to policy.
  • Use dedicated credentials for external vector databases.
  • Keep database authentication enabled in production.
  • Use TLS for ingress and service-to-service connections that cross trust boundaries.
  • Avoid default object-store or observability credentials.
  • Restrict who can read Secrets in the rag namespace.
  • Prevent secrets from appearing in rendered Helm manifests, CI logs, or support bundles.

Add Authorization Above Retrieval

Metadata filtering is not a substitute for access control unless the caller’s identity and entitlements are reliably translated into mandatory server-side filters. Never trust a user-supplied filter to protect sensitive documents.

A governed query path should derive permitted collections and metadata predicates from an authenticated identity, apply them before retrieval, and record the decision in an audit trail.

Deploy with MIG When GPU Consolidation Is Required

MIG can reduce the H100 footprint from the default eight-GPU layout to a documented five-H100 layout by dividing selected GPUs into hardware-isolated instances. It is useful when smaller NIM services do not require a full GPU and predictable isolation matters.

It is not a universal oversubscription mechanism. The large generation model can still require full GPUs and tensor parallelism. MIG profiles are GPU-model specific, reconfiguration can disrupt workloads, and ingestion performance can decline when extraction services receive smaller slices.

Enable the Mixed MIG Strategy

The Blueprint MIG guide uses the GPU Operator’s mixed strategy because the deployment combines full GPUs with MIG resources.

kubectl patch clusterpolicies.nvidia.com/cluster-policy \
  --type=json \
  --patch='[
    {
      "op": "replace",
      "path": "/spec/mig/strategy",
      "value": "mixed"
    }
  ]'

Apply the NVIDIA-provided, GPU-specific MIG configuration from the version-matched Blueprint deployment assets:

kubectl apply \
  --filename mig-slicing/mig-config-h100.yaml

Label the target node with the configuration name defined in that file:

kubectl label node <GPU_NODE_NAME> \
  nvidia.com/mig.config=<MIG_CONFIGURATION_NAME> \
  --overwrite

Wait for the operator to report success:

kubectl get node <GPU_NODE_NAME> \
  -o jsonpath='{.metadata.labels.nvidia\.com/mig\.config\.state}{"\n"}'

Do not install the MIG Helm profile until the state is success.

Deploy the MIG Values Profile

Use the Blueprint’s version-matched MIG values file as an additional overlay:

helm upgrade --install "${RAG_RELEASE}" "${RAG_CHART}" \
  --namespace "${RAG_NAMESPACE}" \
  --values values-enterprise.yaml \
  --values mig-slicing/values-mig-h100.yaml \
  --atomic \
  --timeout 90m

Validate placement and resource names:

kubectl get pods -n "${RAG_NAMESPACE}" -o wide
kubectl describe pod -n "${RAG_NAMESPACE}" <POD_NAME>
kubectl get nodes -L nvidia.com/mig.config,nvidia.com/mig.config.state

The documented H100 layout preserves full GPUs for workloads that need them and uses slices for smaller extraction and reranking services. Do not modify profile names or slice counts without confirming that every selected NIM supports the resulting memory and compute profile.

MIG Operational Caveat

NVIDIA currently warns that the MIG ingestion profile is scaled down and that large bulk ingestion jobs can fail. Test with your largest realistic files and batch sizes. When bulk loading is the dominant requirement, a full-GPU ingestion window or a separate ingestion cluster may be a better design than forcing every workload into the consolidated MIG layout.

Add Evaluation Before Declaring Success

A successful Helm release proves that Kubernetes objects became ready. It does not prove that the system retrieves the right context or produces grounded answers.

Build an evaluation set that includes:

  • Questions with known answers.
  • Exact identifiers and error codes.
  • Paraphrased questions.
  • Multi-document questions.
  • Questions that should return no answer.
  • Questions filtered by business domain or classification.
  • Recently updated documents.
  • Tables, images, and complex PDFs when those formats are in scope.

The NVIDIA evaluation workflow uses Ragas-based metrics, including:

MetricWhat it tests
Answer AccuracyAgreement with a reference answer
Context RelevancyWhether retrieved chunks address the question
Response GroundednessWhether answer claims are supported by retrieved context
Context RecallWhether relevant documents appear at selected top-k cutoffs

Run the same evaluation set before and after changes to the embedding model, chunking, metadata schema, vector database, hybrid-search weights, reranking, prompts, or generation model. Store the configuration, test dataset version, results, and traces together so a regression can be reproduced.

Enable Observability

The Blueprint includes an observability path based on OpenTelemetry, Zipkin, Prometheus, and Grafana. In the Helm deployment, those components are not simply a reason to enable every bundled dashboard. Decide whether to integrate with the enterprise telemetry platform or deploy the packaged stack for a controlled environment.

A minimal chart overlay can enable the bundled components:

serviceMonitor:
  enabled: true

opentelemetry-collector:
  enabled: true

zipkin:
  enabled: true

kube-prometheus-stack:
  enabled: true

envVars:
  APP_TRACING_ENABLED: "True"

Install Prometheus Operator CRDs before enabling resources that depend on them.

At minimum, collect:

  • End-to-end query latency.
  • Retrieval time.
  • Reranking time.
  • LLM generation time.
  • Time to first token.
  • Ingestion throughput and failure count.
  • Pages and elements processed.
  • Vector database latency and index growth.
  • NIM request rate, error rate, queueing, and GPU utilization.
  • PVC usage and node disk pressure.
  • Pod restarts, scheduling failures, and cache readiness.

Correlate application traces with Kubernetes events and NIM metrics. A slow answer can originate in retrieval, reranking, model generation, storage, network, or a cold service. One aggregate latency number is not enough.

Troubleshooting the Deployment

Pods Remain Pending

Check scheduling events and resource requests:

kubectl get pods -n "${RAG_NAMESPACE}" --field-selector=status.phase=Pending

kubectl describe pod \
  --namespace "${RAG_NAMESPACE}" \
  <PENDING_POD_NAME>

Common causes include insufficient GPU resources, a mismatched MIG resource name, unavailable CPU or memory, unbound PVCs, node affinity, taints, and topology constraints.

NIM Services Do Not Become Ready

Inspect NIM resources and controller logs:

kubectl get nimcache,nimservice -n "${RAG_NAMESPACE}"

kubectl describe nimcache \
  --namespace "${RAG_NAMESPACE}" \
  <NIMCACHE_NAME>

kubectl logs \
  --namespace nim-operator \
  deployment/nim-operator-k8s-nim-operator \
  --tail=200

Look for invalid NGC credentials, unsupported model profiles, cache capacity problems, image-pull failures, and insufficient GPU memory.

PVCs Remain Pending

kubectl get pvc -n "${RAG_NAMESPACE}"
kubectl describe pvc -n "${RAG_NAMESPACE}" <PVC_NAME>
kubectl get storageclass

Confirm that the StorageClass supports the requested access mode, volume binding mode, topology, and capacity. Local-path storage can be useful for a lab but is not a substitute for a durable production storage design.

RAG Health Works but Queries Fail

Test each dependency:

kubectl get endpoints -n "${RAG_NAMESPACE}"
kubectl get svc -n "${RAG_NAMESPACE}"
kubectl logs -n "${RAG_NAMESPACE}" deployment/rag-server --tail=200

Check that the RAG server and ingestor use the same vector backend settings, the target collection exists, embedding dimensions match the index, the reranker is reachable, and the generation NIM is ready.

Ingestion Is Slow or Fails

Inspect the ingestor, NV-Ingest, extraction NIMs, vector database, and object store rather than only the upload client.

kubectl logs -n "${RAG_NAMESPACE}" deployment/ingestor-server --tail=200
kubectl get events -n "${RAG_NAMESPACE}" --sort-by='.lastTimestamp'
kubectl top pods -n "${RAG_NAMESPACE}"

The kubectl top command requires a working Kubernetes Metrics API. The current release documents a 400 MB limit for individual file uploads, slower vector database upload behavior with Elasticsearch in Helm deployments, and reduced bulk-ingestion capacity in the MIG profile. Split oversized files, reduce batch concurrency, verify storage throughput, and test without MIG when diagnosing ingestion bottlenecks.

Helm Times Out During the First Install

First deployment can legitimately take 60 to 70 minutes. NIM cache downloads can consume 40 to 50 minutes, followed by service initialization and pod startup.

Before rerunning Helm:

kubectl get nimcache -n "${RAG_NAMESPACE}"
kubectl get pods -n "${RAG_NAMESPACE}"
kubectl get pvc -n "${RAG_NAMESPACE}"
kubectl get events -n "${RAG_NAMESPACE}" --sort-by='.lastTimestamp'

Repeatedly uninstalling a release can make troubleshooting worse if it removes working objects while leaving large caches or PVCs in an unclear state.

Upgrade and Rollback Safely

Treat the Blueprint chart, GPU Operator, NIM Operator, model profiles, embedding model, vector schema, and application configuration as separate lifecycle items.

Before an upgrade:

  • Export the current Helm values.
  • Record chart and operator versions.
  • Snapshot or back up vector and object data.
  • Confirm whether CRDs require separate upgrade handling.
  • Pre-stage images and model caches.
  • Run the evaluation baseline.
  • Verify enough storage for old and new model artifacts.
  • Test the upgrade in a representative non-production cluster.

Capture the current release values:

helm get values "${RAG_RELEASE}" \
  --namespace "${RAG_NAMESPACE}" \
  --all \
  > rag-values-before-upgrade.yaml

helm history "${RAG_RELEASE}" \
  --namespace "${RAG_NAMESPACE}"

Use helm diff in your delivery pipeline if it is approved in your environment, then perform a controlled upgrade with a pinned chart.

For a configuration-only failure, Helm rollback may be sufficient:

helm rollback "${RAG_RELEASE}" <REVISION> \
  --namespace "${RAG_NAMESPACE}" \
  --wait \
  --timeout 90m

Helm rollback does not reverse data migrations, vector-schema changes, reingestion, model-cache changes, or external database updates. Define a data rollback plan before changing the vector backend, embedding model, metadata schema, or collection design.

Conclusion

Deploying the NVIDIA RAG Blueprint on Kubernetes is less about running one Helm command and more about assembling a dependable retrieval service. The GPU Operator, NIM Operator, RAG server, ingestor server, extraction services, vector database, object storage, model caches, and observability stack all have separate operational roles.

Start with the default Elasticsearch path, explicit persistent storage, pre-created secrets, and the smallest configuration overlay that satisfies your platform requirements. Validate the ingestion and query paths separately. Only then add hybrid retrieval, reranking changes, advanced metadata, external databases, or MIG consolidation.

The production gate should be evidence, not pod readiness. A release is ready when documents can be ingested predictably, metadata is enforced, authorized queries retrieve the correct context, answers remain grounded, telemetry identifies bottlenecks, and the platform can be upgraded or recovered without rediscovering its design under pressure.

External References

Leave a Reply

Discover more from Digital Thought Disruption

Subscribe now to keep reading and get access to the full archive.

Continue reading