Site icon Digital Thought Disruption

How to Share NVIDIA GPUs with MIG, Time-Slicing, and Resource Quotas

Paul Bryant

TL;DR

NVIDIA MIG and GPU time-slicing solve different utilization problems. MIG divides a supported physical GPU into hardware-backed instances with dedicated compute and memory resources. Time-slicing advertises multiple schedulable replicas of the same GPU, but those replicas still share memory, execution time, and the same fault domain.

Use MIG when workloads need stronger isolation and predictable resource boundaries. Use time-slicing for bursty, lightweight, or development workloads that can tolerate contention. Keep full GPUs for large models, latency-sensitive services, multi-GPU jobs, and workloads that can consume most of the device.

Kubernetes ResourceQuota can cap how many GPU resources a namespace claims, including MIG profiles and shared resource names. Quotas control admission and concurrency. They do not guarantee a percentage of GPU compute, memory, or performance.

Introduction

Expensive GPUs are often underutilized for a simple reason: Kubernetes normally schedules them as indivisible extended resources. A pod requests one GPU, the scheduler assigns one GPU, and the rest of the cluster treats that device as unavailable even when the application uses only a small portion of its compute or memory.

GPU sharing can improve utilization, but the phrase covers several very different operating models. A MIG instance is not the same thing as a time-sliced replica. A Kubernetes quota is not a performance control. Advertising eight shared resources from one physical GPU does not create eight isolated accelerators.

This distinction matters in production because the wrong sharing model can create unpredictable latency, out-of-memory failures, noisy-neighbor problems, and confusing capacity reports. The objective is not to maximize the number of pods that can start. The objective is to improve useful GPU utilization without violating workload requirements or making the platform impossible to operate.

The examples in this guide assume NVIDIA GPU Operator is already installed and healthy. The commands align with the GPU Operator 26.3 documentation set available in July 2026. Always validate the current component matrix, driver support, and generated MIG profiles for your specific GPU before production rollout.

What You Will Accomplish

By the end of this tutorial, you will be able to:

How GPU Sharing Appears to Kubernetes

Kubernetes does not schedule abstract percentages of a GPU through the traditional device plugin path. It schedules named extended resources advertised by the NVIDIA device plugin.

The resource name and its operational meaning depend on the configuration:

The important point is that Kubernetes schedules what the plugin advertises. It does not independently understand whether a resource is a physical GPU, a MIG partition, or one of several time-sliced references to the same hardware.

That makes naming, labeling, quotas, and admission policy part of the platform design. A shared resource should look shared to users. A MIG profile should expose enough information for workloads to request the correct memory and compute shape. An exclusive GPU should not be accidentally mixed into a shared node pool.

Full GPU Versus MIG Versus Time-Slicing

The best sharing model depends on the isolation, memory, performance, and scheduling characteristics the workload actually needs.

Evaluation AreaFull GPUMIGTime-Slicing
Hardware allocationEntire physical GPUHardware-backed GPU instanceShared access to one physical GPU or MIG instance
Compute isolationExclusive device allocationDedicated compute resources within the configured profileNo dedicated compute partition; processes interleave
Memory isolationEntire device memoryDedicated memory allocation per MIG instanceShared device memory, with no per-replica memory boundary
Fault isolationWorkload owns the allocated GPUStronger isolation between MIG instancesShared fault domain for workloads on the same underlying resource
Performance predictabilityHighestMore predictable within the sliceVariable under contention
OversubscriptionNoNo, unless time-slicing is layered onto MIG resourcesYes
Configuration flexibilitySimple, but coarseFixed profile geometry that may require maintenance to changeReplica count can be changed through device-plugin configuration
Scheduler resourceUsually nvidia.com/gpunvidia.com/gpu with single strategy, or profile-specific resources with mixed strategyOriginal resource name or a .shared resource name
Best fitLarge training, latency-sensitive inference, multi-GPU jobs, memory-heavy workloadsMulti-tenant inference, bounded CUDA workloads, predictable partition sizesDevelopment, notebooks, bursty inference, low-duty-cycle jobs, test environments
Primary riskLow utilizationProfile fragmentation and reconfiguration overheadNoisy neighbors, memory exhaustion, variable latency, weak per-container observability

MIG Is Hardware Partitioning

Multi-Instance GPU partitions supported NVIDIA GPUs into separate GPU instances. Each instance receives dedicated compute and memory resources according to its profile. This creates a stronger boundary than normal process sharing and gives the scheduler discrete resources that can be allocated independently.

MIG is still not infinitely flexible. The available profiles are specific to the GPU model and memory size. An H100 80 GB exposes different profile names from an A100 40 GB or H200 141 GB. The geometry must also fit within the physical GPU, so a platform team cannot create arbitrary combinations of memory and compute.

Time-Slicing Is Oversubscribed Access

Time-slicing creates multiple schedulable references to the same underlying GPU resource. CUDA processes from different workloads interleave on the device. This can materially improve utilization when workloads spend time waiting, perform short bursts, or use little compute.

Time-slicing does not create per-replica GPU memory limits. It does not create proportional compute guarantees. Requesting two shared resources does not mean a pod receives twice the compute time. For that reason, this tutorial enables failRequestsGreaterThanOne and renames shared resources with a .shared suffix.

MIG and Time-Slicing Can Be Combined

NVIDIA supports time-slicing on resource types produced by the mixed MIG strategy. This creates a two-level design:

  1. MIG partitions the physical GPU into isolated hardware instances.
  2. Time-slicing oversubscribes one or more MIG resource types.

This can be useful for many very small workloads, but the isolation boundary remains the MIG instance. Workloads time-sliced inside the same MIG instance still contend for that instance’s memory and compute allocation. The design is more complex to explain, observe, and troubleshoot, so it should be introduced only after the platform team has stable MIG operations.

Prerequisites and Safety Checks

Before changing GPU sharing, confirm the following:

MIG configuration changes are disruptive. MIG Manager stops operator-managed GPU clients while it changes the device geometry, and the GPU must not have active user workloads. Some platforms can require a reboot when changing MIG mode. Treat the operation like a node maintenance event, not a harmless label update.

Time-slicing changes are less disruptive, but they still change the resource capacity advertised to Kubernetes. Existing workloads can continue when the device plugin is restarted, but new scheduling decisions can immediately use the new capacity. Apply changes during a controlled maintenance period and validate allocatable resources before reopening the node pool.

Check GPU and MIG Support

Start by listing GPU-related labels across the cluster:

kubectl get nodes \
  -L nvidia.com/gpu.present \
  -L nvidia.com/gpu.product \
  -L nvidia.com/mig.capable \
  -L nvidia.com/mig.config \
  -L nvidia.com/mig.config.state

A MIG-capable node should report:

nvidia.com/gpu.present=true
nvidia.com/mig.capable=true

Do not assume that every GPU from a modern architecture supports MIG. Use the current NVIDIA supported-GPU list and the nvidia.com/mig.capable label as the practical source of truth for the node.

Inspect the GPU inventory from the driver container on the target node:

DRIVER_POD=$(kubectl get pods -n gpu-operator \
  --field-selector spec.nodeName=gpu-worker-01 \
  -o name | grep nvidia-driver-daemonset | head -n1)

kubectl exec -n gpu-operator "${DRIVER_POD}" -- nvidia-smi -L

This command assumes GPU Operator manages the driver as a container. If drivers are preinstalled on the host, run nvidia-smi -L through your approved node-access or diagnostic process instead.

You can also inspect the node labels in a compact form:

kubectl get node gpu-worker-01 -o jsonpath='{.metadata.labels}' | jq .

Before changing anything, capture the current configuration:

kubectl get node gpu-worker-01 \
  -o jsonpath='{.metadata.labels.nvidia\.com/mig\.config}{"\n"}'

kubectl get clusterpolicy cluster-policy -o yaml > cluster-policy-before-gpu-sharing.yaml
kubectl get node gpu-worker-01 -o yaml > gpu-worker-01-before-gpu-sharing.yaml

These files give you a recoverable baseline for profile names, strategy, and device-plugin configuration.

Choose the Correct MIG Strategy

GPU Operator supports two MIG strategies that affect how resources are advertised.

Single Strategy

Use single when all MIG devices on a node use the same profile. The device plugin advertises the instances as nvidia.com/gpu, and GPU Feature Discovery adds product labels that include the MIG profile.

This simplifies workload manifests because applications continue requesting one nvidia.com/gpu. The tradeoff is that the resource name does not express the slice size, so scheduling policy should include a node selector or affinity rule that targets the correct GPU product label.

Mixed Strategy

Use mixed when a node exposes multiple MIG profile types. Kubernetes receives separate resource names such as:

nvidia.com/mig-1g.10gb
nvidia.com/mig-2g.20gb
nvidia.com/mig-3g.40gb

Mixed strategy creates clearer contracts for application teams. A workload explicitly asks for the profile it needs. It also creates more quota keys, more capacity dimensions, and more chances for stranded capacity when demand does not match the configured geometry.

Practical Selection Rule

Use single strategy for homogeneous worker pools dedicated to one workload class. Use mixed strategy when the business value of multiple instance sizes is worth the additional scheduling and quota complexity.

Configure MIG Manager with Generated Profiles

GPU Operator 26.3 and later can generate a per-node MIG configuration ConfigMap from the hardware discovered through NVML. The ConfigMap name follows this pattern:

<node-name>-mig-config

List the generated ConfigMaps:

kubectl get configmap -n gpu-operator | grep mig-config

Inspect the profiles generated for a specific node:

kubectl get configmap -n gpu-operator \
  gpu-worker-01-mig-config \
  -o yaml

Look for profile names such as:

all-disabled
all-enabled
all-balanced
all-1g.10gb
all-2g.20gb
all-3g.40gb

The exact names depend on the installed GPU. Do not copy a profile name from another model and assume it is valid. If a per-node generated ConfigMap is absent, confirm the GPU Operator version and driver branch. Older drivers that cannot query profiles while MIG mode is disabled can cause GPU Operator to use its static default MIG configuration instead.

Apply a Homogeneous MIG Profile

The following example configures a node with the single strategy and applies an H100 80 GB profile containing homogeneous 1g.10gb instances. Replace the profile with one found in the generated ConfigMap for your node.

kubectl patch clusterpolicy cluster-policy \
  --type='json' \
  -p='[{"op":"replace","path":"/spec/mig/strategy","value":"single"}]'

kubectl label node gpu-worker-01 \
  nvidia.com/mig.config=all-1g.10gb \
  --overwrite

MIG Manager updates the nvidia.com/mig.config.state label as it applies the change. Monitor the state:

kubectl get node gpu-worker-01 \
  -o jsonpath='{.metadata.labels.nvidia\.com/mig\.config\.state}{"\n"}'

Wait for the state to become success before scheduling workloads:

kubectl wait node/gpu-worker-01 \
  --for=jsonpath='{.metadata.labels.nvidia\.com/mig\.config\.state}'=success \
  --timeout=15m

Then verify the resulting devices on the target node:

DRIVER_POD=$(kubectl get pods -n gpu-operator \
  --field-selector spec.nodeName=gpu-worker-01 \
  -o name | grep nvidia-driver-daemonset | head -n1)

kubectl exec -n gpu-operator "${DRIVER_POD}" -- nvidia-smi -L

For a successful homogeneous configuration, the node should advertise multiple nvidia.com/gpu resources and a product label that identifies the MIG profile.

Apply a Balanced Mixed Profile

For multiple profile sizes, switch to the mixed strategy and apply a generated balanced profile:

kubectl patch clusterpolicy cluster-policy \
  --type='json' \
  -p='[{"op":"replace","path":"/spec/mig/strategy","value":"mixed"}]'

kubectl label node gpu-worker-01 \
  nvidia.com/mig.config=all-balanced \
  --overwrite

After nvidia.com/mig.config.state reports success, inspect the allocatable resources:

kubectl get node gpu-worker-01 \
  -o jsonpath='{.status.allocatable}' | jq .

A mixed node can report resources similar to:

nvidia.com/mig-1g.10gb: 2
nvidia.com/mig-2g.20gb: 1
nvidia.com/mig-3g.40gb: 1

These counts are examples. The generated geometry is specific to the GPU model.

Define a Custom MIG Profile

Generated profiles should be the default choice because they are derived from the actual hardware. Create a custom profile only when the workload portfolio requires a geometry that the generated configurations do not provide.

The following ConfigMap defines a custom H100 80 GB geometry with five 1g.10gb instances and one 2g.20gb instance:

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-mig-config
  namespace: gpu-operator
data:
  config.yaml: |-
    version: v1
    mig-configs:
      all-disabled:
        - devices: all
          mig-enabled: false

      five-1g-one-2g:
        - devices: all
          mig-enabled: true
          mig-devices:
            "1g.10gb": 5
            "2g.20gb": 1

Apply the ConfigMap:

kubectl apply -f custom-mig-config.yaml

Point MIG Manager to the custom ConfigMap and select mixed strategy:

kubectl patch clusterpolicy cluster-policy \
  --type='json' \
  -p='[{"op":"replace","path":"/spec/migManager/config/name","value":"custom-mig-config"}]'

kubectl patch clusterpolicy cluster-policy \
  --type='json' \
  -p='[{"op":"replace","path":"/spec/mig/strategy","value":"mixed"}]'

Apply the profile to the node:

kubectl label node gpu-worker-01 \
  nvidia.com/mig.config=five-1g-one-2g \
  --overwrite

A custom ConfigMap must contain a key named config.yaml. If the profile does not fit the GPU geometry, MIG Manager will not reach the success state. Review the MIG Manager logs and restore the previous profile rather than repeatedly forcing the same invalid configuration.

Schedule Workloads Against MIG Slices

The workload manifest depends on the selected MIG strategy.

Schedule with Single Strategy

With a homogeneous single strategy, request nvidia.com/gpu: 1 and use node selection to target the correct MIG-backed pool:

apiVersion: v1
kind: Pod
metadata:
  name: mig-single-validation
spec:
  restartPolicy: Never
  nodeSelector:
    nvidia.com/mig.strategy: single
    nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3-MIG-1g.10gb
  containers:
    - name: cuda-vector-add
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04
      resources:
        limits:
          nvidia.com/gpu: 1

Change the product label to the exact value reported on your node. Successful execution should end with Test PASSED in the pod logs.

Schedule with Mixed Strategy

With mixed strategy, request the profile-specific extended resource:

apiVersion: v1
kind: Pod
metadata:
  name: mig-mixed-validation
spec:
  restartPolicy: Never
  containers:
    - name: cuda-vector-add
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04
      resources:
        limits:
          nvidia.com/mig-1g.10gb: 1

This manifest creates a clearer workload contract. The pod cannot silently land on a larger or smaller MIG profile because the resource name is explicit.

Kubernetes GPU resources normally belong in the container limits section. Kubernetes uses the GPU limit as the request when a separate request is omitted. If both are specified, the values must be equal.

Configure GPU Time-Slicing

Time-slicing is configured through the NVIDIA device plugin, not MIG Manager. The device plugin reads named configurations from a ConfigMap and applies them cluster-wide or per node. Under the current device-plugin model, the selected sharing method applies to all GPUs on a node rather than one chosen physical GPU, which is another reason to use dedicated shared node pools.

For production governance, use two settings:

Create a ConfigMap with an exclusive configuration and a four-replica shared configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: gpu-sharing-config
  namespace: gpu-operator
data:
  exclusive: |-
    version: v1
    flags:
      migStrategy: none

  shared-4: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        renameByDefault: true
        failRequestsGreaterThanOne: true
        resources:
          - name: nvidia.com/gpu
            replicas: 4

Apply the configuration:

kubectl apply -f gpu-sharing-config.yaml

Patch the ClusterPolicy so the device plugin uses the ConfigMap:

kubectl patch clusterpolicy cluster-policy \
  -n gpu-operator \
  --type merge \
  -p '{"spec":{"devicePlugin":{"config":{"name":"gpu-sharing-config"}}}}'

Apply time-slicing to a selected node:

kubectl label node gpu-worker-02 \
  nvidia.com/device-plugin.config=shared-4 \
  --overwrite

The node should restart or reconfigure the device-plugin and GPU Feature Discovery pods for that node. Verify the sharing labels:

kubectl get node gpu-worker-02 \
  -o jsonpath='{.metadata.labels}' | jq .

Look for values similar to:

nvidia.com/gpu.sharing-strategy=time-slicing
nvidia.com/gpu.replicas=4

Then inspect capacity and allocatable resources:

kubectl describe node gpu-worker-02

A node with one physical GPU and four replicas should advertise:

nvidia.com/gpu.shared: 4

The number is scheduling capacity, not four physical GPUs and not four guaranteed performance shares.

Schedule a Time-Sliced Workload

Request one shared access resource:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shared-gpu-validation
  namespace: team-a
spec:
  replicas: 3
  selector:
    matchLabels:
      app: shared-gpu-validation
  template:
    metadata:
      labels:
        app: shared-gpu-validation
    spec:
      nodeSelector:
        nvidia.com/gpu.sharing-strategy: time-slicing
      containers:
        - name: cuda-vector-add
          image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04
          command: ["/bin/bash", "-c", "--"]
          args:
            - while true; do /cuda-samples/vectorAdd; sleep 5; done
          resources:
            limits:
              nvidia.com/gpu.shared: 1

Scale the deployment up to the advertised replica count and confirm that all pods can run:

kubectl apply -f shared-gpu-validation.yaml
kubectl get pods -n team-a -l app=shared-gpu-validation

Do not treat a successful pod start as performance validation. Run representative concurrent workloads and measure latency, throughput, GPU memory use, and failure behavior under contention.

Apply Resource Quotas to GPU Sharing

ResourceQuota limits the total amount of an extended resource that pods in a namespace can request. The quota key must match the exact resource advertised by the device plugin.

Quota a Time-Sliced Resource

The following quota allows namespace team-a to claim at most four shared GPU access resources:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: shared-gpu-quota
  namespace: team-a
spec:
  hard:
    requests.nvidia.com/gpu.shared: "4"

Apply and inspect the quota:

kubectl apply -f shared-gpu-quota.yaml
kubectl describe resourcequota shared-gpu-quota -n team-a

A fifth concurrent pod that requests nvidia.com/gpu.shared should be rejected by admission until another claim is released.

Quota a MIG Profile

For a mixed MIG strategy, quota the specific profile resource:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: mig-profile-quota
  namespace: team-b
spec:
  hard:
    requests.nvidia.com/mig-1g.10gb: "3"

For homogeneous single strategy, the quota key remains:

requests.nvidia.com/gpu: "3"

If time-slicing is layered onto a mixed MIG resource and renameByDefault is enabled, both the workload and quota must use the renamed resource, for example:

requests.nvidia.com/mig-1g.10gb.shared: "4"

Kubernetes permits only the requests. prefix for extended-resource quotas. Do not create a separate limits. quota for the same GPU resource.

What the Quota Does Not Guarantee

A quota is an admission control, not a GPU quality-of-service mechanism.

If one physical GPU advertises eight time-sliced replicas and a namespace receives a quota of four, the namespace can claim four access references. That does not guarantee half of the GPU compute, half of its memory, or a fixed latency target. Other namespaces and other processes can still contend for the same physical device.

For stronger governance, combine ResourceQuota with:

Change MIG Profiles Safely

Changing MIG geometry should follow a repeatable maintenance workflow.

Record the Previous Profile

PREVIOUS_PROFILE=$(kubectl get node gpu-worker-01 \
  -o jsonpath='{.metadata.labels.nvidia\.com/mig\.config}')

echo "Previous MIG profile: ${PREVIOUS_PROFILE}"

Cordon and Drain the Node

kubectl cordon gpu-worker-01
kubectl drain gpu-worker-01 \
  --ignore-daemonsets \
  --delete-emptydir-data

Do not add --force simply to make the drain complete. Investigate unmanaged pods, disruption budgets, and stateful workloads first.

Apply the New Profile

kubectl label node gpu-worker-01 \
  nvidia.com/mig.config=all-2g.20gb \
  --overwrite

Wait for success and verify the hardware:

kubectl wait node/gpu-worker-01 \
  --for=jsonpath='{.metadata.labels.nvidia\.com/mig\.config\.state}'=success \
  --timeout=15m

DRIVER_POD=$(kubectl get pods -n gpu-operator \
  --field-selector spec.nodeName=gpu-worker-01 \
  -o name | grep nvidia-driver-daemonset | head -n1)

kubectl exec -n gpu-operator "${DRIVER_POD}" -- nvidia-smi -L

Restore Service

Run a validation pod for the newly advertised resource. Confirm pod logs, allocatable capacity, and application-level performance. Only then uncordon the node:

kubectl uncordon gpu-worker-01

Update Time-Slicing Safely

The GPU Operator does not automatically monitor changes made inside a time-slicing ConfigMap. After changing the replica count or settings, restart the NVIDIA device-plugin DaemonSet:

kubectl rollout restart \
  -n gpu-operator \
  daemonset/nvidia-device-plugin-daemonset

Existing workloads can continue running, but NVIDIA recommends using a maintenance period. The scheduler can see changed allocatable capacity after the plugin restarts, so verify the node before allowing new deployments.

Check the rollout:

kubectl rollout status \
  -n gpu-operator \
  daemonset/nvidia-device-plugin-daemonset

Then confirm the resource count:

kubectl describe node gpu-worker-02

When reducing replicas, check how many shared resources are already allocated. Do not reduce advertised capacity below current consumption without first scaling down or moving workloads.

Validation Checklist

A configuration is not complete until all control-plane and workload checks pass.

Node and Operator Validation

kubectl get pods -n gpu-operator -o wide
kubectl get nodes -L nvidia.com/gpu.product,nvidia.com/mig.capable,nvidia.com/mig.config,nvidia.com/mig.config.state,nvidia.com/gpu.sharing-strategy,nvidia.com/gpu.replicas

Confirm:

Hardware Validation

DRIVER_POD=$(kubectl get pods -n gpu-operator \
  --field-selector spec.nodeName=gpu-worker-01 \
  -o name | grep nvidia-driver-daemonset | head -n1)

kubectl exec -n gpu-operator "${DRIVER_POD}" -- nvidia-smi -L

For MIG, confirm the expected instance count and profile names. For time-slicing, nvidia-smi -L still shows the physical GPUs because time-slicing does not create hardware partitions.

Workload Validation

kubectl get pods -A -o wide
kubectl describe pod <validation-pod> -n <namespace>
kubectl logs <validation-pod> -n <namespace>

Confirm:

Observability Validation

Time-slicing has an important monitoring limitation: DCGM Exporter cannot reliably associate GPU metrics with individual containers when time-slicing is enabled through the NVIDIA device plugin. Node-level GPU metrics remain useful, but application teams should add service-level telemetry such as request latency, queue time, throughput, error rate, and model load failures.

Rollback Procedures

Roll Back a MIG Profile

Restore the previously captured profile:

kubectl cordon gpu-worker-01
kubectl drain gpu-worker-01 \
  --ignore-daemonsets \
  --delete-emptydir-data

kubectl label node gpu-worker-01 \
  nvidia.com/mig.config="${PREVIOUS_PROFILE}" \
  --overwrite

Wait for success, validate the devices, and uncordon the node.

To disable MIG completely:

kubectl label node gpu-worker-01 \
  nvidia.com/mig.config=all-disabled \
  --overwrite

Some environments require a reboot to complete a MIG mode transition. Follow the platform-specific maintenance process rather than repeatedly changing labels while the node remains in a pending or rebooting state.

Roll Back Time-Slicing

Switch the node back to the exclusive device-plugin configuration:

kubectl label node gpu-worker-02 \
  nvidia.com/device-plugin.config=exclusive \
  --overwrite

Verify that the node no longer advertises nvidia.com/gpu.shared and that nvidia.com/gpu.sharing-strategy returns to none.

If the configuration does not reload as expected, restart the device-plugin DaemonSet:

kubectl rollout restart \
  -n gpu-operator \
  daemonset/nvidia-device-plugin-daemonset

Scale down shared workloads before rollback. Their manifests request nvidia.com/gpu.shared, which will no longer exist after the node returns to exclusive mode.

Troubleshooting Common Failures

MIG Configuration Remains Pending

Likely causes:

Checks:

kubectl logs -n gpu-operator \
  -l app=nvidia-mig-manager \
  -c nvidia-mig-manager

kubectl get node gpu-worker-01 \
  -o jsonpath='{.metadata.labels.nvidia\.com/mig\.config\.state}{"\n"}'

Drain the node, confirm no user GPU processes remain, and restore a generated profile known to be valid for the hardware.

Expected MIG Resources Do Not Appear

Confirm the strategy. Under single strategy, MIG devices are advertised as nvidia.com/gpu. Under mixed strategy, profile-specific resource names appear.

kubectl get clusterpolicy cluster-policy \
  -o jsonpath='{.spec.mig.strategy}{"\n"}'

kubectl get node gpu-worker-01 \
  -o jsonpath='{.status.allocatable}' | jq .

A workload requesting nvidia.com/mig-1g.10gb will remain pending if the node uses single strategy.

Time-Slicing Capacity Does Not Change

The ConfigMap may have been updated without restarting the device plugin.

kubectl rollout restart \
  -n gpu-operator \
  daemonset/nvidia-device-plugin-daemonset

Also confirm that the node label references the correct named configuration:

kubectl get node gpu-worker-02 \
  -o jsonpath='{.metadata.labels.nvidia\.com/device-plugin\.config}{"\n"}'

Shared Pods Fail with UnexpectedAdmissionError

This normally means a container requested more than one shared resource while failRequestsGreaterThanOne is enabled.

Change the request to one:

resources:
  limits:
    nvidia.com/gpu.shared: 1

Scale the workload by adding pod replicas, not by requesting multiple shared GPU references in one container.

Time-Sliced Workloads Fail with GPU Out-of-Memory Errors

Time-slicing does not partition GPU memory. One process can consume memory required by other workloads on the same device.

Reduce the replica count, lower application memory use, move the workload to a larger GPU, use MIG for fixed memory boundaries, or return the workload to exclusive GPU allocation.

Pods Remain Pending After Quota Changes

Inspect both scheduling events and namespace quota usage:

kubectl describe pod <pod-name> -n <namespace>
kubectl describe resourcequota -n <namespace>

A quota error occurs at admission. A scheduler error occurs after admission when no node advertises the requested resource. These are separate failure domains and require different remediation.

When Not to Share a GPU

GPU sharing is a utilization technique, not a default best practice. Keep exclusive GPU allocation when any of the following applies:

A GPU that is already running near its compute, memory, or memory-bandwidth ceiling is not underutilized merely because one pod owns it. Sharing that device usually creates contention rather than efficiency.

A practical enterprise design separates GPU nodes by service class:

Give each pool distinct node labels, taints, quotas, monitoring, and ownership. Avoid switching the same nodes between exclusive, MIG, and time-sliced modes every day. Frequent reconfiguration increases disruption, complicates capacity planning, and makes incident diagnosis harder.

Review the profile mix using real workload evidence. Track queue time, rejected requests, GPU memory peaks, latency under concurrency, and stranded MIG capacity. The right configuration is the one that produces useful work predictably, not the one that advertises the largest resource count.

Conclusion

MIG, time-slicing, and ResourceQuota belong to the same GPU platform conversation, but they solve different problems.

MIG creates hardware-backed partitions with defined compute and memory shapes. Time-slicing creates oversubscribed access to an underlying GPU resource and accepts weaker isolation in exchange for flexibility and higher potential utilization. ResourceQuota limits how many of those advertised resources a namespace can claim, but it does not transform a shared reference into guaranteed compute or memory.

The safest implementation path is to begin with explicit workload classes. Keep demanding workloads on full GPUs, use MIG where fixed partitions match production requirements, and reserve time-slicing for workloads that tolerate contention. Separate the node pools, expose shared resources with clear names, enforce namespace quotas, and validate with concurrent application tests before calling the design complete.

External References

Exit mobile version