GPU Multi-Tenancy Without Security Theater: Isolation, Quotas, Noisy Neighbors, and Confidential Computing

Introduction

GPU sharing is easy to describe and difficult to govern.

A platform team can expose one physical accelerator as several scheduler-visible resources, divide it into Multi-Instance GPU partitions, present virtual GPUs to virtual machines, or assign the entire device to one workload. Kubernetes can place those workloads into separate namespaces. An enterprise scheduler can enforce quotas, fair share, priority, and preemption. Confidential computing can add attestation and encryption for data in use.

Those mechanisms are useful, but they are not interchangeable.

A scheduler can decide who runs next without isolating GPU memory. A namespace can separate API permissions without creating a hardware trust boundary. MIG can partition compute and memory without separating cluster administrators. A dedicated GPU can eliminate same-device contention without preventing a privileged host administrator from inspecting an ordinary virtual machine. Confidential computing can reduce that administrator trust requirement without guaranteeing availability, latency, or fair scheduling.

The architecture therefore cannot begin with, “Should we use MIG or time-slicing?” It must begin with a threat model, a tenancy model, and a definition of the service promise. Only then can the platform team select the correct combination of cluster, namespace, scheduler, GPU, network, storage, identity, and cryptographic controls.

TL;DR

A governed GPUaaS or AIaaS platform needs a stack of controls, not one multi-tenancy feature.

• Kubernetes namespaces and RBAC provide logical and administrative separation inside a cluster. They are not a complete security boundary against a hostile tenant or privileged cluster administrator.
• Quotas, fair-share scheduling, priority, and preemption govern entitlement and contention. They do not isolate GPU memory or protect data.
• Time-slicing improves utilization but does not provide the memory or fault isolation associated with MIG. It should be sold and operated as best-effort shared capacity unless additional controls justify a stronger promise.
• MIG creates hardware-backed GPU partitions with dedicated resources at the GPU Instance boundary, but the physical board, node, management plane, and some failure conditions remain shared.
• vGPU is a family of delivery models. Time-sliced vGPU and one-to-one MIG-backed vGPU have materially different isolation and performance properties.
• Whole-GPU allocation and dedicated nodes reduce noisy-neighbor and shared-failure risks, but they are not cryptographic isolation from platform administrators.
• Confidential computing protects data and models in use through measured launch, attestation, protected memory, and controlled key release. It does not replace scheduling, network policy, application security, or availability engineering.
• As of July 2026, NVIDIA’s published Confidential Containers reference architecture validates full GPU passthrough to one confidential virtual machine, including single-GPU and multi-GPU patterns. It does not establish a general support pattern for several confidential tenants sharing one physical GPU.
• The safest design is risk-tiered. Cooperative internal development, regulated production, and external paid-customer hosting should not receive the same tenancy architecture.

The Tenancy Unit Comes Before the GPU Mechanism

The word tenant is often used too loosely. In a real enterprise platform, several overlapping tenancy units exist, and each one drives a different control.

Tenancy unit	What it represents	Typical controls	Common mistake
Organization	Legal entity, enterprise, or customer account	Contract, identity federation, billing, top-level policy	Treating an organization as one technical namespace
Department	Budget owner or business capability	Guaranteed quota, showback, policy inheritance	Giving each department permanently reserved idle GPUs
Project	Delivery team, application, experiment, or funded initiative	Namespace, project quota, registry scope, storage scope	Letting project admins inherit cluster-wide authority
Environment	Development, test, staging, or production	Separate policy, release gates, priority class, node pool	Mixing production and experimental workloads because they share an owner
Workload	Pod, job, notebook, model server, or virtual machine	Service account, GPU resource, network policy, runtime policy	Assuming workload isolation is inherited automatically from the project
External customer	Contractual and security boundary	Dedicated control plane or cluster, stronger evidence, service limits	Selling shared-cluster convenience as customer isolation
Model or data domain	Sensitive model weights, training data, RAG corpus, or vector index	Registry permissions, encryption keys, lineage, retention	Treating compute isolation as model and data isolation

A useful operating model usually maps these units hierarchically:

The hierarchy matters because capacity policy and security policy rarely share the same owner. Finance may own the departmental entitlement. The platform team may own the cluster. Security may own the data classification and cryptographic requirements. An application team may own the namespace and workload. A customer contract may define performance and evidence obligations that supersede all internal defaults.

The design should therefore preserve both inheritance and exceptions. A project should inherit the department’s spending and capacity controls, but a regulated production environment may need a dedicated node pool or cluster even when the department’s development workloads use a shared pool.

A GPU Multi-Tenancy Threat Model

A threat model prevents the platform team from choosing controls based on feature names. It asks what must be protected, from whom, and under which failure conditions.

Assets That Need Protection

The protected assets normally include:

• model weights, adapters, checkpoints, embeddings, and prompt templates;
• training, fine-tuning, inference, and retrieval data;
• secrets, API credentials, signing keys, and data-encryption keys;
• GPU capacity and the service levels attached to it;
• model outputs, logs, traces, and cached intermediate data;
• tenant identity, billing records, and usage evidence;
• the integrity of the scheduler, device plugin, GPU configuration, and runtime;
• availability of the platform and the ability to recover from a device or node fault.

Relevant Actors and Failure Sources

The risk is not limited to a malicious pod. The platform must account for:

• a cooperative but careless internal user;
• a compromised workload running with ordinary namespace permissions;
• a malicious workload in another tenant;
• a project administrator with excessive Kubernetes privileges;
• a cluster administrator or host administrator;
• an operator, admission webhook, or custom resource with cluster-scoped authority;
• a hypervisor or management-plane administrator;
• a supply-chain compromise in a model, container, driver, or operator;
• a faulty GPU, driver crash, reset, thermal event, or node failure;
• a high-demand workload that unintentionally degrades every neighbor;
• an external customer attempting to escape contractual or technical boundaries.

Threat-to-Control Mapping

Threat or failure	Primary exposure	Control that helps	Control that does not solve it alone
One tenant consumes all available GPUs	Capacity and service fairness	Guaranteed quota, hard limit, fair share, priority, preemption	MIG, encryption
One workload reads or corrupts another workload’s GPU memory	Confidentiality and integrity	Hardware partitioning, strong device isolation, separate virtual machines, whole-GPU assignment	Namespace quota
Long kernels or memory-bandwidth pressure degrade neighbors	Performance	Dedicated GPU, MIG boundary, admission policy, workload profiling	Namespace RBAC
GPU reset affects several tenants	Availability and fault containment	Dedicated GPU or node, failure-domain-aware placement	Confidential computing
Project admin changes another project’s resources	Administrative isolation	Namespace-scoped RBAC, separate control plane, dedicated cluster	MIG
Platform admin can inspect ordinary guest memory	Privileged-administrator threat	Confidential virtual machine, GPU confidential computing, attestation, controlled key release	Dedicated namespace
Tenant accesses another tenant’s model or dataset	Data isolation	Separate identity, registry, storage, encryption key, network policy	GPU scheduler
Shared storage or cache leaks artifacts after reuse	Data remanence	Tenant-scoped volumes and caches, encryption, cleanup, reclaim policy	Whole-GPU allocation
Denial of service against node, fabric, or control plane	Availability	Admission limits, dedicated failure domains, rate limits, capacity reserve	Confidential computing
Malicious model or container compromises its own tenant	Application and supply chain	Provenance, signing, scanning, sandboxing, least privilege	MIG or encryption alone

The important design move is to separate security threats, performance interference, fault propagation, and economic contention. They may appear together during an incident, but they require different evidence and different controls.

The Isolation Stack: What Each Control Actually Solves

GPU multi-tenancy is best understood as a layered control system.

No layer makes the others unnecessary.

A dedicated cluster may still need fair-share scheduling among teams. MIG may still need network and storage isolation. Confidential computing may still need a hard quota because an attested workload can consume excessive capacity. A namespace may still need a dedicated GPU because logical isolation does not create predictable latency.

The architecture should identify the required strength at every layer and then select a service class that satisfies the full control set.

Isolation Dimensions Must Be Designed Separately

The phrase “GPU isolation” hides several different requirements.

Isolation dimension	Question the design must answer	Stronger control examples	Residual concerns
Memory isolation	Can one workload access another workload’s device memory?	MIG GPU Instance, one-to-one MIG-backed vGPU, whole-GPU assignment, validated virtual-machine boundary	Driver, host, board, and implementation trust remain relevant
Compute isolation	Can one workload consume the execution resources expected by another?	MIG partition, dedicated GPU, dedicated node	Shared CPU, PCIe, network, and storage can still bottleneck
Performance isolation	Can the platform defend latency or throughput under contention?	Dedicated GPU, capacity headroom, admission control, bounded MIG service class	Thermal, power, topology, and upstream bottlenecks remain
Fault isolation	Does a workload or device event disrupt other tenants?	Dedicated GPU, dedicated node, separate cluster or failure domain	Board, node, rack, fabric, and control-plane failures may still be shared
Administrative isolation	Can one tenant or operator alter another tenant’s resources?	Namespace RBAC, virtual control plane, dedicated cluster, separated operations roles	Privileged platform administrators may remain in the trust boundary
Cryptographic isolation	Can privileged infrastructure operators read data or model state while in use?	Confidential virtual machine, GPU confidential computing, attestation, key release	Application bugs, logging, side channels, denial of service, and key authority remain
Economic isolation	Can a tenant exceed its commercial or budget entitlement?	Hard limits, metering, showback, chargeback, contract enforcement	Technical usage units must map correctly to billing
Operational isolation	Can one tenant force upgrades, resets, or policy changes that disrupt others?	Stable service classes, dedicated pools, change windows, compatibility boundaries	Shared operators and drivers still couple lifecycle decisions

These dimensions should appear in architecture decisions, service descriptions, and customer contracts. A claim such as “isolated GPU” is incomplete unless the provider states which dimensions are isolated and which remain shared.

Whole GPUs, vGPU, MIG, and Time-Slicing

The allocation mechanism determines part of the isolation boundary, but it does not determine the whole tenancy architecture.

Whole-GPU Allocation

Whole-GPU allocation is the cleanest default when the workload is production-critical, latency-sensitive, difficult to preempt, or handling sensitive data.

In Kubernetes, a device plugin can expose the physical GPU as an extended resource and the scheduler can assign it to one pod. In a virtualized platform, the device may be passed through to one virtual machine. Either approach removes same-GPU workload contention when the platform truly prevents concurrent assignment.

Whole-GPU allocation provides strong compute and performance separation from other workloads that would otherwise share that device. It also simplifies troubleshooting, capacity evidence, and fault attribution. A reset or Xid event has a smaller customer impact than it would on a GPU divided among several tenants.

It does not automatically provide:

• cryptographic protection from a privileged host administrator;
• dedicated CPU, memory, PCIe, network, storage, power, or cooling;
• a separate Kubernetes or hypervisor management plane;
• protection from a malicious container running inside the assigned workload boundary;
• protection from node or fabric failure.

For higher-risk workloads, whole-GPU allocation should be paired with a dedicated node, dedicated cluster, or confidential virtual machine as required by the threat model.

Time-Slicing

NVIDIA time-slicing allows the device plugin to advertise multiple replicas of a GPU so that more workloads can be scheduled onto the device. The workloads interleave execution on the same physical GPU. This is an oversubscription mechanism, not a hardware partition.

NVIDIA’s current GPU Operator documentation explicitly distinguishes time-slicing from MIG. Time-slicing does not provide MIG-style memory or fault isolation, and requesting multiple time-sliced replicas does not create a proportional compute guarantee [1].

That makes time-slicing appropriate for workloads such as:

• cooperative internal notebooks;
• development and test jobs with low duty cycles;
• bursty inference where occasional variance is acceptable;
• classroom, lab, and experimentation environments;
• low-cost service tiers that are explicitly best effort.

It is a poor fit for:

• hostile external tenants;
• contractual latency or throughput guarantees;
• workloads that cannot tolerate another tenant’s memory pressure or long kernels;
• environments that need reliable per-container GPU attribution;
• regulated workloads where the control statement implies hardware separation.

Time-slicing can improve aggregate utilization while reducing the platform’s ability to make strong per-tenant guarantees. The service catalog should say that plainly.

Multi-Instance GPU

MIG is spatial partitioning. A supported physical GPU can be divided into GPU Instances with assigned compute and memory resources. NVIDIA documents dedicated resources, memory quality of service, and fault isolation at the MIG GPU Instance boundary [2].

The terminology matters:

• A GPU Instance combines memory slices, compute resources, and engines into a hardware-backed partition.
• A Compute Instance subdivides compute within a GPU Instance. Compute Instances inside the same GPU Instance share the parent’s memory resources and some engines.

For a strong tenant boundary, the design should prefer one tenant per GPU Instance rather than assuming that every Compute Instance is an independent memory-isolation domain.

MIG is well suited to predictable, right-sized service classes such as:

• small inference endpoints with bounded GPU memory;
• development environments that need stronger separation than time-slicing;
• production services that fit a tested MIG profile;
• one-to-one MIG-backed virtual GPUs;
• capacity pools where several tenants can share a board while retaining a defined partition.

MIG does not eliminate every shared dependency. Tenants still share the physical board, host, driver stack, power and thermal envelope, PCIe path, management plane, and certain recovery operations. Reconfiguring MIG geometry may require workload evacuation, device reconfiguration, or a node maintenance action [3]. A board-level event can still affect more than one partition.

MIG is therefore a strong hardware partitioning control, but it is not equivalent to a dedicated node or cluster.

vGPU

The term vGPU must be qualified because it covers different sharing models.

A conventional time-sliced vGPU uses a hypervisor-mediated virtual device and a configured framebuffer profile. It provides a virtual-machine administrative boundary and controlled device presentation, but workloads still share physical execution time. Performance depends on profile, scheduling policy, oversubscription, and competing demand.

A one-to-one MIG-backed vGPU maps a virtual GPU to a MIG instance. That adds the hardware partitioning properties of MIG to the virtual-machine boundary [4], [5].

A MIG-backed time-sliced vGPU adds temporal sharing inside a MIG partition. The MIG partition separates that group from other partitions, but the virtual GPUs inside the same partition still contend with one another.

The platform catalog should therefore avoid a generic “vGPU” service. It should identify at least:

• time-sliced vGPU;
• one-to-one MIG-backed vGPU;
• time-sliced vGPU inside a MIG partition;
• whole-GPU passthrough.

Those service classes have different security, performance, licensing, lifecycle, and evidence implications.

MIG Plus Time-Slicing

MIG and time-slicing can be combined. This is useful when a platform wants a hardware partition between groups and higher density inside each group.

The boundary should be described accurately:

Tenant B receives the isolation properties of its GPU Instance. Tenants A1 and A2 are isolated from Tenant B by the MIG boundary, but they are not isolated from each other in the same way. They share the temporal scheduling and memory behavior exposed inside that partition.

This hybrid model can be useful, but it should not be marketed as if every replica were a separate MIG partition.

Shared Clusters Versus Dedicated Clusters

Kubernetes supports several multi-tenancy patterns, ranging from namespace-based soft multi-tenancy to separate clusters and hardware [6]. The correct pattern depends on tenant trust, administrative delegation, workload privilege, and evidence requirements.

Shared Cluster with Namespaces

A shared cluster is operationally efficient. It centralizes GPU drivers, GPU Operator components, monitoring, admission policy, networking, storage integration, and lifecycle management.

Namespaces can provide:

• namespace-scoped RBAC;
• service accounts and workload identities;
• ResourceQuota and object-count limits;
• LimitRange defaults for ordinary CPU and memory resources;
• NetworkPolicy selection;
• policy labels and admission controls;
• tenant-scoped deployment and secret management.

Namespaces do not isolate cluster-scoped resources such as CustomResourceDefinitions, StorageClasses, many webhooks, nodes, and cluster-wide operators. They also do not protect against a privileged cluster administrator or a compromised component with cluster-scoped authority [6].

A shared cluster is appropriate when tenants are cooperative, the platform team retains cluster administration, workloads are restricted, and the organization accepts a shared control-plane and node trust model.

Virtual Control Plane Per Tenant

A virtual control plane can give each tenant its own Kubernetes API surface while sharing underlying worker infrastructure. This improves API and cluster-scoped administrative separation, but it does not automatically isolate nodes, kernels, device plugins, network data paths, storage back ends, or GPUs.

This pattern is useful when tenants need more autonomy than a namespace provides but the provider still wants shared infrastructure. It should not be confused with dedicated data-plane isolation.

Dedicated Cluster

A dedicated cluster creates a clearer administrative and lifecycle boundary. It is justified when:

• external customers are mutually untrusted;
• tenants require cluster-scoped extensions or administrators;
• regulatory evidence must be produced per tenant;
• maintenance windows or Kubernetes versions differ;
• tenant workloads require privileged pods or host-level integrations;
• the blast radius of an operator, webhook, CNI, CSI, or GPU component must be contained;
• customer contracts require a dedicated control plane.

A dedicated cluster is not necessarily dedicated hardware. If several dedicated control planes still place workloads on the same nodes, much of the data-plane risk remains. The architecture must state whether the separation includes the control plane, worker nodes, GPU devices, network fabric, storage, and management tooling.

Dedicated Node Pool

A dedicated node pool is often the practical middle ground for regulated internal production. It retains a shared cluster control plane while separating GPU generation, memory size, security zone, workload type, and failure domain.

Node pools should be reinforced with:

• taints and tolerations;
• node affinity or required node selectors;
• admission policies that prevent tenants from selecting unauthorized pools;
• separate capacity and scheduling policies;
• dedicated network and storage classes when required;
• controlled maintenance and driver lifecycles.

A label is not a security control if tenants can bypass or alter the placement policy. Admission enforcement is what converts placement intent into a reliable boundary.

Quotas, Limits, Fair Share, and Reclamation

Capacity governance requires several distinct policy values.

Guaranteed Quota

A guaranteed quota is an entitlement under contention. It answers:

When the platform is busy, how much capacity can this tenant expect to receive?

A guarantee does not have to mean permanently idle reserved hardware. A capable scheduler can allow other tenants to borrow unused entitlement and reclaim it when the owner needs it.

Hard Limit

A hard limit is the maximum a tenant may consume. It answers:

Even when the cluster is idle, how much capacity may this tenant use?

The hard limit protects budgets, contracts, and blast radius. It should normally be higher than or equal to the guarantee.

Borrowed Capacity

Borrowed capacity is idle entitlement temporarily used by another tenant. It improves utilization without transferring ownership.

Borrowing is safe only when reclamation behavior is defined. The platform needs to know which workloads can be queued, checkpointed, suspended, or preempted. Production inference and non-checkpointable training should not be treated like disposable development jobs.

Fair Share

Fair share determines how competing tenants divide capacity after guarantees and policy constraints are considered. It prevents a first-come, first-served workload from monopolizing the pool indefinitely.

NVIDIA Run:ai models departments, projects, quotas, over-quota use, fair-share behavior, workload priority, and preemption. Its current scheduler documentation describes over-quota capacity as opportunistic and subject to reclamation [14], [15].

Priority and Preemption

Priority answers which workload should win when capacity is scarce. Preemption is the enforcement mechanism that removes or suspends lower-priority work.

A sound policy distinguishes:

• production inference from interactive notebooks;
• deadline-driven batch work from ordinary experimentation;
• checkpointable training from non-checkpointable training;
• in-quota workload from borrowed over-quota workload;
• security or incident-response work from normal demand;
• guaranteed customer service from best-effort internal use.

Preemption should first target borrowed, explicitly preemptible capacity. A tenant consuming its guarantee should not normally be displaced by a lower-priority tenant merely because the latter submitted first.

Capacity Policy at a Glance

Physical Pool: 16 GPUs

Department A
  Guaranteed: 6
  Maximum:   10

Department B
  Guaranteed: 4
  Maximum:    8

Department C
  Guaranteed: 2
  Maximum:    4

Platform Reserve: 4

Normal idle period:
  A uses 4
  B uses 3
  C uses 1
  Borrowable capacity remains available

Burst period:
  B borrows 3 above guarantee

A demand arrives:
  Scheduler reclaims B's borrowed capacity
  A receives its guaranteed entitlement
  B returns toward its guarantee

The scheduler should record which capacity was guaranteed, borrowed, queued, reclaimed, and preempted. Without that evidence, a showback report cannot explain why a tenant waited or why a workload was stopped.

Namespace Resource Quotas Are Ceilings, Not Guarantees

Kubernetes ResourceQuota constrains aggregate namespace consumption and object counts [7]. For extended resources such as NVIDIA GPUs, quota uses the requests. resource name because those resources are not overcommitted by the native scheduler [7], [8].

ResourceQuota is valuable, but it does not create:

• guaranteed availability on a busy cluster;
• fair-share allocation across namespaces;
• queueing policy;
• preemption or reclamation;
• a physical reservation;
• performance isolation;
• a tenant billing model.

The following example creates a namespace and applies a ceiling for a specific MIG resource profile. The resource name is an example. The platform team must replace it with the exact resource advertised by its device plugin and MIG configuration.

apiVersion: v1
kind: Namespace
metadata:
  name: ai-risk-dev
  labels:
    platform.example.com/tenant: risk
    platform.example.com/environment: development
    platform.example.com/gpu-service-class: mig-shared
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-resource-ceiling
  namespace: ai-risk-dev
spec:
  hard:
    requests.nvidia.com/mig-1g.10gb: "4"
    requests.cpu: "32"
    limits.cpu: "64"
    requests.memory: 128Gi
    limits.memory: 256Gi
    requests.storage: 2Ti
    persistentvolumeclaims: "8"
    pods: "30"

This policy is intended to cap aggregate namespace demand. It does not promise that four MIG instances will always be available.

The platform team should modify:

• the namespace and label values;
• the MIG resource name;
• the GPU quantity;
• CPU, memory, storage, PVC, and object-count limits;
• admission policies that bind the namespace to the correct node pool.

Successful enforcement can be validated by inspecting the quota status and submitting a workload that exceeds the ceiling. Kubernetes should reject the admission request. Common failures include using a MIG resource name that the cluster does not advertise, changing MIG geometry without updating quotas, assuming existing objects will be retroactively corrected, and treating a quota ceiling as reserved capacity.

For whole GPUs, the same pattern can use requests.nvidia.com/gpu. A namespace should normally receive either a defined whole-GPU service class or a defined MIG service class, not an uncontrolled mixture that makes capacity evidence difficult to interpret.

Noisy Neighbors Exist Beyond the GPU

A noisy neighbor is any tenant whose behavior degrades another tenant’s service. GPU utilization is only one source.

Shared dependency	Typical noisy-neighbor symptom	Stronger control
GPU execution	Variable latency, queue buildup, reduced tokens per second	MIG, whole GPU, concurrency and admission limits
GPU memory or bandwidth	Allocation failure, model eviction, lower throughput	MIG profile, dedicated GPU, model memory budget
CPU	Tokenization or preprocessing delay, low GPU utilization	CPU requests and limits, dedicated node pool
Host memory	Eviction, pinned-memory pressure, unstable data loading	Memory quotas, node sizing, workload class
PCIe or interconnect	Reduced transfer rate, scaling inefficiency	Topology-aware placement, dedicated node or fabric
Network	Slow model download, collective communication delay, inference tail latency	QoS, segmentation, dedicated path, telemetry
Storage	Slow checkpointing, data starvation, cache thrashing	Tenant storage class, throughput limits, local scratch policy
Model server	Queue starvation, KV-cache pressure, unfair batching	Per-tenant queue, rate limit, separate endpoint
Control plane	API throttling, webhook latency, object explosion	Object quotas, API controls, separate control plane
Metrics pipeline	Cardinality explosion, dropped telemetry	Label allow-list, tenant metric budgets
Power and thermals	Clock reduction and correlated performance loss	Headroom, board-level monitoring, dedicated hardware
Device fault domain	Several tenants disrupted by reset or board failure	Dedicated GPU, node, cluster, or availability zone

The operating implication is important: a tenant can remain within its GPU quota and still harm neighbors through CPU, storage, network, object count, or model-server behavior.

A GPU service class therefore needs a complete resource envelope. GPU quantity alone is not a service definition.

Model and Data Isolation Must Follow the Tenant

Compute isolation does not automatically protect the model, data, or outputs.

Workload Identity and Authorization

Each workload should run under a tenant-scoped service account with only the permissions required for its model endpoint, data source, queue, registry, secrets, and telemetry. Kubernetes service accounts are namespace-scoped identities, but their effective isolation depends on the RBAC bindings granted to them [10], [11].

Avoid shared human credentials, shared service accounts, and wildcard access to cluster secrets. Separate the identities used for:

• image and model retrieval;
• data access;
• inference requests;
• observability export;
• deployment automation;
• backup and recovery;
• key release and attestation.

Model Registry and Cache Isolation

Model approval and compute allocation are separate controls. A tenant should have a defined registry project, artifact permissions, promotion path, and immutable digest policy.

Shared model caches need particular attention. They can improve cold-start time, but they can also create data-remanence and authorization problems. The platform should define:

• whether cache entries are public, shared-approved, or tenant-private;
• which identity populates and reads the cache;
• how model digests and licenses are verified;
• how tenant-private artifacts are encrypted;
• how cache eviction and secure deletion are handled;
• whether adapters, fine-tuned weights, and prompt assets may coexist with base models.

External customers should not share a writable model-cache namespace.

Data, RAG, and Vector Isolation

Tenant data boundaries should extend to:

• object-storage buckets or prefixes;
• persistent volumes and snapshots;
• vector database collections or indexes;
• feature stores and experiment tracking;
• training checkpoints and temporary scratch data;
• prompt logs and model outputs;
• backup copies and disaster-recovery replicas.

Use separate identities and, where risk justifies it, separate encryption keys. Kubernetes API encryption protects selected API objects at rest, but application data mounted from storage needs storage-layer or application-layer encryption [13].

Network Segmentation

Kubernetes allows pod-to-pod communication by default unless the network implementation and policy restrict it. NetworkPolicy can create Layer 3 and Layer 4 controls, but enforcement depends on a compatible CNI [6], [9].

A tenant baseline should normally include:

• default-deny ingress;
• default-deny egress;
• explicit DNS access;
• explicit access to approved model registries, data stores, telemetry, and identity services;
• controlled north-south ingress;
• restricted metadata and management endpoints;
• separate load balancers, addresses, or gateways when customer evidence requires them;
• service-mesh identity and mutual TLS when Layer 7 policy is justified.

Default deny without a tested allow-list is not a finished policy. It is an outage waiting to happen.

Storage Isolation

A namespace-scoped PersistentVolumeClaim does not make the underlying storage system tenant-private. PersistentVolumes and StorageClasses are cluster-scoped resources. The CSI driver, storage administrator, snapshot system, and backup platform may all remain shared [6].

Higher-risk tenants may require:

• dedicated StorageClasses or storage policy;
• per-tenant storage credentials;
• encryption keys separated by tenant;
• explicit reclaim and secure-reuse behavior;
• snapshot and backup access controls;
• capacity and throughput limits;
• separate RWX shares rather than one shared export;
• retention and deletion evidence.

The same policy must cover local NVMe scratch space. Deleting a pod is not the same as proving that sensitive scratch data was removed.

Confidential Computing Is a Cryptographic Boundary

Confidential computing addresses a specific problem: reducing the need to trust privileged infrastructure administrators with plaintext data and model state while a workload is running.

A confidential GPU workflow typically combines:

• a confidential virtual machine or protected execution environment;
• measured launch and signed artifacts;
• CPU and GPU attestation evidence;
• encrypted model and data artifacts;
• a key broker or key management service;
• policy that releases decryption keys only to an approved measured environment;
• protected CPU memory and GPU confidential-computing mode;
• restricted in-guest administration and secrets handling.

This boundary protects against a class of infrastructure threats that namespaces, MIG, and dedicated GPUs do not address. A dedicated GPU on an ordinary host can still leave host administrators inside the trust boundary. Confidential computing can remove or reduce that trust when the implementation, attestation policy, and key authorities are sound.

What Confidential Computing Does Not Solve

NVIDIA’s current threat-model documentation is explicit that confidential computing does not automatically protect against:

• malicious or vulnerable code inside the confidential virtual machine;
• an untrusted guest image that is nevertheless approved incorrectly;
• application logging of plaintext;
• compromised attestation or key-release administrators;
• denial of service;
• all side channels;
• sophisticated physical attacks;
• scheduler unfairness or noisy neighbors;
• network, storage, or identity policy mistakes outside the protected workload.

Availability is especially important. Encryption does not stop another actor from starving a queue, exhausting a shared resource, resetting a device, or disrupting a host.

The Current Multi-Tenant Support Boundary

As of July 2026, NVIDIA’s published Confidential Containers reference architecture describes single-GPU and multi-GPU passthrough patterns in which the GPUs on a host are assigned to one confidential virtual machine [18], [19]. NVIDIA’s Secure AI whitepaper likewise describes whole physical GPU passthrough as exclusive to one confidential virtual machine [21].

That is not the same as several confidential tenants sharing one GPU.

The practical conclusion is conservative:

• do not assume that MIG plus confidential computing creates a generally supported multi-tenant confidential GPU service;
• do not use a community discussion as the production support matrix;
• require a current vendor-supported topology, exact hardware and firmware baseline, attestation design, and validated failure test;
• treat unsupported combinations as architecture research, not a billable service tier.

The community question about confidential multi-tenancy on a single GPU is useful because it exposes the gap [23]. It should not be used as evidence that the gap is closed.

Per-Tenant Monitoring and Audit Evidence

A tenancy design is incomplete until the platform can prove who used which resource, under which policy, and with what outcome.

Minimum Identity Dimensions

Every telemetry record should carry enough context to map technical activity to an accountable tenant:

• organization or customer ID;
• department and project;
• environment;
• Kubernetes cluster and namespace;
• service account or workload identity;
• workload, pod, job, or virtual-machine ID;
• node and physical GPU UUID;
• MIG GPU Instance and Compute Instance identifiers where applicable;
• vGPU profile or passthrough assignment;
• model name, version, and immutable digest;
• scheduler queue, priority, quota state, and preemptibility;
• billing or showback code.

Do not place sensitive customer names directly into high-cardinality labels when a stable pseudonymous identifier will work.

GPU and Platform Metrics

NVIDIA DCGM and DCGM Exporter can provide device health and utilization telemetry, including MIG-aware metrics when configured appropriately [16], [17]. A useful tenant dashboard should include:

• allocated GPU or MIG capacity;
• active GPU utilization;
• GPU memory used and available;
• streaming multiprocessor activity;
• memory bandwidth indicators;
• power, temperature, and clock behavior;
• ECC, Xid, reset, and health events;
• queue time and scheduling delay;
• preemption and reclamation events;
• inference requests, tokens per second, time to first token, and latency;
• model load time and cache hit behavior;
• CPU, host memory, network, and storage dependencies.

The platform must distinguish allocated from actively used capacity. An idle but guaranteed production GPU may be a deliberate service decision. A borrowed GPU that appears idle may be reclaimable. A time-sliced GPU with high aggregate utilization may still provide poor per-tenant latency.

Time-Slicing Attribution Limitation

NVIDIA documents an important DCGM Exporter limitation for time-sliced GPUs: metrics cannot be associated with containers through the device-plugin time-slicing path [1].

That limitation affects:

• per-tenant showback;
• performance troubleshooting;
• noisy-neighbor investigations;
• contractual evidence;
• capacity forecasting;
• automated remediation.

A provider that cannot reliably attribute activity should not promise precise per-tenant usage billing for that service class. It may need model-server metrics, request-gateway telemetry, process-level observability, or a different GPU allocation method.

Audit Evidence

Kubernetes auditing provides a chronological record of API activity and should be correlated with scheduler, device, network, storage, registry, and key-management evidence [12].

For each tenant, retain evidence for:

• namespace and RBAC changes;
• quota and limit changes;
• admission decisions;
• workload submissions and deletions;
• node and GPU placement;
• borrowed-capacity use;
• preemption and reclamation;
• network-policy changes and flow events;
• model and container digest promotion;
• storage, snapshot, and backup access;
• attestation verdicts and key-release decisions;
• privileged administrative actions;
• GPU health incidents and resets.

Retention should follow legal, regulatory, contractual, and incident-response requirements. The architecture should identify an evidence owner and prove that one tenant cannot alter another tenant’s audit history.

Tenancy Patterns for Different Risk Classes

A single platform may offer several service classes, but each class needs a clear threat model and service promise.

Cooperative Internal Research

Tenant relationship: Employees and approved contractors inside one enterprise trust domain.

Recommended pattern: Shared Kubernetes cluster, namespace per project or environment, namespace-scoped RBAC, restricted workloads, ResourceQuota, fair-share scheduler, default-deny network policy, tenant-scoped storage and registry access.

GPU service: Time-slicing for notebooks and low-duty-cycle experiments; MIG for teams needing more predictable memory and compute; whole GPU available for large jobs.

Capacity policy: Modest guarantee, higher hard limit, borrowing allowed, development work preemptible.

Evidence: Basic showback, queue time, preemption history, GPU utilization, model-server metrics.

Primary warning: Do not let “internal” become a reason to permit privileged pods, shared credentials, unrestricted egress, or unbounded object creation.

Regulated Internal Production

Tenant relationship: Internal application team, but the data, model, or business process has regulatory or material business impact.

Recommended pattern: Dedicated namespace plus dedicated GPU node pool at minimum. Use a dedicated cluster when cluster-scoped authority, lifecycle independence, or evidence separation is required. Separate production identities, network paths, storage policies, encryption keys, model registry projects, backups, and monitoring.

GPU service: Whole GPU or tested MIG GPU Instance. Avoid cross-tenant time-slicing for latency-sensitive or sensitive workloads.

Capacity policy: Guaranteed capacity, bounded maximum, controlled burst, production priority, no opportunistic preemption of the protected workload.

Confidential computing: Use when infrastructure administrators, hosting operators, or other privileged roles must be excluded from plaintext data and model state.

Evidence: Control changes, allocation, model digest, data access, GPU health, SLOs, attestation and key-release logs when applicable.

Primary warning: A namespace in a general-purpose cluster may not satisfy the audit or administrator-separation requirement.

External Paid Customer

Tenant relationship: Separate legal entity with contractual service, security, and billing expectations.

Recommended pattern: Dedicated cluster or dedicated virtual control plane with dedicated worker and GPU pools. Use a shared cluster only when the provider has designed for hostile multi-tenancy, restricted every privileged path, validated the CNI and CSI boundaries, and stated the residual shared risks contractually.

GPU service: Whole GPU, one-to-one MIG-backed vGPU, or dedicated MIG GPU Instance. A time-sliced service may be offered as a clearly labeled best-effort tier, not as equivalent to a hardware-isolated tier.

Capacity policy: Contractual guarantee, hard maximum, measured burst, defined reclamation rules, explicit SLO and maintenance terms.

Evidence: Customer-specific usage, allocation, performance, change, incident, and billing records. Provide evidence of isolation controls without exposing other tenants’ identities or telemetry.

Primary warning: A customer contract can turn an ambiguous technical claim into a legal liability. Define memory, performance, fault, administrative, and cryptographic isolation separately.

High-Assurance or Mutually Distrustful Workloads

Tenant relationship: External, sovereign, highly regulated, intellectual-property sensitive, or explicitly hostile.

Recommended pattern: Dedicated cluster, nodes, GPUs, network segment, storage boundary, management roles, and evidence pipeline. Use whole-GPU passthrough. Add confidential computing when privileged infrastructure operators are outside the trust boundary.

Capacity policy: No cross-tenant borrowing on the protected hardware unless policy explicitly allows a trusted workload to use it and guarantees secure reclamation.

Evidence: Measured launch, attestation, key release, immutable logs, hardware and firmware baseline, recovery tests, chain of custody, and independent review.

Primary warning: Confidential computing does not make shared failure domains or denial-of-service risk disappear. Dedicated failure domains may still be required.

Risk-Tiered Tenancy Decision Matrix

Risk tier	Tenant and workload pattern	Cluster and administrative boundary	GPU service class	Capacity policy	Confidential computing and evidence
Cooperative sandbox	Trusted internal notebooks, labs, low-risk experiments	Shared cluster, namespace per project, restricted RBAC	Time-slicing or small MIG profile	Best effort, low guarantee, borrowing and preemption allowed	Usually not required; basic audit and showback
Governed internal	Internal development, fine-tuning, batch, shared services	Shared cluster with enforced namespaces and node pools	MIG GPU Instance, whole GPU for large jobs	Department guarantee, project limits, fair share, reclaimable over-quota use	Add only for a defined privileged-admin threat; retain allocation and policy evidence
Production regulated	Sensitive internal production and material business services	Dedicated node pool or dedicated cluster; separate data and network controls	Whole GPU, tested MIG GPU Instance, one-to-one MIG-backed vGPU	Protected guarantee, controlled burst, production priority, limited preemption	Use when administrators are out of trust; retain SLO, access, change, model, and key evidence
External paid customer	Contracted customer with SLA and billing boundary	Prefer dedicated cluster or control plane plus dedicated workers; hardened shared cluster only by exception	Whole GPU, dedicated MIG GPU Instance, one-to-one MIG-backed vGPU	Contractual guarantee and maximum; explicit burst and reclamation terms	Customer-specific audit and usage evidence; CC based on admin trust and product support
High assurance	Mutually distrustful, sovereign, high-value model or data	Dedicated cluster, nodes, GPUs, network, storage, operators, and evidence	Whole-GPU passthrough	No unapproved borrowing; dedicated reserve and recovery capacity	Confidential VM and GPU CC with attestation and key release; independent validation

This matrix is a starting point, not a substitute for workload-specific risk review. A low-sensitivity model with a strict latency SLO may need dedicated hardware for performance rather than confidentiality. A batch workload with highly sensitive data may need confidential computing even when it tolerates queue delay.

Decision Rules That Prevent Security Theater

The following rules keep the service catalog honest.

Scheduler Separation Is Not Memory Isolation

Quotas and fair share decide entitlement. They do not stop one time-sliced workload from sharing the same physical memory system or fault domain.

Namespace Separation Is Not a Hostile-Tenant Boundary

Namespaces are valuable logical units. They do not isolate cluster-scoped resources, privileged administrators, the node kernel, GPU drivers, CNI, CSI, or hardware.

MIG Is Not Administrative Isolation

MIG partitions GPU resources. It does not create separate cluster administrators, driver lifecycles, network policies, storage systems, or key authorities.

Dedicated Hardware Is Not Cryptographic Isolation

A dedicated GPU or node removes neighbors, but an ordinary host administrator may remain capable of inspecting or modifying the workload environment.

Confidential Computing Is Not Availability Isolation

Attestation and protected memory do not guarantee queue position, latency, throughput, device health, or resistance to denial of service.

A Quota Is Not a Reservation

Kubernetes ResourceQuota limits admitted demand. Guaranteed capacity requires a scheduler and enough physical capacity to honor the promise under failure and maintenance conditions.

A Service Class Must Name Its Residual Risks

“Shared GPU,” “isolated GPU,” and “secure GPU” are insufficient product descriptions. The catalog should state:

• allocation method;
• memory boundary;
• performance expectation;
• fault domain;
• administrative trust boundary;
• data and network controls;
• cryptographic protection;
• quota, burst, and reclamation policy;
• telemetry and evidence available;
• unsupported combinations.

Reference Telemetry and Control Architecture

A governed AIaaS platform can combine the controls into one accountable flow.

The evidence plane should be designed with the same care as the workload plane. It is what allows the provider to prove that policy was enforced, a service level was met, borrowed capacity was reclaimed correctly, and a security event did not cross tenant boundaries.

Implementation Checklist

Before opening the platform to another tenant, complete the following checks.

Tenancy and Threat Model

• Define whether the tenant is an organization, department, project, environment, customer, or combination.
• Identify the assets, trusted roles, untrusted roles, privileged administrators, and acceptable shared failure domains.
• Separate confidentiality, integrity, availability, performance, economic, and administrative requirements.
• Document conditions that require a dedicated cluster, node, GPU, network, storage system, or cryptographic boundary.

Service Catalog

• Publish distinct whole-GPU, MIG, vGPU, and time-sliced service classes.
• State the exact memory, compute, performance, fault, administrative, and cryptographic properties of each class.
• Identify whether the service is guaranteed, burstable, borrowed, reclaimable, or best effort.
• Define supported GPU profiles, node pools, runtime versions, and maintenance behavior.
• Prohibit unsupported combinations from self-service catalogs.

Kubernetes and Platform Controls

• Create namespace, RBAC, service account, admission, ResourceQuota, and object-count policies.
• Enforce node-pool placement with admission controls, not labels alone.
• Apply Pod Security controls and restrict privileged, host-network, host-path, and device access.
• Use default-deny network policies with tested allowances.
• Separate storage, registry, secrets, model, cache, backup, and telemetry access.
• Limit cluster-scoped permissions and review operators, webhooks, CRDs, CNI, CSI, and GPU components.

Capacity and Scheduling

• Define guarantee, hard maximum, fair-share weight, priority, and preemptibility for each department and project.
• Reserve capacity for failures, upgrades, and protected production demand.
• Test queueing, borrowing, reclamation, and preemption under realistic contention.
• Verify checkpoint and restart behavior before marking training workloads preemptible.
• Ensure customer billing distinguishes guaranteed, borrowed, allocated, and actively used capacity.

GPU Isolation

• Validate the actual device-plugin resources exposed by each node.
• Verify MIG geometry, GPU Instance ownership, and reconfiguration runbooks.
• Distinguish MIG GPU Instances from Compute Instances inside a shared GPU Instance.
• Validate vGPU mode and whether the profile is time-sliced or one-to-one MIG-backed.
• Test reset, Xid, thermal, power, and node-failure behavior.
• Confirm that time-sliced services do not claim MIG-style isolation.

Confidential Computing

• Verify current supported hardware, firmware, drivers, guest image, operator, and attestation components.
• Define the measured artifacts and acceptable attestation claims.
• Separate attestation administration from key-release administration where feasible.
• Prove that keys are withheld from an unapproved measurement.
• Test recovery, patching, certificate rotation, and evidence retention.
• Do not deploy single-GPU confidential multi-tenancy without an explicit supported topology and validation result.

Monitoring and Audit

• Correlate tenant, namespace, workload, node, GPU UUID, MIG identifiers, model digest, and scheduler state.
• Monitor CPU, memory, storage, network, power, thermal, clock, health, and model-server metrics in addition to GPU utilization.
• Record quota changes, borrowing, reclamation, preemption, resets, and health events.
• Account for time-slicing attribution limitations.
• Protect the audit pipeline from tenant modification and define evidence retention.
• Run periodic isolation and noisy-neighbor tests, not only configuration reviews.

Conclusion

Safe GPU multi-tenancy is not achieved by selecting one sharing feature.

The platform needs a layered control model that begins with the tenant relationship and threat model. Namespaces and RBAC establish a logical administrative boundary. Quotas, fair share, priority, and preemption establish capacity policy. Time-slicing, vGPU, MIG, and whole-GPU assignment establish different device-sharing properties. Network, storage, identity, model, and data controls keep the rest of the AI service inside the tenant boundary. Confidential computing adds a cryptographic boundary when privileged infrastructure administrators must be excluded from plaintext data and model state.

The practical design principle is simple: use the control that matches the threat, then keep the other controls that solve different threats.

A scheduler cannot substitute for memory isolation. MIG cannot substitute for a dedicated control plane. Dedicated hardware cannot substitute for attestation and protected memory. Confidential computing cannot substitute for availability, application security, or fair scheduling.

A credible GPUaaS or AIaaS service catalog should make those distinctions visible. It should state what is shared, what is isolated, what is guaranteed, what is best effort, what evidence is retained, and which risks remain. That clarity is what turns GPU sharing from security theater into an operable enterprise service.

External References

• [1] NVIDIA: Time-Slicing GPUs in Kubernetes
  Canonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html
• [2] NVIDIA: Concepts, NVIDIA Multi-Instance GPU User Guide
  Canonical URL: https://docs.nvidia.com/datacenter/tesla/mig-user-guide/latest/concepts.html
• [3] NVIDIA: GPU Operator with MIG
  Canonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html
• [4] NVIDIA: Deployment, NVIDIA Virtual GPU FAQ
  Canonical URL: https://docs.nvidia.com/vgpu/faq/latest/deployment.html
• [5] NVIDIA: MIG-Backed vGPU
  Canonical URL: https://docs.nvidia.com/ai-enterprise/release-8/latest/infra-software/vgpu/features/mig-backed-vgpu.html
• [6] Kubernetes: Multi-tenancy
  Canonical URL: https://kubernetes.io/docs/concepts/security/multi-tenancy/
• [7] Kubernetes: Resource Quotas
  Canonical URL: https://kubernetes.io/docs/concepts/policy/resource-quotas/
• [8] Kubernetes: Device Plugins
  Canonical URL: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/
• [9] Kubernetes: Network Policies
  Canonical URL: https://kubernetes.io/docs/concepts/services-networking/network-policies/
• [10] Kubernetes: Role Based Access Control Good Practices
  Canonical URL: https://kubernetes.io/docs/concepts/security/rbac-good-practices/
• [11] Kubernetes: Service Accounts
  Canonical URL: https://kubernetes.io/docs/concepts/security/service-accounts/
• [12] Kubernetes: Auditing
  Canonical URL: https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/
• [13] Kubernetes: Encrypting Confidential Data at Rest
  Canonical URL: https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/
• [14] NVIDIA Run:ai: The NVIDIA Run:ai Scheduler, Concepts and Principles
  Canonical URL: https://run-ai-docs.nvidia.com/saas/platform-management/runai-scheduler/scheduling/concepts-and-principles
• [15] NVIDIA Run:ai: Over Quota, Fairness and Preemption
  Canonical URL: https://run-ai-docs.nvidia.com/saas/platform-management/runai-scheduler/scheduling/quick-starts/over-quota
• [16] NVIDIA: Feature Overview, NVIDIA DCGM Documentation
  Canonical URL: https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html
• [17] NVIDIA: DCGM-Exporter
  Canonical URL: https://docs.nvidia.com/datacenter/dcgm/latest/gpu-telemetry/dcgm-exporter.html
• [18] NVIDIA: NVIDIA Confidential Containers Reference Architecture
  Canonical URL: https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/overview.html
• [19] NVIDIA: Supported Platforms, NVIDIA Confidential Containers Architecture
  Canonical URL: https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/supported-platforms.html
• [20] NVIDIA: Trust and Threat Model, Deploying Proprietary Models Securely with NVIDIA Confidential Computing on Self-Hosted Virtual Machines
  Canonical URL: https://docs.nvidia.com/enterprise-reference-architectures/deploying-proprietary-models-confidential-compute-self-hosted-vms/latest/trust-and-threat-model.html
• [21] NVIDIA: NVIDIA Secure AI with Blackwell and Hopper GPUs
  Canonical URL: https://docs.nvidia.com/nvidia-secure-ai-with-blackwell-and-hopper-gpus-whitepaper.pdf
• [22] NVIDIA Developer Forums: GPU Operator MIG feature ResourceQuota
  Canonical URL: https://forums.developer.nvidia.com/t/gpu-operator-mig-feature-resourcequota/255634
• [23] NVIDIA Developer Forums: Multi-tenant support in NVIDIA confidential computing (CC) on a single GPU
  Canonical URL: https://forums.developer.nvidia.com/t/multi-tenant-support-in-nvidia-confidential-computing-cc-on-a-single-gpu/363235
• [24] NVIDIA Developer Forums: How to use GPU Operator with MIG to configure 2 GPUs on one node separately
  Canonical URL: https://forums.developer.nvidia.com/t/how-to-use-gpu-operator-with-mig-to-configure-2-gpus-on-one-node-separately/306251