Site icon Digital Thought Disruption

How to Configure Multi-Tenant GPU Scheduling with NVIDIA Run:ai

Paul Bryant

TL;DR

NVIDIA Run:ai can turn a shared Kubernetes GPU cluster into a governed multi-tenant platform by organizing workloads into departments and projects, assigning guaranteed GPU quotas per node pool, and allowing controlled over-quota use when capacity would otherwise remain idle.

The design depends on four controls working together:

Production workloads should normally run from quota-backed projects as non-preemptible workloads. Development and experimental workloads can use preemptible over-quota capacity, provided they support checkpointing and tolerate interruption.

The objective is not to keep every GPU permanently allocated. It is to make allocation predictable, measurable, recoverable, and aligned with business ownership.

Introduction

A Kubernetes cluster can expose GPUs as schedulable resources, but that does not automatically make it a multi-tenant AI platform.

Without an additional governance layer, the team that submits first, requests the most GPUs, or leaves the most workloads running can consume a disproportionate amount of the cluster. Other teams may have technically valid workloads but no predictable path to capacity.

NVIDIA Run:ai addresses this problem by placing organizational and scheduling controls above the raw Kubernetes resource model. Departments, projects, node pools, quotas, workload policies, priority, and preemption become part of one allocation system.

The difficult part is not creating a few projects and assigning arbitrary GPU numbers. The difficult part is deciding:

This tutorial builds a practical configuration model that answers those questions.

What You Will Build

By the end of the tutorial, you will have a design for:

The examples assume a shared Kubernetes cluster containing multiple GPU types. The same governance model can be scaled down for a smaller environment or extended across a larger NVIDIA Run:ai deployment.

Prerequisites and Assumptions

Before configuring tenant scheduling, confirm the following:

This tutorial does not cover the initial NVIDIA Run:ai installation, Kubernetes GPU enablement, MIG configuration, or time-slicing configuration. Those capabilities should be operational before tenant quotas are introduced.

Understand the Run:ai Scheduling Hierarchy

The administrative hierarchy and the scheduling hierarchy are related, but they are not exactly the same thing.

Departments and projects are visible organizational objects. Queues are part of the scheduler’s allocation process. A queue is not usually another tenant container that administrators must create for every team.

The scheduler evaluates project and department demand for each applicable node pool.

The following diagram shows the relationship.

There are several important operational implications.

First, quota is evaluated in the context of a node pool. A project can have a guaranteed allocation on one GPU class and only best-effort access to another.

Second, a project’s workload priority affects the ordering of work in its scheduling context. Priority should not be treated as a substitute for cross-team quota design.

Third, unused capacity can be distributed as over-quota capacity. This increases utilization, but it does not convert borrowed capacity into a permanent entitlement.

Fourth, capacity reclamation depends on workloads being preemptible. A non-preemptible workload should fit within the project’s available deserved quota.

Map the Organization Before Creating Projects

The first configuration task should happen outside the platform.

Create a mapping of business ownership, technical ownership, workload purpose, and resource requirements before creating departments or projects. Otherwise, the platform structure tends to reproduce inconsistent team names and short-lived organizational charts.

Decide What a Department Represents

A department should represent a durable governance or ownership boundary.

Good department candidates include:

A department is useful when several related projects should share:

Do not create a department for every temporary experiment. That creates unnecessary administrative depth and makes quota ownership difficult to maintain.

Decide What a Project Represents

A project is the practical scheduling and access boundary for workloads.

A project can represent:

Projects are where administrators apply resource quotas, node-pool preferences, access controls, and workload policies.

Avoid using one project for an entire department if its workloads have different availability, security, or preemption requirements.

Separate Environment and Workload Intent

Development and production should not share a project merely because the same team owns both.

A useful starting structure is:

DepartmentProjectPurposeScheduling intent
Product AIproduct-ai-devInteractive development and test trainingSmall guarantee, preemptible over-quota allowed
Product AIproduct-ai-prodProduction inference and release trainingGuaranteed capacity, non-preemptible
Applied Researchfoundation-researchLong-running model experimentsModerate guarantee, preemptible over-quota
Shared Platformai-platform-servicesMonitoring, controllers, and shared servicesSmall reserved guarantee
Shared Platformperformance-validationNCCL, benchmark, and validation jobsScheduled windows or controlled best effort

This structure makes operational intent visible before a workload is submitted.

Build a Source-Controlled Tenant Plan

NVIDIA Run:ai objects can be configured through supported interfaces, but the approved allocation model should also exist in source control.

The following YAML is a governance worksheet. It is not presented as a native Run:ai import format. Its purpose is to give the platform team a reviewable source of truth before applying configuration through the user interface or API.

organization:
  departments:
    - name: product-ai
      owner_group: grp-product-ai-platform
      node_pool_quotas:
        h100-production:
          guaranteed_gpus: 6
          allow_over_quota: false
          max_gpus: 6
        a100-shared:
          guaranteed_gpus: 6
          allow_over_quota: true
          over_quota_weight: medium
          max_gpus: 10

      projects:
        - name: product-ai-dev
          access_group: grp-product-ai-developers
          node_pool_quotas:
            h100-production:
              guaranteed_gpus: 0
              allow_over_quota: false
              max_gpus: 0
            a100-shared:
              guaranteed_gpus: 2
              allow_over_quota: true
              over_quota_weight: medium
              max_gpus: 6
          default_preemptibility: preemptible
          default_priority: medium-low

        - name: product-ai-prod
          access_group: grp-product-ai-production
          node_pool_quotas:
            h100-production:
              guaranteed_gpus: 6
              allow_over_quota: false
              max_gpus: 6
            a100-shared:
              guaranteed_gpus: 4
              allow_over_quota: false
              max_gpus: 4
          default_preemptibility: non-preemptible
          default_priority: very-high

    - name: applied-research
      owner_group: grp-applied-research
      node_pool_quotas:
        h100-production:
          guaranteed_gpus: 2
          allow_over_quota: true
          over_quota_weight: low
          max_gpus: 4
        a100-shared:
          guaranteed_gpus: 8
          allow_over_quota: true
          over_quota_weight: high
          max_gpus: 14

      projects:
        - name: foundation-research
          access_group: grp-foundation-models
          node_pool_quotas:
            h100-production:
              guaranteed_gpus: 2
              allow_over_quota: true
              over_quota_weight: low
              max_gpus: 4
            a100-shared:
              guaranteed_gpus: 8
              allow_over_quota: true
              over_quota_weight: high
              max_gpus: 14
          default_preemptibility: preemptible
          default_priority: medium

The values that matter most are:

Successful implementation means the live Run:ai configuration matches this approved model and any variance has a recorded exception.

Configure Node Pools Around Infrastructure Characteristics

Node pools group Kubernetes nodes using labels. They are useful for separating heterogeneous hardware and controlling where workloads are entitled to run.

Common node-pool dimensions include:

A practical example might include:

Node poolTypical hardwarePrimary use
h100-productionH100 systems with validated high-speed fabricProduction inference and critical distributed training
a100-sharedA100 systemsGeneral training, research, and burst capacity
l40s-interactiveL40S systemsWorkspaces, visualization, development, and smaller inference
validationRepresentative GPU nodesUpgrade testing, NCCL validation, and platform qualification

Avoid a Node Pool for Every Team

Node pools should normally describe infrastructure characteristics, not mirror the tenant hierarchy.

Creating one node pool per team can lead to:

Use departments and projects to represent ownership. Use node pools to represent hardware or placement characteristics.

Apply Stable Node Labels

Choose labels that describe characteristics unlikely to change during ordinary operations.

For example:

runai.node-pool=h100-production
gpu.platform=nvidia-h100
service.class=production
fabric.class=high-bandwidth

The exact label convention should match your platform standards. Do not rely on ad hoc labels added by individual application teams.

Review Default Access to New Pools

A newly created node pool may appear in project and department configuration with zero guaranteed GPU quota while still permitting over-quota use, depending on the effective settings.

After creating a pool:

This review is particularly important for high-value GPU pools.

Create Departments and Assign Parent Quotas

Create departments after the node-pool design is stable.

For each department:

Make Parent Quotas Internally Consistent

A department quota constrains the projects beneath it.

If project guarantees under a department total 12 GPUs on a node pool but the department has only 8 GPUs of deserved quota, all project guarantees cannot be honored simultaneously.

Use this validation:

Project quotas can still be shaped differently when not all projects require simultaneous guarantees, but that should be a deliberate capacity decision rather than an accidental oversubscription.

Reserve Capacity for Shared Platform Functions

Do not allocate the complete physical pool to user departments without considering platform services.

Capacity may be required for:

A small platform-services department or project makes this capacity visible and prevents it from being treated as permanently free.

Create Projects and Assign Guaranteed Quotas

Each project should receive quota according to the service it is expected to provide.

Production Project Pattern

A production project commonly uses:

Production should not depend on opportunistic capacity for its minimum service objective.

Development Project Pattern

A development project commonly uses:

This gives developers useful access without allowing interactive experiments to displace production services.

Research Project Pattern

Research often needs a hybrid model:

The guarantee protects ongoing research. Over-quota access lets the team use idle GPUs without turning temporary availability into a permanent allocation.

Configure Users, Groups, Roles, and Access Rules

Run:ai access control should be integrated with the enterprise identity model.

An access rule combines:

Use groups instead of assigning large numbers of users individually.

SubjectScopeSuggested responsibility
Central AI platform administratorsOrganization or clusterPlatform configuration, node pools, scheduling policy, and support
Department owner groupDepartmentDepartment visibility, project governance, and quota requests
Project administrator groupProjectProject workload administration and local assets
Developer groupDevelopment projectSubmit and manage development workloads
Production operator groupProduction projectOperate approved production workloads
CI/CD service accountSpecific projectSubmit controlled automated workloads
Audit or FinOps groupOrganization or departmentRead-only usage and allocation review

Apply Least Privilege by Scope

A developer who only needs access to product-ai-dev should not receive a broad role at the Product AI department scope.

A project administrator should not automatically receive permission to:

Run:ai RBAC and Kubernetes RBAC are related operationally, but they should still be reviewed as part of one access model. Direct Kubernetes access must not become a bypass around platform governance.

Use Service Accounts for Automation

Use dedicated service accounts for:

Do not use a human administrator’s credentials in pipelines.

Each service account should have:

Configure Controlled Over-Quota Use

Guaranteed quota and maximum allocation solve different problems.

Guaranteed quota answers:

What capacity is this tenant entitled to receive?

Maximum allocation answers:

What is the largest amount of capacity this tenant may consume?

Over-quota configuration answers:

May this tenant temporarily borrow unused capacity between those boundaries?

A well-designed project can therefore have:

Guaranteed quota: 2 GPUs
Maximum allocation: 8 GPUs
Over-quota: Enabled
Preemptibility: Required for borrowed capacity

The team can always make progress on two GPUs. It can consume up to eight GPUs when capacity is unused. The extra six GPUs remain reclaimable.

Use Over-Quota Weights Deliberately

When over-quota weights are enabled, they influence how unused capacity is divided between eligible departments or projects.

Use weights to represent business preference, not political importance.

For example:

TenantGuaranteed quotaOver-quota weightInterpretation
Production release validation2HighPrefer this workload when temporary capacity is available
General model research4MediumReceive a normal share of excess capacity
Personal experiments0LowBest-effort access only

Do not give every project a high weight. When every tenant is exceptional, the weights no longer express meaningful policy.

Treat Zero Quota Carefully

A project with zero guaranteed quota and over-quota enabled can be useful for:

It has no guaranteed entitlement. Its workloads may remain pending or be preempted when quota-backed demand appears.

That limitation should be communicated to users before the project is offered as a service.

Configure Fairness, Priority, and Preemption

Quota, fairness, priority, and preemption are separate controls.

Fairness Controls Cross-Tenant Sharing

Run:ai calculates fairshare for departments and projects per node pool.

Conceptually:

Fairshare =
Guaranteed deserved quota
+
Eligible share of unused over-quota resources

This prevents the first tenant to consume idle GPUs from treating those resources as permanently owned.

Priority Controls Scheduling Order

Workload priority controls which workload should be considered ahead of another workload in the relevant project queue.

Current Run:ai CLI priority values include:

Changing priority does not automatically change preemptibility. Configure both fields explicitly.

Priority should reflect workload urgency and service intent:

WorkloadSuggested priority
Personal experimentvery-low or low
Development workspacemedium-low
Routine trainingmedium
Release validationmedium-high or high
Production batch operationhigh
Latency-sensitive production inferencevery-high

Do not mark ordinary development jobs as very-high merely to shorten queue time. That destroys the priority model.

Preemptibility Controls Reclamation

A preemptible workload may use opportunistic capacity and may be interrupted when higher-entitlement demand requires those resources.

A non-preemptible workload:

Use non-preemptible workloads for services that cannot safely tolerate interruption. Use preemptible workloads for restartable training, experiments, and batch work.

Checkpoint Before Allowing Preemption

Preemption without checkpointing can turn high GPU utilization into low useful throughput.

A training workload should save:

Checkpoint intervals should balance storage overhead against the amount of work that could be lost.

Enforce Scheduling Intent with Workload Policies

Do not depend on every user selecting the correct values manually.

Workload policies can apply defaults and restrictions across system, cluster, department, or project scopes. Policies can govern workloads submitted through the user interface, CLI, API, or supported Kubernetes YAML path.

Useful policy controls include:

Example Policy Intent

For product-ai-dev:

Default priority: medium-low
Default preemptibility: preemptible
Maximum GPUs per workload: 4
Allowed node pools: a100-shared, l40s-interactive
Privileged containers: prohibited
Required label: environment=development

For product-ai-prod:

Default priority: very-high
Default preemptibility: non-preemptible
Maximum GPUs per workload: approved production scale
Allowed node pools: h100-production
Privileged containers: prohibited
Required label: environment=production
Required owner annotation: mandatory

The exact policy schema depends on the workload type and current Run:ai release. Validate the available fields before translating this intent into an enforceable policy.

Validate Over-Quota Allocation and Preemption

Do not consider the design complete until quota, over-quota, fairness, and preemption have been tested.

The following test uses two projects with a quota of two GPUs each on a four-GPU node pool.

Test Scenario

  1. Submit three one-GPU preemptible workloads to team-a.
  2. Submit one one-GPU workload to team-b.
  3. Confirm team-a is using one GPU over quota.
  4. Submit a second workload to team-b.
  5. Confirm one preemptible team-a workload is reclaimed.
  6. Confirm both projects receive their two-GPU deserved allocation.

The current Run:ai CLI supports explicit project, node-pool, priority, GPU request, and preemptibility settings.

runai login

runai training standard submit team-a-job-1 \
  -p team-a \
  -i runai.jfrog.io/demo/quickstart-demo \
  --gpu-devices-request 1 \
  --node-pools a100-shared \
  --priority low \
  --preemptibility preemptible

runai training standard submit team-a-job-2 \
  -p team-a \
  -i runai.jfrog.io/demo/quickstart-demo \
  --gpu-devices-request 1 \
  --node-pools a100-shared \
  --priority low \
  --preemptibility preemptible

runai training standard submit team-a-job-3 \
  -p team-a \
  -i runai.jfrog.io/demo/quickstart-demo \
  --gpu-devices-request 1 \
  --node-pools a100-shared \
  --priority low \
  --preemptibility preemptible

runai training standard submit team-b-job-1 \
  -p team-b \
  -i runai.jfrog.io/demo/quickstart-demo \
  --gpu-devices-request 1 \
  --node-pools a100-shared \
  --priority medium \
  --preemptibility preemptible

At this stage, the expected allocation is:

team-a quota:       2 GPUs
team-a allocation:  3 GPUs
team-a over quota:  1 GPU

team-b quota:       2 GPUs
team-b allocation:  1 GPU

cluster allocation: 4 of 4 GPUs

Submit the second Team B workload:

runai training standard submit team-b-job-2 \
  -p team-b \
  -i runai.jfrog.io/demo/quickstart-demo \
  --gpu-devices-request 1 \
  --node-pools a100-shared \
  --priority medium \
  --preemptibility preemptible

The expected steady state is:

team-a allocation: 2 GPUs
team-b allocation: 2 GPUs

team-a-job-3:
  Preempted, terminating, or returned to pending

team-b-job-2:
  Scheduled and running

The specific workload selected for preemption can depend on scheduler state and workload characteristics. Validate the entitlement outcome rather than assuming a particular pod name will always be reclaimed.

Test Development and Production Separation

A second validation should prove that development cannot displace production incorrectly.

Production Workload

runai training standard submit production-service \
  -p product-ai-prod \
  -i runai.jfrog.io/demo/quickstart-demo \
  --gpu-devices-request 2 \
  --node-pools h100-production \
  --priority very-high \
  --preemptibility non-preemptible

Development Workload

runai training standard submit development-training \
  -p product-ai-dev \
  -i runai.jfrog.io/demo/quickstart-demo \
  --gpu-devices-request 2 \
  --node-pools a100-shared \
  --priority medium-low \
  --preemptibility preemptible

Confirm that:

Measure Quota Utilization

Quota management must be based on measured behavior rather than annual estimates.

Run:ai project and department views expose allocation relative to quota. Monitoring and telemetry can also provide GPU allocation, utilization, pending-time, and workload information.

Track at least the following metrics per project, department, and node pool.

MetricWhy it matters
Allocated GPUs divided by guaranteed quotaIndicates how much of the entitlement is being consumed
Over-quota GPU hoursShows dependence on borrowed capacity
Pending workload timeReveals insufficient quota, placement constraints, or fragmentation
Preemption countShows how frequently opportunistic work is displaced
Lost work after preemptionTests whether checkpointing is effective
Idle allocated GPUsIdentifies workloads holding GPUs without useful compute
Node-pool utilizationIdentifies stranded or overloaded hardware classes
Maximum concurrent GPU allocationSupports future quota sizing
Production scheduling delayValidates whether guaranteed capacity is sufficient
Exception frequencyReveals whether the standard quota model fits real demand

Starting Review Thresholds

The following are operating-policy examples, not NVIDIA defaults:

Use longer windows for research workloads with irregular experiments. Use tighter thresholds for production services.

Prevent One Team from Monopolizing the Cluster

No single control prevents monopolization. Use multiple boundaries.

Assign Meaningful Guarantees

Do not assign the complete shared node pool as one project’s guaranteed quota unless that project truly owns the hardware.

Use Maximum Allocations

A project may have a small guarantee and permission to borrow, but it should not necessarily be able to consume the entire pool.

Require Preemptibility for Opportunistic Work

Borrowed capacity must remain reclaimable. Otherwise, over-quota access becomes an uncontrolled reservation.

Limit GPUs per Workload

A single accidental request for all available GPUs can create unnecessary queue pressure even when project-level quotas are working correctly.

Control Expensive Node Pools

For scarce pools:

Detect Idle Allocation

A workload can monopolize GPUs without performing useful work. Monitor GPU utilization, GPU memory behavior, workload phase, and idle allocation together.

Expire Temporary Exceptions

A temporary quota increase without an expiration date is usually a permanent quota increase with incomplete documentation.

Troubleshooting Common Scheduling Problems

SymptomLikely causeCorrective action
Workload remains pending despite apparently free GPUsFree GPUs are in a different node pool or do not satisfy affinity, topology, taint, memory, or workload-size requirementsCheck selected node pools, ordering, node labels, affinity, tolerations, GPU type, and contiguous capacity
Non-preemptible workload cannot start above quotaNon-preemptible workloads cannot rely on over-quota capacityIncrease guaranteed quota, reduce the request, or use an approved preemptible workload
Project cannot use a newly created node poolProject has zero quota, over-quota is disabled, maximum is zero, or policy blocks the poolReview department and project configuration for the new pool
Development workload runs on production GPUsNode-pool ordering, over-quota defaults, or policy permits accessSet project quota and maximum to zero for the production pool and restrict it through policy
Production workload is preemptedIt was submitted as preemptible or inherited an incorrect defaultCorrect the workload policy and resubmit as non-preemptible within deserved quota
Team reports a quota utilization ratio over 100 percentThe project is consuming over-quota capacityReview allocation, guaranteed quota, maximum allocation, and over-quota weight
One project receives most unused GPUsOther projects are not eligible, have lower over-quota weight, are blocked by placement, or have no pending workloadsReview eligibility, weights, queue state, and node-pool constraints
Department guarantees appear ineffectiveProject guarantees exceed the parent department quotaReconcile department and project quota totals per node pool
Frequent preemption wastes training progressCheckpoints are too infrequent, incomplete, or not restoredValidate checkpoint persistence and resume behavior before allowing large preemptible jobs
GPUs are allocated but utilization remains lowWorkload is blocked on data, CPU, storage, network, initialization, or application logicCorrelate GPU metrics with CPU, storage, network, and workload logs
High-priority development jobs dominate queue orderPriority policy is too permissiveRestrict allowed priority values at the development project or department scope
Quota changes cause unexpected scheduling behaviorSeveral quota, maximum, weight, and policy settings changed togetherRoll back to the previous baseline and change one control at a time

Operate Quota Changes as Controlled Platform Changes

Changing quotas can trigger workload movement and preemption. Treat quota changes as production changes.

Use the following sequence:

  1. Capture current project, department, and node-pool allocation.
  2. Record pending workloads and current preemptible workloads.
  3. Confirm the parent department has capacity for the proposed project guarantee.
  4. Check whether maximum allocations must also change.
  5. Check over-quota eligibility and weight.
  6. Identify workloads that may be preempted.
  7. Verify checkpoint status for affected training jobs.
  8. Apply the smallest required change.
  9. Observe allocation, pending time, and preemption.
  10. Confirm that production service objectives remain satisfied.
  11. Record the result and expiration date.
  12. Roll back if the expected entitlement outcome is not observed.

Do not simultaneously change quota, node-pool order, workload policy, and access rules unless the change has been validated in a non-production environment.

Define Operational Ownership

A shared GPU platform requires named owners for both technology and allocation decisions.

CapabilityAccountable ownerResponsible operator
Cluster and scheduler healthAI platform ownerPlatform engineering
Node-pool labels and membershipInfrastructure ownerKubernetes or GPU platform team
Department quotaAI governance or capacity ownerCentral platform administrator
Project quotaDepartment ownerProject administrator with approval
Workload priority policyPlatform governance ownerRun:ai administrator
Identity groups and access lifecycleSecurity or IAM ownerIdentity operations
Checkpointing and restart behaviorApplication ownerML engineering team
Production service capacityProduct service ownerProduction operations
Utilization reportingCapacity or FinOps ownerPlatform analytics
Exception approvalNamed governance authorityPlatform service manager

Quota ownership should not sit entirely with Kubernetes administrators. The administrator can implement a quota, but the business or service owner must justify the entitlement.

Establish an Exception Process

Exceptions are inevitable. Undocumented exceptions are optional.

A quota exception should record:

Requesting department:
Requesting project:
Business owner:
Technical owner:
Requested node pool:
Current guaranteed quota:
Temporary guaranteed quota:
Maximum requested allocation:
Over-quota requirement:
Preemptibility:
Workload priority:
Business justification:
Expected start:
Expiration:
Checkpoint validated:
Production impact assessment:
Approver:
Rollback action:

Use separate exception paths for:

Every exception should expire automatically or enter a mandatory review state.

A defensible starting baseline is:

Production

Development

Research

Shared Platform Services

Conclusion

Multi-tenant GPU scheduling is not solved by dividing the number of GPUs by the number of teams.

A workable NVIDIA Run:ai design maps durable organizational boundaries into departments, isolates practical workload boundaries into projects, and assigns quota separately for each relevant node pool. Guaranteed quota provides predictability, while controlled over-quota use improves utilization when capacity would otherwise remain idle.

Fairness and preemption make borrowed capacity recoverable. Priority helps order work, but it should never replace quota design. Production workloads should receive quota-backed, non-preemptible capacity. Development and research workloads can use preemptible excess capacity when checkpointing and restart behavior are proven.

The operating model is as important as the scheduler configuration. Departments need accountable owners, projects need access boundaries, quota changes need evidence, and exceptions need expiration dates. When those controls are in place, a shared GPU cluster can support multiple teams without becoming either permanently underutilized or operationally unpredictable.

External References

Exit mobile version