Site icon Digital Thought Disruption

Who Used the GPU? Building Per Tenant Telemetry, Showback, and Capacity Evidence for AIaaS

Paul Bryant

Introduction

A shared GPU platform creates a deceptively simple question: who used the GPU?

The question becomes difficult as soon as the platform supports more than one operating model. A Kubernetes pod may receive an entire GPU, a MIG instance, or a time-sliced share. A virtual machine may receive a vGPU profile or a pass-through device. A model server may serve requests for several customers from the same process. A scheduler may let one department borrow another department’s unused quota. The GPU may look busy while the application produces little useful work, or it may appear lightly utilized while holding scarce memory capacity that prevents another model from starting.

This is why a single GPU utilization graph cannot support tenant accountability. NVIDIA DCGM and DCGM Exporter are foundational components, but device telemetry does not automatically become department, project, model, or customer attribution [1]-[4]. Kubernetes metadata tells you which pod received a device, but not whether the pod produced valuable work [5], [6]. Model-server metrics expose tokens, requests, latency, and queue depth, but they do not establish who funded the underlying capacity [12]-[15]. A finance ledger can assign cost, but only after the platform has reliable allocation and identity evidence.

An enterprise AI-as-a-Service platform therefore needs an evidence chain, not a dashboard shortcut. That evidence chain must connect business ownership to scheduler decisions, workload identity, device identifiers, hardware telemetry, model-server outcomes, rate cards, and retained monthly records. It must also preserve uncertainty when attribution is approximate, especially for time-sliced GPUs, shared model processes, short-lived workloads, and unsupported hardware paths. Community discussions around emerging GPU observability tools reinforce the operational demand for attribution that goes beyond fleet-wide graphs [24].

TL;DR

A defensible GPU showback system measures four different things and never treats them as synonyms:

Use DCGM and DCGM Exporter for device health and activity, Kubernetes PodResources or hypervisor inventory for ownership mapping, model-server metrics for workload outcomes, Prometheus for collection and recording rules, Grafana for role-specific dashboards, Alertmanager for actionable routing, and a separate financial evidence store for monthly showback or chargeback. Bill exact capacity from allocation records. Use utilization to expose efficiency. Use productive-work metrics to explain value. Mark process-level and time-slicing attribution with an explicit confidence level rather than presenting modeled estimates as exact measurements.

The Four Measurements That Must Not Be Collapsed

The most important design decision is conceptual, not technical. The platform must maintain separate definitions for allocation, utilization, productive work, and financial consumption.

MeasurementPrimary questionTypical sourceUseful unitWhat it does not prove
AllocationWho controlled capacity?Scheduler, kubelet, device plugin, DRA, hypervisor, quota managerGPU-hours, MIG-profile-hours, vGPU-profile-hoursThat the device was busy
UtilizationWhat was the device doing?DCGM, NVML, vGPU host metricsSM active ratio, GPU busy ratio, memory used, wattsThat useful business work completed
Productive workWhat service outcome was delivered?Triton, vLLM, NIM, training framework, application telemetryRequests, tokens, samples, steps, jobs, successful outcomesWho should pay for reserved capacity
Financial consumptionWhich cost policy applies?Rate card, asset model, scheduler ledger, finance rulesCurrency, cost per GPU-hour, cost per request, cost per tokenThe physical reason for poor efficiency

Allocation

Allocation is a control-plane fact. It answers who had the right to use capacity, not whether they exercised that right efficiently.

For an exclusive Kubernetes allocation, the unit may be one GPU assigned to a pod for three hours. For MIG, the unit should preserve the profile, such as a specific MIG profile held for three hours. For vGPU, the unit should preserve the vGPU profile and VM assignment. For time-slicing, the unit should state the configured replica or share policy rather than pretending the tenant owned an isolated fraction of the silicon.

A general allocation equation is:

Allocated GPU-equivalent hours
  = integral of assigned capacity fraction over elapsed time

The capacity fraction must come from an approved inventory model. It should not be inferred from a momentary utilization percentage.

Utilization

Utilization is sampled device behavior. Depending on the metric, it may describe the fraction of time the GPU was active, the fraction of streaming multiprocessors with at least one active warp, memory-copy engine activity, framebuffer memory consumption, power draw, or another device condition.

Utilization is useful for identifying idle reservations, saturation, poor batching, thermal throttling, memory pressure, or workload inefficiency. It is not a universal billing unit. A tenant may be financially responsible for a reserved GPU even when utilization is low, just as a reserved virtual machine still has a cost when its CPU is quiet.

An efficiency-oriented derived measure can be useful:

Active-equivalent GPU-hours
  = integral of allocated capacity fraction
    x normalized active ratio
    over elapsed time

This is an analytical measure, not automatically an invoice measure. The organization must decide whether active-equivalent hours affect pricing, efficiency targets, or only operational reporting.

Productive Work

Productive work is application-specific. For an inference service, productive work may include successful requests, input tokens, output tokens, images generated, embeddings produced, or requests meeting a service-level objective. For training, it may include completed steps, samples processed, checkpoints produced, or jobs completed successfully. For interactive development, a productive outcome may be harder to automate and may require a different service model.

The same GPU activity can produce very different amounts of useful work depending on model size, precision, batch size, sequence length, runtime, cache behavior, concurrency, and latency target. This is why tokens per second, requests per second, time to first token, queue delay, and success rate must sit beside GPU telemetry.

Financial Consumption

Financial consumption applies policy to evidence. It determines whether a tenant pays for guaranteed quota, actual allocation, borrowed capacity, energy, premium support, model-serving consumption, or shared platform overhead.

A defensible chargeback model is explicit:

Tenant charge
  = guaranteed capacity charge
  + borrowed allocation charge
  + variable service or energy charge
  + allocated shared-platform overhead
  - approved credits and adjustments

The platform team should be able to explain every term. A finance reviewer should be able to reproduce the monthly result. An application owner should be able to see why an idle reservation still cost money, why borrowed usage was treated differently, and why high device utilization did not necessarily translate into a low cost per successful outcome.

Build an Attribution Chain, Not a Label Collection

Per-tenant telemetry depends on a stable chain of identities across business, orchestration, workload, device, and service layers.

Pod names, process identifiers, and local GPU indices are not durable enough by themselves. Pods are recreated. Process identifiers are reused. GPU index zero means something different on every node. Namespace names can be reused after deletion. Model names may remain unchanged while the weight digest, quantization, runtime profile, or deployment configuration changes.

The evidence model should use stable identifiers and preserve time validity.

LayerRecommended evidence keysWhy they matter
Businesstenant ID, department ID, customer ID, cost center, project ID, service ownerEstablishes accountability and reporting hierarchy
Kubernetescluster ID, namespace UID, workload controller UID, pod UID, container ID, node nameSurvives pod-name reuse and connects device allocation to workload ownership
Physical GPUGPU UUID, node, PCI bus identity, GPU model, serial where availableEstablishes the physical device and prevents local-index ambiguity
MIGparent GPU UUID, GPU instance ID, compute instance ID, MIG device UUID, profileIdentifies the allocated partition and its capacity class
Virtualizationhypervisor cluster, host ID, VM UUID, vGPU UUID, vGPU profile, guest instance IDConnects host-side vGPU evidence to the correct VM and tenant
Model serviceendpoint ID, model name, model version or digest, runtime, deployment revisionConnects hardware activity to the service delivering work
Request or jobtrace ID, request ID, job ID, queue ID, user or service identitySupports detailed investigation outside high-cardinality metric labels
Financerate-card version, allocation class, billing period, adjustment IDMakes financial output reproducible

Preserve Time-Valid Ownership

The platform should maintain an ownership mapping with valid_from and valid_to timestamps. When a pod is deleted, a VM is reassigned, a namespace changes owners, or a vGPU moves, the old relationship must remain queryable for the period in which it was true.

This temporal mapping is essential for month-end close. A current CMDB lookup cannot reliably explain who owned a workload three weeks earlier. The evidence pipeline should capture assignment events as they happen and retain them independently from the live orchestration system.

Keep High-Cardinality Identity Out of Prometheus Labels

Prometheus labels are effective for bounded dimensions such as cluster, namespace, model, service tier, GPU class, and a manageable set of tenant identifiers. They are a poor place for request IDs, user email addresses, raw process IDs, session IDs, or arbitrary customer strings because every unique label combination creates another time series [8].

Use metrics for bounded aggregation and alerting. Use logs, traces, or an analytical event store for per-request and per-process detail. Join those records into the financial ledger during controlled aggregation rather than multiplying the number of Prometheus time series without limit.

Reference Telemetry Architecture

The reference architecture separates collection, identity enrichment, operational analytics, and financial evidence. Notice that Kubernetes and virtual-machine paths converge only after each has produced a reliable device-to-workload mapping.

What Each Component Owns

What DCGM and DCGM Exporter Actually Measure

NVIDIA Data Center GPU Manager provides the device-level telemetry, health, diagnostics, policy, profiling, and accounting capabilities that make GPU operations observable [2], [3]. DCGM Exporter exposes selected DCGM fields in Prometheus format and is commonly deployed as a DaemonSet on GPU nodes [1].

The exporter is a collection layer. It does not replace the identity model, scheduler ledger, or model-server telemetry.

Core Metric Domains

DomainExample evidenceOperational use
Identity and inventoryGPU UUID, device model, node, MIG identifiers and profileDevice mapping, inventory reconciliation, capacity segmentation
Memoryframebuffer total, used, free, reserved, memory-copy activityModel fit, memory pressure, stranded memory, leak detection
Compute activityGPU active ratio, SM activity, SM occupancy or related profiling fieldsIdle detection, saturation analysis, kernel efficiency clues
Power and thermalspower draw, temperature, memory temperature, clock values, clock-event reasonsCooling issues, throttling, energy estimates, performance anomalies
InterconnectPCIe and NVLink counters or throughput where supportedData-path bottlenecks, topology issues, multi-GPU diagnosis
ReliabilityXID incidents, ECC errors, row-remap or memory-repair indicators, device healthIncident response, quarantine, warranty and support evidence
Process and job accountingprocess activity, process memory, job statistics where enabled and supportedAttribution refinement and workload summaries

Metric Names Are Versioned Interfaces

DCGM field names and exporter aliases evolve. A release may rename labels, add cumulative counters, change field aliases, or improve MIG and Kubernetes metadata behavior [4]. The platform should not let raw vendor metric names spread through every dashboard and financial query.

Use a normalization layer based on Prometheus recording rules or a telemetry transform. For example:

groups:
  - name: ai-gpu-normalization
    interval: 30s
    rules:
      - record: ai_gpu:sm_active_ratio
        expr: clamp_max(DCGM_FI_PROF_SM_ACTIVE, 1)

      - record: ai_gpu:framebuffer_used_bytes
        expr: DCGM_FI_DEV_FB_USED * 1024 * 1024

      - record: ai_gpu:power_watts
        expr: DCGM_FI_DEV_POWER_USAGE

      - record: ai_gpu:temperature_celsius
        expr: DCGM_FI_DEV_GPU_TEMP

The example assumes the listed exporter aliases exist in the deployed release. Validate the active collector CSV, exporter output, units, labels, and field support on each GPU generation before promoting these rules. The stable contract for dashboards should be the normalized ai_gpu:* series, not an unreviewed dependency on every upstream field name.

SM Activity Is Not Business Productivity

A high SM activity ratio indicates that streaming multiprocessors were active during the sample window. It does not prove that kernels were efficient, that memory access was optimal, that the model met latency targets, or that the work produced a successful business result.

Low activity can indicate idle reservation, input starvation, CPU or storage bottlenecks, small batches, synchronization, or an interactive workload waiting for a user. High activity can coexist with low requests per second, excessive time to first token, retries, or failed requests.

Sampling Can Miss Short Work

GPU telemetry is sampled. Short kernels and short-lived processes may start and finish between samples. Process-level attribution therefore has a confidence boundary even when the platform exposes process metrics. Increase collection frequency only after measuring the storage, CPU, and cardinality cost. High-frequency raw metrics may belong in short retention, while aggregated evidence belongs in long retention.

Kubernetes Attribution from Device Assignment to Tenant

Kubernetes attribution requires several control-plane facts to line up:

  1. A device plugin or Dynamic Resource Allocation driver advertises GPU resources [6].
  2. The scheduler selects a node based on the resource request, policy, topology, and available capacity.
  3. The kubelet assigns device identifiers to containers.
  4. The local PodResources API exposes container and exclusive-resource assignments to monitoring agents [5].
  5. DCGM Exporter maps device telemetry to pod and container metadata where the allocation mode and release support that mapping.
  6. Kubernetes metadata is joined to approved tenant, project, service, and cost-center identifiers.
  7. Allocation events are retained after the pod disappears.

Use Pod UID and Controller UID, Not Pod Name Alone

A deployment may create many pods with changing names. A job may be retried. An operator may recreate a model-serving pod under the same logical service. The ownership record should therefore preserve:

The human-readable namespace, deployment, and pod names remain useful labels, but the immutable identifiers are the evidence keys.

Create a Bounded Identity Metric

One practical pattern is a small metadata exporter that emits a gauge with value 1 for each active workload identity. The gauge carries only allowlisted, bounded labels.

ai_workload_identity{
  cluster="prod-ai-1",
  namespace="risk-models",
  pod="fraud-llm-7c9d8",
  pod_uid="4d31...",
  workload_uid="7a4c...",
  tenant_id="finance",
  project_id="fraud-platform",
  service_tier="production"
} 1

Recording rules can then enrich normalized GPU series:

groups:
  - name: ai-gpu-tenant-attribution
    rules:
      - record: ai_tenant_gpu:sm_active_ratio
        expr: |
          ai_gpu:sm_active_ratio
            * on (cluster, namespace, pod)
              group_left(tenant_id, project_id, service_tier, pod_uid, workload_uid)
              ai_workload_identity

      - record: ai_tenant_gpu:framebuffer_used_bytes
        expr: |
          ai_gpu:framebuffer_used_bytes
            * on (cluster, namespace, pod)
              group_left(tenant_id, project_id, service_tier, pod_uid, workload_uid)
              ai_workload_identity

This design gives the platform team a deliberate schema and a testable failure mode. When a GPU series cannot join to an identity series, it becomes an unattributed telemetry incident instead of silently landing in the wrong tenant report.

Queue Time Has Multiple Meanings

A single panel named “queue time” is not enough.

Queue or delayStartEndWhat it reveals
Admission delayWorkload submittedWorkload admittedPolicy, quota, approval, or admission-control pressure
Scheduler delayPod created or job queuedNode assignmentCapacity shortage, topology constraints, fairness, priority, or fragmentation
Startup delayNode assignmentContainer or endpoint readyImage pull, model load, storage, initialization, or health-check delay
Model-server queueRequest acceptedRequest scheduled for executionConcurrency, batching, KV cache, runtime, or GPU saturation
End-to-end latencyClient requestFinal responseFull service experience, including network and application layers

Capacity forecasting should treat sustained scheduler delay as unmet demand. Model-server queue delay is a service bottleneck and may occur even when Kubernetes has already allocated the GPU.

MIG Attribution Is Stronger Than Time-Slicing, but the Math Still Matters

MIG creates hardware-isolated GPU instances with identifiable profiles and instance identifiers. This makes allocation attribution stronger than time-slicing because the platform can associate a specific MIG device with a pod or VM.

The billing unit should be the allocated MIG profile over time:

MIG allocation cost
  = MIG-profile-hours x approved profile rate

Do not bill a MIG tenant from the parent GPU utilization percentage. Different instance profiles have different capacity denominators, and parent-device activity is not necessarily a simple sum or average of instance percentages.

Capacity-Normalized Utilization

For fleet planning, the platform may calculate a GPU-equivalent active measure:

Capacity-normalized active equivalent
  = sum for each MIG instance of
      profile capacity share x instance active ratio

The profile capacity share must come from an approved profile inventory and validation method. It should not be guessed from framebuffer memory alone. Memory, compute slices, media engines, and supported metrics do not always scale identically. Community questions involving heterogeneous MIG layouts also demonstrate why a simple weighted reconstruction of parent utilization should not be treated as authoritative without validation [23].

Keep three distinct views:

Fragmentation Is Its Own Capacity State

A GPU may have free aggregate capacity but still be unable to satisfy a requested MIG profile. That is fragmentation, not ordinary utilization. Capacity dashboards should report:

Without this view, a platform can report low average utilization while users experience long queue times.

Time-Slicing Attribution Requires an Explicit Confidence Boundary

Time-slicing lets multiple workloads share a physical GPU through scheduling rather than hard partitioning. It can improve accessibility for development and bursty work, but it complicates attribution.

A physical-device metric may be repeated across time-slice replicas because the underlying measurement belongs to the GPU, not to each logical replica. Dividing the physical utilization equally among configured replicas is not exact attribution. It is a modeled allocation assumption.

Recent DCGM Exporter releases added per-process GPU metrics for time-sharing and MIG, and newer releases continue to improve Kubernetes and container-runtime mapping [4]. That does not make process attribution universal. Hardware support, driver support, DCGM support, exporter configuration, process-accounting support, container-runtime mapping, and sampling behavior still determine what evidence is available.

Process-Level Attribution Chain

A process-level design needs to resolve:

GPU process sample
      -> host PID
      -> container cgroup or runtime container ID
      -> pod UID or VM UUID
      -> model-server worker
      -> tenant, project, model, and endpoint

Failure can occur at every step. PIDs are reused. Processes terminate between samples. Multi-process services fork workers. NVIDIA MPS can change process behavior. A shared model server may intentionally serve several tenants from one process. In that last case, process telemetry can attribute activity to the service but not accurately divide it among customers. Request and token records must perform the customer split.

Attribution Confidence Levels

ConfidenceTypical allocation modeEvidence qualityAppropriate use
HighExclusive GPU, MIG device, dedicated vGPUStable device-to-workload mappingChargeback, SLO analysis, audit evidence
MediumSupported process mapping on shared GPUSampled PID-to-container mapping with known gapsShowback, efficiency analysis, provisional allocation
ModeledTime-slice replicas without complete process evidencePolicy-based proportional split or equal-share assumptionCapacity planning with disclosure, not exact billing
UnattributedMissing metadata, unsupported device, telemetry gapNo defensible mappingPlatform exception queue and cost suspense account

A community report involving Blackwell GB10 is a useful warning. The visible symptom looked like missing per-process attribution, but the vendor response identified the platform as unsupported by DCGM [22]. The operational lesson is to validate the exact GPU and software support boundary before treating exporter output as proof of a general product limitation.

vGPU and VM Attribution Needs Host and Guest Evidence

Virtualized GPU environments need two connected views.

The host or hypervisor view establishes the physical GPU, vGPU instance, vGPU profile, VM UUID, and assignment interval. NVIDIA’s management interfaces can expose vGPU-instance and vGPU-process utilization where supported [16], [17]. The guest view provides telemetry scoped to the VM and, when applicable, the model server or application running inside it.

Physical GPU UUID
   -> vGPU UUID and profile
   -> hypervisor VM UUID
   -> VM owner and cost center
   -> guest workload or model endpoint
   -> requests, tokens, latency, and outcomes

Use vGPU-profile-hours for allocation cost. Use vGPU activity for efficiency. Use guest model metrics for productive work.

Watch the Utilization Denominator

A utilization percentage reported inside a vGPU can be relative to the vGPU’s permitted engine capacity, depending on the scheduler. A VM that fully uses its allocated share may report 100 percent even though it is consuming only part of the physical GPU. Host and guest percentages therefore need documented semantics before they are placed on the same chart.

Avoid Double Counting

If host-side vGPU telemetry and guest-side DCGM or NVML telemetry describe the same activity, do not sum them. Use one as the authoritative allocation and device view, and the other as a reconciliation and service detail view.

Pass-through devices require another boundary. The vGPU management layer may not manage pass-through GPUs from the hypervisor, so guest telemetry and hypervisor assignment records become the primary evidence sources.

Model-Server Metrics Convert GPU Activity into Productive Work

Device telemetry explains hardware behavior. Model-server telemetry explains service behavior.

Triton Inference Server exposes request counts, inference counts, execution counts, request timing, and GPU statistics. Its inference and execution counters can also reveal effective batching behavior [12]. vLLM exposes request, token, queue, latency, KV cache, and optional model-flops metrics [13], [14]. NVIDIA NIM for large language models provides Prometheus-compatible metrics and, for supported runtime paths, surfaces runtime-native metrics rather than inventing a separate semantic layer [15].

Core Inference Metric Definitions

MetricDefinitionWhy it matters
Requests per secondRate of accepted, successful, failed, or completed request countersDemand, throughput, errors, service capacity
Input tokens per secondRate of prompt-token countersPrefill workload and input intensity
Output tokens per secondRate of generated-token countersDecode throughput and productive output
Time to first tokenRequest timing to first generated token, using the runtime’s documented start pointUser-perceived responsiveness and prefill pressure
Inter-token latencyTime between output tokens during decodeStreaming quality and decode consistency
End-to-end latencyRequest arrival to completionFull service experience
Model queue timeTime waiting in the runtime scheduler before executionSaturation, batching, concurrency, KV cache pressure
Running requestsRequests actively executing or in an execution batchConcurrency and active load
Waiting requestsRequests queued for executionBacklog and unmet service demand
KV cache usageFraction or bytes of key-value cache occupiedConcurrency limit, sequence pressure, eviction risk
Average dynamic batch sizeInferences divided by executions when semantics support the calculationBatching efficiency and throughput behavior
Success rateSuccessful requests divided by total completed requestsProductive output quality gate
Per-model latencyLatency histogram segmented by model and deployment revisionModel-specific SLO and regression analysis

Token and Request Rates

Use counters and calculate rates in the query layer:

sum by (tenant_id, model) (
  rate(ai_workload:output_tokens_total[5m])
)
sum by (tenant_id, model) (
  rate(ai_workload:requests_total{outcome="success"}[5m])
)

For latency, use histograms and agreed percentile windows:

histogram_quantile(
  0.95,
  sum by (le, tenant_id, model) (
    rate(ai_workload:ttft_seconds_bucket[10m])
  )
)

Do Not Put Request IDs in Prometheus

Per-request metrics are useful, but they should normally be returned in the response, written to structured logs, or attached to distributed traces. A request ID as a Prometheus label creates an unbounded series dimension.

A practical split is:

Shared Model Servers Need Request-Level Tenant Accounting

A single model-server process can serve multiple tenants. Device metrics and process metrics can attribute GPU activity to the server, but not necessarily to each customer. For this pattern, customer showback should be based on request and token evidence, then reconcile against the service’s allocated GPU cost.

One useful approach is to calculate a tenant’s share of a shared endpoint using an approved workload driver:

Tenant service share
  = tenant weighted productive units
    / total endpoint weighted productive units

Weighted units may account for input tokens, output tokens, image size, model class, priority tier, or validated compute coefficients. The coefficient model must be versioned and tested. A raw output-token count alone may not fairly represent prefill-heavy or multimodal workloads.

Normalize Metrics Before Building Dashboards or Bills

Raw sources expose different names, units, labels, and update intervals. A platform-wide semantic layer prevents every dashboard from inventing its own interpretation.

A useful canonical contract includes:

Canonical seriesTypeRequired bounded dimensionsPurpose
ai_gpu:allocation_fractionGauge or interval recordcluster, tenant, project, workload, GPU class, allocation classAssigned capacity
ai_gpu:sm_active_ratioGaugecluster, node, GPU or instance identityNormalized activity
ai_gpu:framebuffer_used_bytesGaugecluster, device identityMemory occupancy
ai_gpu:power_wattsGaugecluster, physical GPUPower observation
ai_gpu:health_incidents_totalCountercluster, physical GPU, category, severityReliability evidence
ai_scheduler:queue_secondsHistogramcluster, queue, tenant, priority, GPU classScheduling delay
ai_workload:requests_totalCountertenant, project, model, endpoint, outcomeService demand and success
ai_workload:input_tokens_totalCountertenant, model, endpointPrefill volume
ai_workload:output_tokens_totalCountertenant, model, endpointGenerated output
ai_workload:ttft_secondsHistogramtenant class, model, endpointResponsiveness
ai_workload:e2e_latency_secondsHistogramtenant class, model, endpoint, outcomeService-level evidence
ai_finance:gpu_rate_per_hourVersioned ledger dimensionGPU class, profile, service tier, periodCost calculation input

Prometheus should hold operational series. The financial rate record should normally live in a controlled data store and be exported into the reporting model, not edited casually as a dashboard constant.

Showback and Chargeback Need Different Governance

Showback reports consumption and cost without creating a financial transfer. Chargeback posts or reallocates cost to a tenant, department, or customer. The telemetry can be shared, but the governance bar is different.

Showback Can Tolerate Provisional Data

Showback can include near-real-time allocation, current utilization, idle reservations, queue delay, estimated cost, and attribution-confidence flags. It is useful for behavior change and capacity conversations.

Chargeback Requires a Closed Ledger

Chargeback should use a closed period with:

Guaranteed Quota Versus Borrowed Usage

A fair scheduler may let a tenant exceed guaranteed quota when other tenants are not using their entitlement [18], [19]. That creates at least two distinct capacity classes:

Report them separately:

Guaranteed GPU-hours
Borrowed GPU-hours
Preempted borrowed GPU-hours
Idle guaranteed GPU-hours
Productive units delivered by each capacity class

A common financial model charges guaranteed quota at a predictable committed rate and borrowed allocation at a burst rate. Another model charges only actual allocation but reports unused entitlement as a planning inefficiency. The correct choice is organizational policy, not a telemetry fact.

Classify Idle Capacity Correctly

Idle classDescriptionLikely financial owner
Tenant-reserved idleGuaranteed or dedicated capacity assigned but inactiveTenant, unless centrally subsidized
Borrowed idleOver-quota capacity assigned but no longer usefulTenant until reclaimed, with aggressive scheduler cleanup
Platform unallocated idleCapacity available to the shared poolPlatform or central AI service
Fragmentation idleCapacity unusable because of MIG profile, topology, memory, or scheduling constraintsPlatform, or shared with the tenant when caused by a dedicated design choice
Reliability reserveCapacity deliberately held for failure, maintenance, or recoveryPlatform overhead
Telemetry-unknown idleCapacity appears idle because evidence is missingSuspense category pending investigation

Calling all idle GPU time “waste” produces the wrong behavior. Cost frameworks such as OpenCost also distinguish shared, allocated, and idle cost treatment rather than collapsing them into one figure [20]. Reliability reserve is intentional. Tenant-reserved idle may be the cost of an availability promise. Fragmentation may require layout changes, not user coaching.

Example Grafana Dashboards

A useful dashboard system provides different views for platform operators, tenant owners, model-service teams, and finance. One giant dashboard will satisfy none of them.

Fleet Capacity and Health Dashboard

Primary audience: platform operations and hardware teams.

Recommended variables:

Recommended panels:

Tenant Showback Dashboard

Primary audience: department, customer, project, and service owners.

Recommended variables:

Recommended panels:

Model Efficiency and SLO Dashboard

Primary audience: model platform and application teams.

Recommended panels:

Scheduler and Capacity Dashboard

Primary audience: platform engineering, capacity planning, and service management.

Recommended panels:

Dashboard Wireframe

Grafana variables should make the dashboards reusable instead of duplicating one dashboard per tenant. Provision data sources, folders, dashboards, and alert integrations through version-controlled configuration [10], [11]. Apply role-based access so one customer cannot query another customer’s telemetry.

Alert Examples That Protect Evidence and Capacity

Alerts should identify actionable conditions, not merely display high values. Route hardware-health alerts to platform operations, tenant-efficiency alerts to service owners, and billing-evidence alerts to the telemetry team.

The following example uses normalized metrics. Thresholds are placeholders and must be validated by GPU model, workload class, service level, and operating policy.

groups:
  - name: aias-gpu-accountability
    rules:
      - alert: GPUAllocationMissingTenantIdentity
        expr: |
          ai_gpu:allocation_fraction > 0
            unless on (cluster, workload_uid)
          ai_workload_identity == 1
        for: 10m
        labels:
          severity: critical
          owner: telemetry-platform
        annotations:
          summary: Allocated GPU capacity has no tenant identity
          action: Quarantine the interval from chargeback and repair the mapping

      - alert: TenantGPUAllocatedButIdle
        expr: |
          avg_over_time(ai_tenant_gpu:allocation_fraction[30m]) > 0
            and on (cluster, tenant_id, workload_uid)
          avg_over_time(ai_tenant_gpu:sm_active_ratio[30m]) < 0.05
        for: 30m
        labels:
          severity: warning
          owner: tenant-platform
        annotations:
          summary: Tenant workload holds GPU capacity with minimal activity
          action: Validate workload intent before reclaiming or resizing capacity

      - alert: GPUHealthIncidentDetected
        expr: |
          increase(ai_gpu:health_incidents_total{severity=~"critical|fatal"}[5m]) > 0
        labels:
          severity: critical
          owner: gpu-operations
        annotations:
          summary: Critical GPU health incident detected
          action: Preserve evidence, evaluate workload evacuation, and follow the hardware runbook

      - alert: InferenceQueueSLOAtRisk
        expr: |
          histogram_quantile(
            0.95,
            sum by (le, cluster, endpoint, model) (
              rate(ai_workload:queue_seconds_bucket[10m])
            )
          ) > 30
        for: 10m
        labels:
          severity: warning
          owner: model-platform
        annotations:
          summary: Model-server queue delay threatens the service objective
          action: Check concurrency, batching, KV cache, replicas, and GPU saturation

      - alert: BorrowedGPUCapacityApproachingPolicyLimit
        expr: |
          ai_tenant_gpu:borrowed_allocation_fraction
            / ai_tenant_gpu:borrow_limit_fraction > 0.9
        for: 15m
        labels:
          severity: warning
          owner: capacity-management
        annotations:
          summary: Tenant borrowed GPU consumption is near its policy limit
          action: Review reclaim risk, queued demand, and production priority

Alertmanager should group related device failures by cluster and node, inhibit secondary tenant alerts when a node is already declared unavailable, and route financial-evidence failures differently from performance warnings [9]. Missing attribution is a critical data-quality alert because it can invalidate chargeback even when workloads continue running.

Capacity Forecasting Requires More Than Average Utilization

Average GPU utilization hides peaks, queue pressure, model mix, profile fragmentation, and failure reserve. Forecast at least four demand curves:

Add a fifth curve for unmet demand, derived from pending workloads, scheduler delay, admission delay, and model-server queues.

Forecast by Capacity Class

Do not treat every GPU as interchangeable. Segment forecasts by:

Use Validated Service Throughput

For an inference service, estimate capacity from a controlled benchmark that preserves the target SLO:

Required service GPUs
  = forecast peak productive throughput
    / validated sustainable throughput per GPU at the target SLO
    + failure and maintenance reserve

The validated denominator must match model, precision, runtime, batch policy, sequence distribution, concurrency, and hardware. A tokens-per-second number obtained under a different latency target is not a safe capacity input.

Queue Pressure Is Demand Evidence

When utilization looks moderate but queue time rises, investigate fragmentation, memory reservation, topology, admission policy, model loading, and runtime bottlenecks. Queue pressure can expose demand that utilization alone cannot see.

Evidence Retention and Auditability

Prometheus is excellent for operational metrics, but local Prometheus storage alone is not a financial evidence system. Long-term evidence needs durable retention, controlled schemas, and reproducible monthly close.

A practical design can use Prometheus for local collection and alerting, then remote-write or upload blocks into a durable Prometheus-compatible backend backed by object storage [21]. Request and trace events can use a separate analytical store. Monthly cost output should be materialized into an immutable ledger or governed warehouse table.

Example Retention Tiers

These are planning examples, not universal legal requirements:

EvidenceExample retentionReason
High-resolution device and process metrics30 to 90 daysIncident response, tuning, short-term attribution investigation
Five-minute operational rollups13 to 25 monthsSeasonality, trend, capacity, budget comparison
Allocation events and ownership historyAt least the financial and audit horizonReconstructs who controlled capacity
Request and token aggregatesBased on customer, security, and finance policyProductive-work showback and service economics
Monthly chargeback ledgerCommonly aligned to finance record-retention policyAudit, reconciliation, dispute handling
Raw prompts and outputsOnly when required, minimized and protectedHigh privacy and data-governance risk

Preserve the Calculation Context

Each closed period should retain:

A future reviewer should be able to reproduce the charge without relying on the current state of Kubernetes, Grafana, or the hypervisor.

Implementation Sequence

A platform team can build this architecture incrementally, but the sequence matters.

PhaseKey workExit evidence
Identity contractDefine tenant, project, service, device, workload, and finance keysApproved schema and ownership policy
Device inventoryRecord physical GPU, MIG, vGPU, node, VM, and capacity-class relationshipsReconciled inventory with stable identifiers
Device telemetryDeploy DCGM Exporter or supported VM telemetry and validate fieldsKnown-good metrics by GPU generation and allocation mode
Allocation capturePersist Kubernetes, scheduler, quota, VM, and vGPU assignment intervalsNo orphan allocation intervals in test cases
Metadata enrichmentJoin device evidence to workload and business identitiesAttribution confidence and exception reporting
Productive-work telemetryInstrument Triton, vLLM, NIM, training, or application outcomesRequest, token, latency, queue, and success evidence
Semantic normalizationCreate canonical metrics and recording rulesVersioned metric contract and unit tests
Showback pilotProduce tenant dashboards and estimated costTenant review and dispute findings
Chargeback readinessAdd rate cards, close controls, reconciliation, and correctionsFinance-approved monthly close process
Forecasting and optimizationAdd peak, queue, profile, failure-reserve, and outcome forecastsCapacity plan tied to service demand and SLOs

Start with Showback

Showback exposes data-quality problems without immediately creating financial disputes. Use the pilot to discover missing labels, pod-lifecycle gaps, ambiguous shared services, unsupported GPU modes, and confusing rate policies. Promote only high-confidence allocation classes into chargeback.

Validation Tests Before Financial Use

A telemetry architecture should be tested like a production control, not accepted because a dashboard displays data.

Exclusive GPU Test

Run one known pod or VM on one exclusive GPU. Confirm:

MIG Test

Create different MIG profiles and run controlled workloads. Confirm:

Time-Slicing Test

Run two workloads with distinct load patterns. Confirm whether the deployed stack provides process-level evidence. Compare observed attribution with controlled request or job totals. Mark the result as modeled when the system cannot isolate activity reliably.

vGPU Test

Map the physical GPU, vGPU UUID, profile, VM UUID, guest telemetry, and model endpoint. Confirm host and guest percentages use documented denominators and are not summed.

Shared Model-Service Test

Send requests from multiple tenants through one process. Confirm that request and token events divide customer consumption while device and process metrics remain assigned to the shared service. Reconcile tenant service shares to the endpoint’s total cost.

Lifecycle and Failure Tests

Test pod deletion, pod rescheduling, VM reassignment, exporter restart, Prometheus outage, clock skew, model revision change, node failure, telemetry gaps, and late-arriving data. A month-end process that works only during steady state is not ready for chargeback.

Reconciliation Controls

Use explicit equations:

Physical capacity hours
  = allocated hours
  + unallocated hours
  + unavailable or maintenance hours
  + approved reconciliation difference
Recoverable platform cost
  = tenant charges
  + central platform charges
  + approved subsidies
  - credits
  + approved reconciliation difference

Set a small tolerated difference, investigate exceptions, and prevent the period from closing when unattributed capacity or cost exceeds policy.

Operational Guardrails

A mature AIaaS telemetry service needs operating ownership.

Document the support boundary for every GPU generation, driver, DCGM release, exporter release, vGPU release, Kubernetes version, runtime, and hypervisor integration. A metric that works on one validated cluster should not be assumed to work on every device or virtualization path.

Conclusion

The question “who used the GPU?” has no reliable one-metric answer.

DCGM and DCGM Exporter can show what a supported GPU was doing. Kubernetes PodResources, scheduler evidence, device allocation records, and hypervisor inventory can show which workload controlled the device. Triton, vLLM, NIM, training frameworks, and application telemetry can show whether the workload produced useful service outcomes. A governed cost ledger can then apply guaranteed-quota, borrowed-capacity, variable-service, and shared-overhead rules.

The architecture becomes defensible when those measurements remain separate but connected. Allocation answers ownership. Utilization answers hardware activity. Productive work answers service value. Financial consumption answers policy-based cost responsibility.

That separation also makes the difficult cases manageable. MIG can be billed by profile-hours while utilization is analyzed per instance. vGPU can be billed by profile assignment while guest and host telemetry are reconciled. Time-slicing can carry a modeled confidence flag instead of an invented exact percentage. Shared model services can allocate customer cost through request and token evidence rather than trying to split a single process metric.

The platform team’s goal is not to produce the most colorful GPU dashboard. It is to create an evidence chain that survives pod deletion, process churn, node failure, model upgrades, tenant disputes, capacity reviews, and month-end close. When that evidence chain exists, GPU observability becomes more than monitoring. It becomes the operating record for AI-as-a-Service.

External References

Exit mobile version