Who Used the GPU? Building Per Tenant Telemetry, Showback, and Capacity Evidence for AIaaS

Introduction

A shared GPU platform creates a deceptively simple question: who used the GPU?

The question becomes difficult as soon as the platform supports more than one operating model. A Kubernetes pod may receive an entire GPU, a MIG instance, or a time-sliced share. A virtual machine may receive a vGPU profile or a pass-through device. A model server may serve requests for several customers from the same process. A scheduler may let one department borrow another department’s unused quota. The GPU may look busy while the application produces little useful work, or it may appear lightly utilized while holding scarce memory capacity that prevents another model from starting.

This is why a single GPU utilization graph cannot support tenant accountability. NVIDIA DCGM and DCGM Exporter are foundational components, but device telemetry does not automatically become department, project, model, or customer attribution [1]-[4]. Kubernetes metadata tells you which pod received a device, but not whether the pod produced valuable work [5], [6]. Model-server metrics expose tokens, requests, latency, and queue depth, but they do not establish who funded the underlying capacity [12]-[15]. A finance ledger can assign cost, but only after the platform has reliable allocation and identity evidence.

An enterprise AI-as-a-Service platform therefore needs an evidence chain, not a dashboard shortcut. That evidence chain must connect business ownership to scheduler decisions, workload identity, device identifiers, hardware telemetry, model-server outcomes, rate cards, and retained monthly records. It must also preserve uncertainty when attribution is approximate, especially for time-sliced GPUs, shared model processes, short-lived workloads, and unsupported hardware paths. Community discussions around emerging GPU observability tools reinforce the operational demand for attribution that goes beyond fleet-wide graphs [24].

TL;DR

A defensible GPU showback system measures four different things and never treats them as synonyms:

• Allocation identifies who controlled or reserved GPU capacity during a time window.
• Utilization measures what the GPU hardware was doing during sampled intervals.
• Productive work measures the service outcome, such as successful requests, generated tokens, completed training steps, or processed images.
• Financial consumption applies an approved cost policy to allocation, variable usage, shared overhead, and service outcomes.

Use DCGM and DCGM Exporter for device health and activity, Kubernetes PodResources or hypervisor inventory for ownership mapping, model-server metrics for workload outcomes, Prometheus for collection and recording rules, Grafana for role-specific dashboards, Alertmanager for actionable routing, and a separate financial evidence store for monthly showback or chargeback. Bill exact capacity from allocation records. Use utilization to expose efficiency. Use productive-work metrics to explain value. Mark process-level and time-slicing attribution with an explicit confidence level rather than presenting modeled estimates as exact measurements.

The Four Measurements That Must Not Be Collapsed

The most important design decision is conceptual, not technical. The platform must maintain separate definitions for allocation, utilization, productive work, and financial consumption.

Measurement	Primary question	Typical source	Useful unit	What it does not prove
Allocation	Who controlled capacity?	Scheduler, kubelet, device plugin, DRA, hypervisor, quota manager	GPU-hours, MIG-profile-hours, vGPU-profile-hours	That the device was busy
Utilization	What was the device doing?	DCGM, NVML, vGPU host metrics	SM active ratio, GPU busy ratio, memory used, watts	That useful business work completed
Productive work	What service outcome was delivered?	Triton, vLLM, NIM, training framework, application telemetry	Requests, tokens, samples, steps, jobs, successful outcomes	Who should pay for reserved capacity
Financial consumption	Which cost policy applies?	Rate card, asset model, scheduler ledger, finance rules	Currency, cost per GPU-hour, cost per request, cost per token	The physical reason for poor efficiency

Allocation

Allocation is a control-plane fact. It answers who had the right to use capacity, not whether they exercised that right efficiently.

For an exclusive Kubernetes allocation, the unit may be one GPU assigned to a pod for three hours. For MIG, the unit should preserve the profile, such as a specific MIG profile held for three hours. For vGPU, the unit should preserve the vGPU profile and VM assignment. For time-slicing, the unit should state the configured replica or share policy rather than pretending the tenant owned an isolated fraction of the silicon.

A general allocation equation is:

Allocated GPU-equivalent hours
  = integral of assigned capacity fraction over elapsed time

The capacity fraction must come from an approved inventory model. It should not be inferred from a momentary utilization percentage.

Utilization

Utilization is sampled device behavior. Depending on the metric, it may describe the fraction of time the GPU was active, the fraction of streaming multiprocessors with at least one active warp, memory-copy engine activity, framebuffer memory consumption, power draw, or another device condition.

Utilization is useful for identifying idle reservations, saturation, poor batching, thermal throttling, memory pressure, or workload inefficiency. It is not a universal billing unit. A tenant may be financially responsible for a reserved GPU even when utilization is low, just as a reserved virtual machine still has a cost when its CPU is quiet.

An efficiency-oriented derived measure can be useful:

Active-equivalent GPU-hours
  = integral of allocated capacity fraction
    x normalized active ratio
    over elapsed time

This is an analytical measure, not automatically an invoice measure. The organization must decide whether active-equivalent hours affect pricing, efficiency targets, or only operational reporting.

Productive Work

Productive work is application-specific. For an inference service, productive work may include successful requests, input tokens, output tokens, images generated, embeddings produced, or requests meeting a service-level objective. For training, it may include completed steps, samples processed, checkpoints produced, or jobs completed successfully. For interactive development, a productive outcome may be harder to automate and may require a different service model.

The same GPU activity can produce very different amounts of useful work depending on model size, precision, batch size, sequence length, runtime, cache behavior, concurrency, and latency target. This is why tokens per second, requests per second, time to first token, queue delay, and success rate must sit beside GPU telemetry.

Financial Consumption

Financial consumption applies policy to evidence. It determines whether a tenant pays for guaranteed quota, actual allocation, borrowed capacity, energy, premium support, model-serving consumption, or shared platform overhead.

A defensible chargeback model is explicit:

Tenant charge
  = guaranteed capacity charge
  + borrowed allocation charge
  + variable service or energy charge
  + allocated shared-platform overhead
  - approved credits and adjustments

The platform team should be able to explain every term. A finance reviewer should be able to reproduce the monthly result. An application owner should be able to see why an idle reservation still cost money, why borrowed usage was treated differently, and why high device utilization did not necessarily translate into a low cost per successful outcome.

Build an Attribution Chain, Not a Label Collection

Per-tenant telemetry depends on a stable chain of identities across business, orchestration, workload, device, and service layers.

Pod names, process identifiers, and local GPU indices are not durable enough by themselves. Pods are recreated. Process identifiers are reused. GPU index zero means something different on every node. Namespace names can be reused after deletion. Model names may remain unchanged while the weight digest, quantization, runtime profile, or deployment configuration changes.

The evidence model should use stable identifiers and preserve time validity.

Layer	Recommended evidence keys	Why they matter
Business	tenant ID, department ID, customer ID, cost center, project ID, service owner	Establishes accountability and reporting hierarchy
Kubernetes	cluster ID, namespace UID, workload controller UID, pod UID, container ID, node name	Survives pod-name reuse and connects device allocation to workload ownership
Physical GPU	GPU UUID, node, PCI bus identity, GPU model, serial where available	Establishes the physical device and prevents local-index ambiguity
MIG	parent GPU UUID, GPU instance ID, compute instance ID, MIG device UUID, profile	Identifies the allocated partition and its capacity class
Virtualization	hypervisor cluster, host ID, VM UUID, vGPU UUID, vGPU profile, guest instance ID	Connects host-side vGPU evidence to the correct VM and tenant
Model service	endpoint ID, model name, model version or digest, runtime, deployment revision	Connects hardware activity to the service delivering work
Request or job	trace ID, request ID, job ID, queue ID, user or service identity	Supports detailed investigation outside high-cardinality metric labels
Finance	rate-card version, allocation class, billing period, adjustment ID	Makes financial output reproducible

Preserve Time-Valid Ownership

The platform should maintain an ownership mapping with valid_from and valid_to timestamps. When a pod is deleted, a VM is reassigned, a namespace changes owners, or a vGPU moves, the old relationship must remain queryable for the period in which it was true.

This temporal mapping is essential for month-end close. A current CMDB lookup cannot reliably explain who owned a workload three weeks earlier. The evidence pipeline should capture assignment events as they happen and retain them independently from the live orchestration system.

Keep High-Cardinality Identity Out of Prometheus Labels

Prometheus labels are effective for bounded dimensions such as cluster, namespace, model, service tier, GPU class, and a manageable set of tenant identifiers. They are a poor place for request IDs, user email addresses, raw process IDs, session IDs, or arbitrary customer strings because every unique label combination creates another time series [8].

Use metrics for bounded aggregation and alerting. Use logs, traces, or an analytical event store for per-request and per-process detail. Join those records into the financial ledger during controlled aggregation rather than multiplying the number of Prometheus time series without limit.

Reference Telemetry Architecture

The reference architecture separates collection, identity enrichment, operational analytics, and financial evidence. Notice that Kubernetes and virtual-machine paths converge only after each has produced a reliable device-to-workload mapping.

What Each Component Owns

• DCGM and DCGM Exporter own hardware telemetry and device health evidence.
• Kubernetes PodResources, device allocation data, and hypervisor inventory own device-to-workload assignment evidence.
• Kubernetes metadata and business mappings own workload-to-tenant attribution.
• Model servers and applications own productive-work and service-level evidence.
• Prometheus owns time-series collection, short-term query, recording rules, and alert-rule evaluation.
• Grafana owns reusable operational and showback views, not the source-of-truth billing calculation.
• Alertmanager owns deduplication, grouping, silencing, inhibition, and receiver routing for Prometheus alerts.
• The financial ledger owns closed-period cost evidence, rate-card versions, adjustments, and reconciliation.

What DCGM and DCGM Exporter Actually Measure

NVIDIA Data Center GPU Manager provides the device-level telemetry, health, diagnostics, policy, profiling, and accounting capabilities that make GPU operations observable [2], [3]. DCGM Exporter exposes selected DCGM fields in Prometheus format and is commonly deployed as a DaemonSet on GPU nodes [1].

The exporter is a collection layer. It does not replace the identity model, scheduler ledger, or model-server telemetry.

Core Metric Domains

Domain	Example evidence	Operational use
Identity and inventory	GPU UUID, device model, node, MIG identifiers and profile	Device mapping, inventory reconciliation, capacity segmentation
Memory	framebuffer total, used, free, reserved, memory-copy activity	Model fit, memory pressure, stranded memory, leak detection
Compute activity	GPU active ratio, SM activity, SM occupancy or related profiling fields	Idle detection, saturation analysis, kernel efficiency clues
Power and thermals	power draw, temperature, memory temperature, clock values, clock-event reasons	Cooling issues, throttling, energy estimates, performance anomalies
Interconnect	PCIe and NVLink counters or throughput where supported	Data-path bottlenecks, topology issues, multi-GPU diagnosis
Reliability	XID incidents, ECC errors, row-remap or memory-repair indicators, device health	Incident response, quarantine, warranty and support evidence
Process and job accounting	process activity, process memory, job statistics where enabled and supported	Attribution refinement and workload summaries

Metric Names Are Versioned Interfaces

DCGM field names and exporter aliases evolve. A release may rename labels, add cumulative counters, change field aliases, or improve MIG and Kubernetes metadata behavior [4]. The platform should not let raw vendor metric names spread through every dashboard and financial query.

Use a normalization layer based on Prometheus recording rules or a telemetry transform. For example:

groups:
  - name: ai-gpu-normalization
    interval: 30s
    rules:
      - record: ai_gpu:sm_active_ratio
        expr: clamp_max(DCGM_FI_PROF_SM_ACTIVE, 1)

      - record: ai_gpu:framebuffer_used_bytes
        expr: DCGM_FI_DEV_FB_USED * 1024 * 1024

      - record: ai_gpu:power_watts
        expr: DCGM_FI_DEV_POWER_USAGE

      - record: ai_gpu:temperature_celsius
        expr: DCGM_FI_DEV_GPU_TEMP

The example assumes the listed exporter aliases exist in the deployed release. Validate the active collector CSV, exporter output, units, labels, and field support on each GPU generation before promoting these rules. The stable contract for dashboards should be the normalized ai_gpu:* series, not an unreviewed dependency on every upstream field name.

SM Activity Is Not Business Productivity

A high SM activity ratio indicates that streaming multiprocessors were active during the sample window. It does not prove that kernels were efficient, that memory access was optimal, that the model met latency targets, or that the work produced a successful business result.

Low activity can indicate idle reservation, input starvation, CPU or storage bottlenecks, small batches, synchronization, or an interactive workload waiting for a user. High activity can coexist with low requests per second, excessive time to first token, retries, or failed requests.

Sampling Can Miss Short Work

GPU telemetry is sampled. Short kernels and short-lived processes may start and finish between samples. Process-level attribution therefore has a confidence boundary even when the platform exposes process metrics. Increase collection frequency only after measuring the storage, CPU, and cardinality cost. High-frequency raw metrics may belong in short retention, while aggregated evidence belongs in long retention.

Kubernetes Attribution from Device Assignment to Tenant

Kubernetes attribution requires several control-plane facts to line up:

1. A device plugin or Dynamic Resource Allocation driver advertises GPU resources [6].
2. The scheduler selects a node based on the resource request, policy, topology, and available capacity.
3. The kubelet assigns device identifiers to containers.
4. The local PodResources API exposes container and exclusive-resource assignments to monitoring agents [5].
5. DCGM Exporter maps device telemetry to pod and container metadata where the allocation mode and release support that mapping.
6. Kubernetes metadata is joined to approved tenant, project, service, and cost-center identifiers.
7. Allocation events are retained after the pod disappears.

Use Pod UID and Controller UID, Not Pod Name Alone

A deployment may create many pods with changing names. A job may be retried. An operator may recreate a model-serving pod under the same logical service. The ownership record should therefore preserve:

• namespace UID
• pod UID
• container ID
• owning controller UID and kind
• workload revision
• node
• allocated device UUID or MIG identity
• assignment start and end time

The human-readable namespace, deployment, and pod names remain useful labels, but the immutable identifiers are the evidence keys.

Create a Bounded Identity Metric

One practical pattern is a small metadata exporter that emits a gauge with value 1 for each active workload identity. The gauge carries only allowlisted, bounded labels.

ai_workload_identity{
  cluster="prod-ai-1",
  namespace="risk-models",
  pod="fraud-llm-7c9d8",
  pod_uid="4d31...",
  workload_uid="7a4c...",
  tenant_id="finance",
  project_id="fraud-platform",
  service_tier="production"
} 1

Recording rules can then enrich normalized GPU series:

groups:
  - name: ai-gpu-tenant-attribution
    rules:
      - record: ai_tenant_gpu:sm_active_ratio
        expr: |
          ai_gpu:sm_active_ratio
            * on (cluster, namespace, pod)
              group_left(tenant_id, project_id, service_tier, pod_uid, workload_uid)
              ai_workload_identity

      - record: ai_tenant_gpu:framebuffer_used_bytes
        expr: |
          ai_gpu:framebuffer_used_bytes
            * on (cluster, namespace, pod)
              group_left(tenant_id, project_id, service_tier, pod_uid, workload_uid)
              ai_workload_identity

This design gives the platform team a deliberate schema and a testable failure mode. When a GPU series cannot join to an identity series, it becomes an unattributed telemetry incident instead of silently landing in the wrong tenant report.

Queue Time Has Multiple Meanings

A single panel named “queue time” is not enough.

Queue or delay	Start	End	What it reveals
Admission delay	Workload submitted	Workload admitted	Policy, quota, approval, or admission-control pressure
Scheduler delay	Pod created or job queued	Node assignment	Capacity shortage, topology constraints, fairness, priority, or fragmentation
Startup delay	Node assignment	Container or endpoint ready	Image pull, model load, storage, initialization, or health-check delay
Model-server queue	Request accepted	Request scheduled for execution	Concurrency, batching, KV cache, runtime, or GPU saturation
End-to-end latency	Client request	Final response	Full service experience, including network and application layers

Capacity forecasting should treat sustained scheduler delay as unmet demand. Model-server queue delay is a service bottleneck and may occur even when Kubernetes has already allocated the GPU.

MIG Attribution Is Stronger Than Time-Slicing, but the Math Still Matters

MIG creates hardware-isolated GPU instances with identifiable profiles and instance identifiers. This makes allocation attribution stronger than time-slicing because the platform can associate a specific MIG device with a pod or VM.

The billing unit should be the allocated MIG profile over time:

MIG allocation cost
  = MIG-profile-hours x approved profile rate

Do not bill a MIG tenant from the parent GPU utilization percentage. Different instance profiles have different capacity denominators, and parent-device activity is not necessarily a simple sum or average of instance percentages.

Capacity-Normalized Utilization

For fleet planning, the platform may calculate a GPU-equivalent active measure:

Capacity-normalized active equivalent
  = sum for each MIG instance of
      profile capacity share x instance active ratio

The profile capacity share must come from an approved profile inventory and validation method. It should not be guessed from framebuffer memory alone. Memory, compute slices, media engines, and supported metrics do not always scale identically. Community questions involving heterogeneous MIG layouts also demonstrate why a simple weighted reconstruction of parent utilization should not be treated as authoritative without validation [23].

Keep three distinct views:

• Physical GPU view: health, total power, thermals, parent-device activity, and failure domain.
• MIG allocation view: profile, assigned tenant, profile-hours, and stranded profile capacity.
• MIG workload view: instance activity, memory, model metrics, queue time, and productive work.

Fragmentation Is Its Own Capacity State

A GPU may have free aggregate capacity but still be unable to satisfy a requested MIG profile. That is fragmentation, not ordinary utilization. Capacity dashboards should report:

• unallocated physical GPU equivalents
• available profiles by GPU model
• requested profiles waiting in queue
• capacity trapped in incompatible layouts
• reconfiguration opportunities and disruption cost

Without this view, a platform can report low average utilization while users experience long queue times.

Time-Slicing Attribution Requires an Explicit Confidence Boundary

Time-slicing lets multiple workloads share a physical GPU through scheduling rather than hard partitioning. It can improve accessibility for development and bursty work, but it complicates attribution.

A physical-device metric may be repeated across time-slice replicas because the underlying measurement belongs to the GPU, not to each logical replica. Dividing the physical utilization equally among configured replicas is not exact attribution. It is a modeled allocation assumption.

Recent DCGM Exporter releases added per-process GPU metrics for time-sharing and MIG, and newer releases continue to improve Kubernetes and container-runtime mapping [4]. That does not make process attribution universal. Hardware support, driver support, DCGM support, exporter configuration, process-accounting support, container-runtime mapping, and sampling behavior still determine what evidence is available.

Process-Level Attribution Chain

A process-level design needs to resolve:

GPU process sample
      -> host PID
      -> container cgroup or runtime container ID
      -> pod UID or VM UUID
      -> model-server worker
      -> tenant, project, model, and endpoint

Failure can occur at every step. PIDs are reused. Processes terminate between samples. Multi-process services fork workers. NVIDIA MPS can change process behavior. A shared model server may intentionally serve several tenants from one process. In that last case, process telemetry can attribute activity to the service but not accurately divide it among customers. Request and token records must perform the customer split.

Attribution Confidence Levels

Confidence	Typical allocation mode	Evidence quality	Appropriate use
High	Exclusive GPU, MIG device, dedicated vGPU	Stable device-to-workload mapping	Chargeback, SLO analysis, audit evidence
Medium	Supported process mapping on shared GPU	Sampled PID-to-container mapping with known gaps	Showback, efficiency analysis, provisional allocation
Modeled	Time-slice replicas without complete process evidence	Policy-based proportional split or equal-share assumption	Capacity planning with disclosure, not exact billing
Unattributed	Missing metadata, unsupported device, telemetry gap	No defensible mapping	Platform exception queue and cost suspense account

A community report involving Blackwell GB10 is a useful warning. The visible symptom looked like missing per-process attribution, but the vendor response identified the platform as unsupported by DCGM [22]. The operational lesson is to validate the exact GPU and software support boundary before treating exporter output as proof of a general product limitation.

vGPU and VM Attribution Needs Host and Guest Evidence

Virtualized GPU environments need two connected views.

The host or hypervisor view establishes the physical GPU, vGPU instance, vGPU profile, VM UUID, and assignment interval. NVIDIA’s management interfaces can expose vGPU-instance and vGPU-process utilization where supported [16], [17]. The guest view provides telemetry scoped to the VM and, when applicable, the model server or application running inside it.

Recommended vGPU Evidence Chain

Physical GPU UUID
   -> vGPU UUID and profile
   -> hypervisor VM UUID
   -> VM owner and cost center
   -> guest workload or model endpoint
   -> requests, tokens, latency, and outcomes

Use vGPU-profile-hours for allocation cost. Use vGPU activity for efficiency. Use guest model metrics for productive work.

Watch the Utilization Denominator

A utilization percentage reported inside a vGPU can be relative to the vGPU’s permitted engine capacity, depending on the scheduler. A VM that fully uses its allocated share may report 100 percent even though it is consuming only part of the physical GPU. Host and guest percentages therefore need documented semantics before they are placed on the same chart.

Avoid Double Counting

If host-side vGPU telemetry and guest-side DCGM or NVML telemetry describe the same activity, do not sum them. Use one as the authoritative allocation and device view, and the other as a reconciliation and service detail view.

Pass-through devices require another boundary. The vGPU management layer may not manage pass-through GPUs from the hypervisor, so guest telemetry and hypervisor assignment records become the primary evidence sources.

Model-Server Metrics Convert GPU Activity into Productive Work

Device telemetry explains hardware behavior. Model-server telemetry explains service behavior.

Triton Inference Server exposes request counts, inference counts, execution counts, request timing, and GPU statistics. Its inference and execution counters can also reveal effective batching behavior [12]. vLLM exposes request, token, queue, latency, KV cache, and optional model-flops metrics [13], [14]. NVIDIA NIM for large language models provides Prometheus-compatible metrics and, for supported runtime paths, surfaces runtime-native metrics rather than inventing a separate semantic layer [15].

Core Inference Metric Definitions

Metric	Definition	Why it matters
Requests per second	Rate of accepted, successful, failed, or completed request counters	Demand, throughput, errors, service capacity
Input tokens per second	Rate of prompt-token counters	Prefill workload and input intensity
Output tokens per second	Rate of generated-token counters	Decode throughput and productive output
Time to first token	Request timing to first generated token, using the runtime’s documented start point	User-perceived responsiveness and prefill pressure
Inter-token latency	Time between output tokens during decode	Streaming quality and decode consistency
End-to-end latency	Request arrival to completion	Full service experience
Model queue time	Time waiting in the runtime scheduler before execution	Saturation, batching, concurrency, KV cache pressure
Running requests	Requests actively executing or in an execution batch	Concurrency and active load
Waiting requests	Requests queued for execution	Backlog and unmet service demand
KV cache usage	Fraction or bytes of key-value cache occupied	Concurrency limit, sequence pressure, eviction risk
Average dynamic batch size	Inferences divided by executions when semantics support the calculation	Batching efficiency and throughput behavior
Success rate	Successful requests divided by total completed requests	Productive output quality gate
Per-model latency	Latency histogram segmented by model and deployment revision	Model-specific SLO and regression analysis

Token and Request Rates

Use counters and calculate rates in the query layer:

sum by (tenant_id, model) (
  rate(ai_workload:output_tokens_total[5m])
)

sum by (tenant_id, model) (
  rate(ai_workload:requests_total{outcome="success"}[5m])
)

For latency, use histograms and agreed percentile windows:

histogram_quantile(
  0.95,
  sum by (le, tenant_id, model) (
    rate(ai_workload:ttft_seconds_bucket[10m])
  )
)

Do Not Put Request IDs in Prometheus

Per-request metrics are useful, but they should normally be returned in the response, written to structured logs, or attached to distributed traces. A request ID as a Prometheus label creates an unbounded series dimension.

A practical split is:

• Prometheus: model, endpoint, tenant class, project, outcome, latency bucket, and bounded service dimensions.
• Logs or traces: request ID, caller identity, prompt and output token counts, exact model revision, queue time, TTFT, retry chain, and failure detail.
• Financial aggregation: daily or hourly tenant totals produced from immutable request events.

Shared Model Servers Need Request-Level Tenant Accounting

A single model-server process can serve multiple tenants. Device metrics and process metrics can attribute GPU activity to the server, but not necessarily to each customer. For this pattern, customer showback should be based on request and token evidence, then reconcile against the service’s allocated GPU cost.

One useful approach is to calculate a tenant’s share of a shared endpoint using an approved workload driver:

Tenant service share
  = tenant weighted productive units
    / total endpoint weighted productive units

Weighted units may account for input tokens, output tokens, image size, model class, priority tier, or validated compute coefficients. The coefficient model must be versioned and tested. A raw output-token count alone may not fairly represent prefill-heavy or multimodal workloads.

Normalize Metrics Before Building Dashboards or Bills

Raw sources expose different names, units, labels, and update intervals. A platform-wide semantic layer prevents every dashboard from inventing its own interpretation.

A useful canonical contract includes:

Canonical series	Type	Required bounded dimensions	Purpose
ai_gpu:allocation_fraction	Gauge or interval record	cluster, tenant, project, workload, GPU class, allocation class	Assigned capacity
ai_gpu:sm_active_ratio	Gauge	cluster, node, GPU or instance identity	Normalized activity
ai_gpu:framebuffer_used_bytes	Gauge	cluster, device identity	Memory occupancy
ai_gpu:power_watts	Gauge	cluster, physical GPU	Power observation
ai_gpu:health_incidents_total	Counter	cluster, physical GPU, category, severity	Reliability evidence
ai_scheduler:queue_seconds	Histogram	cluster, queue, tenant, priority, GPU class	Scheduling delay
ai_workload:requests_total	Counter	tenant, project, model, endpoint, outcome	Service demand and success
ai_workload:input_tokens_total	Counter	tenant, model, endpoint	Prefill volume
ai_workload:output_tokens_total	Counter	tenant, model, endpoint	Generated output
ai_workload:ttft_seconds	Histogram	tenant class, model, endpoint	Responsiveness
ai_workload:e2e_latency_seconds	Histogram	tenant class, model, endpoint, outcome	Service-level evidence
ai_finance:gpu_rate_per_hour	Versioned ledger dimension	GPU class, profile, service tier, period	Cost calculation input

Prometheus should hold operational series. The financial rate record should normally live in a controlled data store and be exported into the reporting model, not edited casually as a dashboard constant.

Showback and Chargeback Need Different Governance

Showback reports consumption and cost without creating a financial transfer. Chargeback posts or reallocates cost to a tenant, department, or customer. The telemetry can be shared, but the governance bar is different.

Showback Can Tolerate Provisional Data

Showback can include near-real-time allocation, current utilization, idle reservations, queue delay, estimated cost, and attribution-confidence flags. It is useful for behavior change and capacity conversations.

Chargeback Requires a Closed Ledger

Chargeback should use a closed period with:

• approved rate-card version
• finalized allocation intervals
• late telemetry and correction policy
• unassigned-cost treatment
• guaranteed and borrowed capacity rules
• shared-overhead allocation rule
• credits and service-level adjustments
• finance and platform approval
• immutable monthly output

Guaranteed Quota Versus Borrowed Usage

A fair scheduler may let a tenant exceed guaranteed quota when other tenants are not using their entitlement [18], [19]. That creates at least two distinct capacity classes:

• Guaranteed capacity: capacity the platform promises to make available under defined conditions.
• Borrowed capacity: temporary over-quota capacity that can be reclaimed or preempted according to policy.

Report them separately:

Guaranteed GPU-hours
Borrowed GPU-hours
Preempted borrowed GPU-hours
Idle guaranteed GPU-hours
Productive units delivered by each capacity class

A common financial model charges guaranteed quota at a predictable committed rate and borrowed allocation at a burst rate. Another model charges only actual allocation but reports unused entitlement as a planning inefficiency. The correct choice is organizational policy, not a telemetry fact.

Classify Idle Capacity Correctly

Idle class	Description	Likely financial owner
Tenant-reserved idle	Guaranteed or dedicated capacity assigned but inactive	Tenant, unless centrally subsidized
Borrowed idle	Over-quota capacity assigned but no longer useful	Tenant until reclaimed, with aggressive scheduler cleanup
Platform unallocated idle	Capacity available to the shared pool	Platform or central AI service
Fragmentation idle	Capacity unusable because of MIG profile, topology, memory, or scheduling constraints	Platform, or shared with the tenant when caused by a dedicated design choice
Reliability reserve	Capacity deliberately held for failure, maintenance, or recovery	Platform overhead
Telemetry-unknown idle	Capacity appears idle because evidence is missing	Suspense category pending investigation

Calling all idle GPU time “waste” produces the wrong behavior. Cost frameworks such as OpenCost also distinguish shared, allocated, and idle cost treatment rather than collapsing them into one figure [20]. Reliability reserve is intentional. Tenant-reserved idle may be the cost of an availability promise. Fragmentation may require layout changes, not user coaching.

Example Grafana Dashboards

A useful dashboard system provides different views for platform operators, tenant owners, model-service teams, and finance. One giant dashboard will satisfy none of them.

Fleet Capacity and Health Dashboard

Primary audience: platform operations and hardware teams.

Recommended variables:

• cluster
• site
• GPU model
• node pool
• node
• physical GPU UUID
• MIG mode
• health severity

Recommended panels:

• installed, allocatable, allocated, unavailable, and unassigned GPU equivalents
• current MIG layouts and free profiles
• GPU memory used and reserved
• SM activity distribution
• power and temperature by node and rack
• thermal or power clock events
• XID, ECC, row-remap, PCIe, and NVLink health signals where supported
• telemetry freshness and missing exporter targets
• top nodes by fragmentation and queue pressure

Tenant Showback Dashboard

Primary audience: department, customer, project, and service owners.

Recommended variables:

• billing period
• tenant
• department
• project
• namespace or VM group
• service tier
• allocation class
• GPU or vGPU profile

Recommended panels:

• guaranteed, borrowed, and total allocated GPU-hours
• idle allocated GPU-hours
• active-equivalent GPU-hours
• requests, tokens, jobs, or other productive units
• cost by guaranteed, borrowed, variable, and overhead component
• cost per successful request, token, training job, or approved outcome
• queue time and preemption history
• attribution-confidence breakdown
• top idle workloads with owner and age

Model Efficiency and SLO Dashboard

Primary audience: model platform and application teams.

Recommended panels:

• requests per second
• input and output tokens per second
• p50, p95, and p99 TTFT
• inter-token latency and end-to-end latency
• runtime queue depth and queue time
• running and waiting requests
• KV cache usage
• GPU memory and SM activity aligned to service metrics
• failures, retries, cancellations, and SLO compliance
• cost per successful request and cost per million output tokens
• model, quantization, runtime, and deployment-revision comparison

Scheduler and Capacity Dashboard

Primary audience: platform engineering, capacity planning, and service management.

Recommended panels:

• pending GPU workloads by tenant, priority, GPU class, and age
• scheduler delay percentiles
• guaranteed quota versus borrowed consumption
• preemption and reclaim events
• available capacity that cannot satisfy queued requests
• MIG and topology fragmentation
• requested versus delivered GPU profiles
• forecast peak by week and month
• failure-domain reserve and maintenance headroom

Dashboard Wireframe

Grafana variables should make the dashboards reusable instead of duplicating one dashboard per tenant. Provision data sources, folders, dashboards, and alert integrations through version-controlled configuration [10], [11]. Apply role-based access so one customer cannot query another customer’s telemetry.

Alert Examples That Protect Evidence and Capacity

Alerts should identify actionable conditions, not merely display high values. Route hardware-health alerts to platform operations, tenant-efficiency alerts to service owners, and billing-evidence alerts to the telemetry team.

The following example uses normalized metrics. Thresholds are placeholders and must be validated by GPU model, workload class, service level, and operating policy.

groups:
  - name: aias-gpu-accountability
    rules:
      - alert: GPUAllocationMissingTenantIdentity
        expr: |
          ai_gpu:allocation_fraction > 0
            unless on (cluster, workload_uid)
          ai_workload_identity == 1
        for: 10m
        labels:
          severity: critical
          owner: telemetry-platform
        annotations:
          summary: Allocated GPU capacity has no tenant identity
          action: Quarantine the interval from chargeback and repair the mapping

      - alert: TenantGPUAllocatedButIdle
        expr: |
          avg_over_time(ai_tenant_gpu:allocation_fraction[30m]) > 0
            and on (cluster, tenant_id, workload_uid)
          avg_over_time(ai_tenant_gpu:sm_active_ratio[30m]) < 0.05
        for: 30m
        labels:
          severity: warning
          owner: tenant-platform
        annotations:
          summary: Tenant workload holds GPU capacity with minimal activity
          action: Validate workload intent before reclaiming or resizing capacity

      - alert: GPUHealthIncidentDetected
        expr: |
          increase(ai_gpu:health_incidents_total{severity=~"critical|fatal"}[5m]) > 0
        labels:
          severity: critical
          owner: gpu-operations
        annotations:
          summary: Critical GPU health incident detected
          action: Preserve evidence, evaluate workload evacuation, and follow the hardware runbook

      - alert: InferenceQueueSLOAtRisk
        expr: |
          histogram_quantile(
            0.95,
            sum by (le, cluster, endpoint, model) (
              rate(ai_workload:queue_seconds_bucket[10m])
            )
          ) > 30
        for: 10m
        labels:
          severity: warning
          owner: model-platform
        annotations:
          summary: Model-server queue delay threatens the service objective
          action: Check concurrency, batching, KV cache, replicas, and GPU saturation

      - alert: BorrowedGPUCapacityApproachingPolicyLimit
        expr: |
          ai_tenant_gpu:borrowed_allocation_fraction
            / ai_tenant_gpu:borrow_limit_fraction > 0.9
        for: 15m
        labels:
          severity: warning
          owner: capacity-management
        annotations:
          summary: Tenant borrowed GPU consumption is near its policy limit
          action: Review reclaim risk, queued demand, and production priority

Alertmanager should group related device failures by cluster and node, inhibit secondary tenant alerts when a node is already declared unavailable, and route financial-evidence failures differently from performance warnings [9]. Missing attribution is a critical data-quality alert because it can invalidate chargeback even when workloads continue running.

Capacity Forecasting Requires More Than Average Utilization

Average GPU utilization hides peaks, queue pressure, model mix, profile fragmentation, and failure reserve. Forecast at least four demand curves:

• Committed demand: guaranteed quota and dedicated allocations promised to tenants.
• Allocated demand: observed concurrent GPU equivalents, MIG profiles, and vGPU profiles.
• Active demand: capacity-normalized device activity during allocated periods.
• Productive demand: requests, tokens, training throughput, or other service outcomes.

Add a fifth curve for unmet demand, derived from pending workloads, scheduler delay, admission delay, and model-server queues.

Forecast by Capacity Class

Do not treat every GPU as interchangeable. Segment forecasts by:

• GPU model and memory size
• MIG profile
• vGPU profile
• interconnect and topology requirement
• security zone
• production, development, or batch service tier
• model family, precision, and runtime
• whole-GPU, MIG, time-slice, or vGPU operating mode

Use Validated Service Throughput

For an inference service, estimate capacity from a controlled benchmark that preserves the target SLO:

Required service GPUs
  = forecast peak productive throughput
    / validated sustainable throughput per GPU at the target SLO
    + failure and maintenance reserve

The validated denominator must match model, precision, runtime, batch policy, sequence distribution, concurrency, and hardware. A tokens-per-second number obtained under a different latency target is not a safe capacity input.

Queue Pressure Is Demand Evidence

When utilization looks moderate but queue time rises, investigate fragmentation, memory reservation, topology, admission policy, model loading, and runtime bottlenecks. Queue pressure can expose demand that utilization alone cannot see.

Evidence Retention and Auditability

Prometheus is excellent for operational metrics, but local Prometheus storage alone is not a financial evidence system. Long-term evidence needs durable retention, controlled schemas, and reproducible monthly close.

A practical design can use Prometheus for local collection and alerting, then remote-write or upload blocks into a durable Prometheus-compatible backend backed by object storage [21]. Request and trace events can use a separate analytical store. Monthly cost output should be materialized into an immutable ledger or governed warehouse table.

Example Retention Tiers

These are planning examples, not universal legal requirements:

Evidence	Example retention	Reason
High-resolution device and process metrics	30 to 90 days	Incident response, tuning, short-term attribution investigation
Five-minute operational rollups	13 to 25 months	Seasonality, trend, capacity, budget comparison
Allocation events and ownership history	At least the financial and audit horizon	Reconstructs who controlled capacity
Request and token aggregates	Based on customer, security, and finance policy	Productive-work showback and service economics
Monthly chargeback ledger	Commonly aligned to finance record-retention policy	Audit, reconciliation, dispute handling
Raw prompts and outputs	Only when required, minimized and protected	High privacy and data-governance risk

Preserve the Calculation Context

Each closed period should retain:

• metric schema version
• normalization-rule version
• rate-card version
• tenant and project hierarchy snapshot
• allocation-class definitions
• source-system versions
• data-gap and correction records
• attribution-confidence distribution
• total platform cost and reconciliation result
• approver and close timestamp

A future reviewer should be able to reproduce the charge without relying on the current state of Kubernetes, Grafana, or the hypervisor.

Implementation Sequence

A platform team can build this architecture incrementally, but the sequence matters.

Phase	Key work	Exit evidence
Identity contract	Define tenant, project, service, device, workload, and finance keys	Approved schema and ownership policy
Device inventory	Record physical GPU, MIG, vGPU, node, VM, and capacity-class relationships	Reconciled inventory with stable identifiers
Device telemetry	Deploy DCGM Exporter or supported VM telemetry and validate fields	Known-good metrics by GPU generation and allocation mode
Allocation capture	Persist Kubernetes, scheduler, quota, VM, and vGPU assignment intervals	No orphan allocation intervals in test cases
Metadata enrichment	Join device evidence to workload and business identities	Attribution confidence and exception reporting
Productive-work telemetry	Instrument Triton, vLLM, NIM, training, or application outcomes	Request, token, latency, queue, and success evidence
Semantic normalization	Create canonical metrics and recording rules	Versioned metric contract and unit tests
Showback pilot	Produce tenant dashboards and estimated cost	Tenant review and dispute findings
Chargeback readiness	Add rate cards, close controls, reconciliation, and corrections	Finance-approved monthly close process
Forecasting and optimization	Add peak, queue, profile, failure-reserve, and outcome forecasts	Capacity plan tied to service demand and SLOs

Start with Showback

Showback exposes data-quality problems without immediately creating financial disputes. Use the pilot to discover missing labels, pod-lifecycle gaps, ambiguous shared services, unsupported GPU modes, and confusing rate policies. Promote only high-confidence allocation classes into chargeback.

Validation Tests Before Financial Use

A telemetry architecture should be tested like a production control, not accepted because a dashboard displays data.

Exclusive GPU Test

Run one known pod or VM on one exclusive GPU. Confirm:

• the physical GPU UUID is correct
• assignment start and end times match the scheduler or hypervisor
• tenant and project mapping is correct
• memory, SM activity, power, and model metrics move together
• allocated GPU-hours reconcile to elapsed time

MIG Test

Create different MIG profiles and run controlled workloads. Confirm:

• each device maps to the correct profile and tenant
• instance metrics remain separate
• the parent GPU is not double counted
• capacity-normalized reporting uses approved profile weights
• reconfiguration and fragmentation events are visible

Time-Slicing Test

Run two workloads with distinct load patterns. Confirm whether the deployed stack provides process-level evidence. Compare observed attribution with controlled request or job totals. Mark the result as modeled when the system cannot isolate activity reliably.

vGPU Test

Map the physical GPU, vGPU UUID, profile, VM UUID, guest telemetry, and model endpoint. Confirm host and guest percentages use documented denominators and are not summed.

Shared Model-Service Test

Send requests from multiple tenants through one process. Confirm that request and token events divide customer consumption while device and process metrics remain assigned to the shared service. Reconcile tenant service shares to the endpoint’s total cost.

Lifecycle and Failure Tests

Test pod deletion, pod rescheduling, VM reassignment, exporter restart, Prometheus outage, clock skew, model revision change, node failure, telemetry gaps, and late-arriving data. A month-end process that works only during steady state is not ready for chargeback.

Reconciliation Controls

Use explicit equations:

Physical capacity hours
  = allocated hours
  + unallocated hours
  + unavailable or maintenance hours
  + approved reconciliation difference

Recoverable platform cost
  = tenant charges
  + central platform charges
  + approved subsidies
  - credits
  + approved reconciliation difference

Set a small tolerated difference, investigate exceptions, and prevent the period from closing when unattributed capacity or cost exceeds policy.

Operational Guardrails

A mature AIaaS telemetry service needs operating ownership.

• Platform operations own GPU health, node telemetry, exporter availability, and hardware incident response.
• Kubernetes or virtualization teams own allocation mappings and scheduler or hypervisor evidence.
• Model platform teams own request, token, queue, latency, and model-revision telemetry.
• FinOps or finance owns approved rates, overhead policy, and financial close.
• Tenant owners review idle reservations, service outcomes, and forecast demand.
• Security and privacy teams review customer identifiers, trace content, retention, and dashboard access.

Document the support boundary for every GPU generation, driver, DCGM release, exporter release, vGPU release, Kubernetes version, runtime, and hypervisor integration. A metric that works on one validated cluster should not be assumed to work on every device or virtualization path.

Conclusion

The question “who used the GPU?” has no reliable one-metric answer.

DCGM and DCGM Exporter can show what a supported GPU was doing. Kubernetes PodResources, scheduler evidence, device allocation records, and hypervisor inventory can show which workload controlled the device. Triton, vLLM, NIM, training frameworks, and application telemetry can show whether the workload produced useful service outcomes. A governed cost ledger can then apply guaranteed-quota, borrowed-capacity, variable-service, and shared-overhead rules.

The architecture becomes defensible when those measurements remain separate but connected. Allocation answers ownership. Utilization answers hardware activity. Productive work answers service value. Financial consumption answers policy-based cost responsibility.

That separation also makes the difficult cases manageable. MIG can be billed by profile-hours while utilization is analyzed per instance. vGPU can be billed by profile assignment while guest and host telemetry are reconciled. Time-slicing can carry a modeled confidence flag instead of an invented exact percentage. Shared model services can allocate customer cost through request and token evidence rather than trying to split a single process metric.

The platform team’s goal is not to produce the most colorful GPU dashboard. It is to create an evidence chain that survives pod deletion, process churn, node failure, model upgrades, tenant disputes, capacity reviews, and month-end close. When that evidence chain exists, GPU observability becomes more than monitoring. It becomes the operating record for AI-as-a-Service.

External References

• [1] NVIDIA: DCGM Exporter
  Canonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/latest/dcgm-exporter.html
• [2] NVIDIA: Feature Overview, NVIDIA DCGM Documentation
  Canonical URL: https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html
• [3] NVIDIA: Field Identifiers, NVIDIA DCGM Documentation
  Canonical URL: https://docs.nvidia.com/datacenter/dcgm/latest/dcgm-api/dcgm-api-field-ids.html
• [4] NVIDIA: NVIDIA/dcgm-exporter Releases
  Canonical URL: https://github.com/NVIDIA/dcgm-exporter/releases
• [5] Kubernetes: Kubernetes 1.28: Node podresources API Graduates to GA
  Canonical URL: https://kubernetes.io/blog/2023/08/23/kubelet-podresources-api-ga/
• [6] Kubernetes: Device Plugins
  Canonical URL: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/
• [7] Prometheus: Overview
  Canonical URL: https://prometheus.io/docs/introduction/overview/
• [8] Prometheus: Metric and Label Naming
  Canonical URL: https://prometheus.io/docs/practices/naming/
• [9] Prometheus: Alertmanager
  Canonical URL: https://prometheus.io/docs/alerting/latest/alertmanager/
• [10] Grafana Labs: Prometheus Template Variables
  Canonical URL: https://grafana.com/docs/grafana/latest/datasources/prometheus/template-variables/
• [11] Grafana Labs: Provision Grafana
  Canonical URL: https://grafana.com/docs/grafana/latest/administration/provisioning/
• [12] NVIDIA: Metrics, NVIDIA Triton Inference Server
  Canonical URL: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/metrics.html
• [13] vLLM Project: Production Metrics
  Canonical URL: https://docs.vllm.ai/en/v0.18.1/usage/metrics/
• [14] vLLM Project: Per-Request Metrics
  Canonical URL: https://docs.vllm.ai/en/latest/features/per_request_metrics/
• [15] NVIDIA: Logging and Observability, NVIDIA NIM for Large Language Models
  Canonical URL: https://docs.nvidia.com/nim/large-language-models/latest/reference/logging-and-observability.html
• [16] NVIDIA: Management SDK User Guide
  Canonical URL: https://docs.nvidia.com/vgpu/latest/grid-management-sdk-user-guide/index.html
• [17] NVIDIA: NVML API Reference Guide, vGPU Utilization and Accounting APIs
  Canonical URL: https://docs.nvidia.com/deploy/nvml-api/group__nvmlUtil.html
• [18] NVIDIA Run:ai: Over Quota, Fairness and Preemption
  Canonical URL: https://run-ai-docs.nvidia.com/saas/platform-management/runai-scheduler/scheduling/quick-starts/over-quota
• [19] NVIDIA Run:ai: The NVIDIA Run:ai Scheduler, Concepts and Principles
  Canonical URL: https://run-ai-docs.nvidia.com/saas/platform-management/runai-scheduler/scheduling/concepts-and-principles
• [20] OpenCost: OpenCost Specification
  Canonical URL: https://opencost.io/docs/specification/
• [21] Grafana Labs: Introduction to Grafana Mimir
  Canonical URL: https://grafana.com/docs/mimir/latest/introduction/
• [22] NVIDIA Developer Forums: DCGM-Exporter, Missing Process-Level Attribution for GPU Time-Slicing on Blackwell GB10
  Canonical URL: https://forums.developer.nvidia.com/t/dcgm-exporter-missing-process-level-attribution-for-gpu-time-slicing-on-blackwell-gb10/363037
• [23] NVIDIA Developer Forums: MIG Instances Utilization Calculation
  Canonical URL: https://forums.developer.nvidia.com/t/mig-instances-utilization-calculation/341194
• [24] Hacker News: Show HN, Neurox, GPU Observability for AI Infra
  Canonical URL: https://news.ycombinator.com/item?id=43835948