GPU observability¶

GPU telemetry comes from the NVIDIA substrate via the DCGM exporter (Data Center GPU Manager). The exact metric set exposed to Prometheus depends on your specific hardware, the NVIDIA driver version, and DCGM feature support.

[!IMPORTANT] Some older consumer GPUs (e.g., earlier GeForce GTX series) expose only a subset of DCGM fields. Treat absent metrics as unsupported until verified, rather than assuming they are zeros. Keep host nvidia-smi and runtime checks in your acceptance matrix.

Three Views of GPU State¶

To properly diagnose an issue, you must verify the GPU state across three distinct layers. Use all three; none is sufficient alone.

View	Tool	What it proves
Host Layer	`nvidia-smi`	The physical driver sees the device, firmware is loaded, and reports point-in-time hardware usage.
Kubernetes Layer	`kubectl describe node`	The Kubelet sees the resource (`nvidia.com/gpu: 1`), and the scheduler can expose it to pods.
Model Runtime	`ollama ps`	The intended model is resident in memory, and you can see its CPU/GPU offload split.

DCGM Exporter Setup¶

The DCGM exporter is deployed as a DaemonSet within the gpu-operator namespace. It runs on every node equipped with an NVIDIA GPU and exposes metrics on port 9400.

A Prometheus ServiceMonitor automatically discovers this endpoint. You can manually verify the exporter:

# Port-forward the DCGM exporter service
kubectl port-forward svc/nvidia-dcgm-exporter -n gpu-operator 9400:9400 &

# Fetch the raw metrics
curl -s http://127.0.0.1:9400/metrics | grep DCGM_FI_DEV_GPU_UTIL

# Expected output format:
# # HELP DCGM_FI_DEV_GPU_UTIL GPU utilization (in %).
# # TYPE DCGM_FI_DEV_GPU_UTIL gauge
# DCGM_FI_DEV_GPU_UTIL{gpu="0",UUID="GPU-...",device="nvidia0"} 0

Core GPU Metric Catalog¶

These are the most critical metrics to track for LLM inference stability:

DCGM_FI_DEV_GPU_UTIL: Overall GPU processing utilization (percentage).
DCGM_FI_DEV_FB_USED: Framebuffer (VRAM) used in megabytes.
DCGM_FI_DEV_FB_TOTAL: Total available VRAM.
DCGM_FI_DEV_GPU_TEMP: Current GPU core temperature in Celsius.
DCGM_FI_DEV_POWER_USAGE: Current power draw in Watts.

Operational Measurements¶

When deploying a new model to the edge, record a baseline:

Idle Baseline: Record VRAM usage (DCGM_FI_DEV_FB_USED) before any model is loaded. This accounts for OS display servers and driver overhead.
Peak Usage: Record peak and steady VRAM usage during a controlled smoke request.
Model Configuration: Document the model profile, quantization level (e.g., Q4_K_M), context window size, batch size, and offloaded layer count (num_gpu).
Thermals: Observe temperature (DCGM_FI_DEV_GPU_TEMP) and any throttling during sustained, back-to-back request runs.
System RAM: Observe system RAM and swap behavior. If VRAM is exhausted, Ollama may push layers to system RAM, causing extreme latency spikes.

Guardrail Alerting¶

A 900 MiB operating target on a 1 GiB device should leave a healthy margin for the display, driver, and measurement noise.

[!TIP] Alert before you hit the target limit, not at the physical limit. A software parameter cannot enforce hard hardware isolation on a non-MIG GPU.

Example Prometheus Alerting Rules¶

You can configure Prometheus rules to alert when GPU health degrades:

groups:
- name: gpu-alerts
  rules:
  - alert: HighGPUTemperature
    expr: DCGM_FI_DEV_GPU_TEMP > 85
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "GPU {{ $labels.gpu }} is running hot"
      description: "GPU temperature is > 85C for 5 minutes. Check cooling."

  - alert: VRAMNearCapacity
    expr: (DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL) * 100 > 90
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "GPU {{ $labels.gpu }} VRAM > 90%"
      description: "VRAM is nearing exhaustion. The model may swap to system RAM or crash."