Skip to content

Metrics and dashboards

Prometheus acts as the measurement source and time-series database; Grafana serves as the presentation layer. Before interpreting dashboard panels, always validate the underlying queries directly in Prometheus to ensure data is flowing and accurately represents the edge environment.

Initial PromQL Checks

Before loading any dashboard, verify the baseline metrics in Prometheus. These queries confirm that the data sources are alive.

# Are the scrape jobs healthy?
up

# How many targets are up per job?
count by (job) (up)

# Are any containers crash-looping?
sum by (namespace) (kube_pod_container_status_restarts_total)

For a small edge node, it is critical to watch resource capacity versus requests:

# Node CPU Utilization (percentage)
1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))

# Node Memory Utilization (percentage)
1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

# Total Memory Requested by Pods
sum(kube_pod_container_resource_requests{resource="memory"})

[!NOTE] Query metric names can vary slightly depending on dependency versions (e.g., kube-state-metrics updates). Always use metric discovery in the deployed Prometheus UI (http://localhost:9090/graph) before hardcoding a permanent panel.

Dashboard Design Principles

A highly useful edge dashboard answers these critical operational questions at a glance:

  1. Host Health: Is the node online, and is system CPU/RAM usage safe?
  2. GPU Health: Is the GPU available, within thermal limits, and below VRAM guardrails?
  3. Application State: Is Ollama Ready and successfully serving the intended GGUF model?
  4. Performance: Are request rate and token-generation latency stable or degrading?
  5. Stability: Are restarts, CPU throttling, or storage pressure rising?
  6. Telemetry Health: Are the telemetry targets themselves dropping scrapes?

Edge LLM Observability Dashboard

The provided edge-llm-observability dashboard is pre-configured via Grafana provisioning. It includes the following specific sections.

GPU Metrics Visualization

Panels tracking NVIDIA hardware performance: - VRAM Utilization: Uses DCGM_FI_DEV_FB_USED vs DCGM_FI_DEV_FB_TOTAL. Crucial for ensuring models fit in memory. - GPU Compute: Tracks DCGM_FI_DEV_GPU_UTIL. High utilization during inference is normal; high utilization when idle indicates a runaway process. - Temperature & Power: Monitors DCGM_FI_DEV_GPU_TEMP. Essential for edge devices with limited cooling.

LLM Inference Latency Panels

Panels tracking Ollama and application performance (ingested via OTel or directly scraped): - Requests per Second (RPS): Overall traffic volume. - Time to First Token (TTFT): The latency before the first chunk of text is returned. High TTFT indicates slow prompt processing or cold-starts. - Tokens per Second (TPS): Generation speed. Drops in TPS often correlate with thermal throttling or GPU memory swapping.

Accessing Grafana

To view the dashboards:

  1. Port-forward the Grafana service: 
    kubectl port-forward svc/grafana -n llm-observability 8080:80
  2. Open your browser to http://localhost:8080.
  3. Log in with the configured credentials (default: admin / dynamically generated password, accessible via kubectl get secret).
  4. Navigate to Dashboards -> Edge Computing LLM.

Custom Dashboard Creation Guide

If you need to visualize new metrics (e.g., custom business logic from Open WebUI):

  1. Draft the Query: Test your PromQL in Prometheus first. Ensure you aggregate correctly (e.g., sum(rate(metric[5m]))).
  2. Create Panel: In Grafana, add a new panel to the dashboard. Paste the PromQL query.
  3. Format: Set the appropriate units (e.g., bytes, seconds, percent).
  4. Export: Do not rely on Grafana's internal database for long-term storage of dashboard definitions. Click Share -> Export -> Save to file and commit the JSON file to your repository under the Grafana provisioning configmap.