Telemetry flow¶

The observability layer in the Edge Computing LLM stack combines infrastructure metrics, Kubernetes state, GPU telemetry, and application instrumentation into a unified pipeline. This pipeline ensures that you have visibility from the bare metal up to the LLM inference requests.

Comprehensive Architecture¶

The following diagram illustrates the complete telemetry flow, detailing ports, protocols, and data direction.

flowchart LR
    subgraph Layer 0: Host
        NE[node-exporter]
        DCGM[DCGM Exporter]
    end

    subgraph Layer 1: Kubernetes State
        KSM[kube-state-metrics]
        API[Kubernetes API / kubelet]
    end

    subgraph Layer 2: Applications
        Ollama[Ollama Server]
        WebUI[Open WebUI]
    end

    subgraph Layer 3: Telemetry Pipeline
        OTEL[OpenTelemetry Collector]
        P[Prometheus]
        G[Grafana]
        Alerts[Alertmanager]
    end

    NE -- "HTTP 9100\n(Host Metrics)" --> P
    DCGM -- "HTTP 9400\n(GPU Metrics)" --> P
    KSM -- "HTTP 8080\n(K8s State)" --> P
    API -- "HTTPS 443\n(cAdvisor)" --> P

    Ollama -- "OTLP gRPC 4317 / HTTP 4318" --> OTEL
    WebUI -- "OTLP gRPC 4317 / HTTP 4318" --> OTEL

    OTEL -- "Prometheus Exporter\n(HTTP 8889)" --> P

    P -- "PromQL\n(HTTP 9090)" --> G
    P -- "Alerts" --> Alerts

Telemetry Signals¶

• Node metrics: Gathered by node-exporter. Includes CPU usage, system memory, filesystem utilization, network I/O, and host resource pressure.
• Kubernetes state: Gathered by kube-state-metrics. Compares desired versus current workload state (e.g., Pending pods, crash loops).
• GPU metrics: Gathered by DCGM exporter. Includes VRAM usage, GPU utilization, temperature, power draw, and process context (where supported by the GPU/driver/DCGM combination).
• Application metrics/traces: Gathered via the OpenTelemetry (OTel) Collector. Includes request latency, token generation speed, and component behavior when explicitly instrumented.

[!WARNING] Logs are not automatically safe evidence. Application logs can contain user prompts, model responses, headers, private filesystem paths, and IP addresses. The telemetry pipeline prioritizes metrics over logs for automated alerting to avoid exposing PII.

Pipeline Components¶

1. OpenTelemetry Collector¶

The OTel Collector serves as the ingestion point for application traces and metrics. - OTLP gRPC: Port 4317 - OTLP HTTP: Port 4318 - Prometheus Exporter: The collector translates OTel metrics and exposes them on port 8889 for Prometheus to scrape.

2. Prometheus Scrape Targets¶

Prometheus is configured dynamically via the Prometheus Operator. It uses ServiceMonitor and PodMonitor Custom Resource Definitions (CRDs) to discover endpoints.

Data retention is typically configured to 15 days for edge deployments to preserve disk space, but this can be adjusted in the Helm values.

Validating the Pipeline¶

You must verify the telemetry pipeline from end to end to ensure data is flowing correctly.

Step 1: Check Custom Resources¶

Ensure the service monitors are deployed:

kubectl get servicemonitor,podmonitor,prometheusrule -n llm-observability

Step 2: Check Component Readiness¶

kubectl get pods -n llm-observability

Step 3: Validate Prometheus Targets¶

Port-forward to Prometheus to inspect its internal state:

kubectl -n llm-observability port-forward svc/prometheus-operated 9090:9090 &

Then query the ready endpoint and the targets list:

curl --fail http://127.0.0.1:9090/-/ready
curl -s http://127.0.0.1:9090/api/v1/targets | grep -o '"health":"up"' | wc -l

[!TIP] In the Prometheus UI (http://127.0.0.1:9090/targets), review the up state by job before trusting a Grafana dashboard. A rendered dashboard with no data is not a healthy telemetry path—it's a broken pipeline.