Architecture¶

The Edge-Computing-LLM platform provides a production-grade, local-first inference engine for GGUF-backed LLMs running on constrained edge infrastructure (Ubuntu, k3s, NVIDIA GPUs). The system enforces strict separation between host enablement, infrastructure orchestration, inference workload management, observability, and automated verification.

High-Level System Architecture¶

flowchart TB
  subgraph ControlPlane["Control Plane & Verification"]
    CLI["edge-cli<br/>Go Orchestration & Dry-Run Engine"]
    PROOF["edge-computing-openai (EdgeProof)<br/>Codex Verification & Evidence"]
    TST["edge-llm-tests<br/>Cross-Repository Verification Suite"]
    OBS["gguf-observability<br/>Runtime Contract Verifier"]
  end

  subgraph Layer2["Layer 2: Workload & Telemetry (llm-observability-stack)"]
    UI["Open WebUI<br/>(Port 8080)"]
    REDIS["Redis Cache<br/>(Port 6379)"]
    OLLAMA["Ollama Engine<br/>(Port 11434 / GGUF Models)"]
    OTEL["OpenTelemetry Collector<br/>(Ports 4317, 4318, 8888)"]
    PROM["Prometheus<br/>(Port 9090)"]
    GRAF["Grafana<br/>(Port 80 / Dashboards)"]
    ALERT["Alertmanager<br/>(Port 9093)"]
  end

  subgraph Layer1["Layer 1: Infrastructure Substrate (k3s-nvidia-edge)"]
    NFD["Node Feature Discovery<br/>(nfd-master, nfd-worker, nfd-gc)"]
    NVIDIA_OP["NVIDIA GPU Operator<br/>(gpu-operator)"]
    TOOLKIT["Container Toolkit DaemonSet"]
    PLUGIN["NVIDIA Device Plugin DaemonSet"]
    DCGM["DCGM Exporter DaemonSet<br/>(Port 9400)"]
    RUNTIME["RuntimeClass / nvidia"]
  end

  subgraph HostSubstrate["Host Hardware & System"]
    K3S["k3s Control Plane & containerd"]
    HOST["Ubuntu 22.04+ / Linux Host"]
    GPU["NVIDIA GPU (CUDA 12.8+ / Driver 535+)"]
  end

  CLI -->|Deploys / Manages| Layer1
  CLI -->|Deploys / Manages| Layer2
  PROOF -.->|Validates| Layer1
  PROOF -.->|Validates| Layer2
  TST -.->|Audits Contracts| Layer1
  TST -.->|Audits Contracts| Layer2
  OBS -.->|Verifies GGUF Resident State| OLLAMA

  UI --> OLLAMA
  UI --> REDIS
  OLLAMA -->|Requests GPU Allocations| RUNTIME
  OTEL -->|Metrics & Traces| PROM
  PROM -->|Scrapes| DCGM
  PROM -->|Scrapes| OLLAMA
  GRAF -->|Queries| PROM
  ALERT -->|Evaluates Alerts| PROM

  Layer1 --> K3S
  K3S --> HOST
  HOST --> GPU

Architectural Layer Boundaries¶

1. Control Plane & CLI (edge-cli)¶

edge-cli acts as the unified, Go-based operator entry point. It orchestrates deployments without duplicating Helm chart logic: - Preflight Diagnostics: Validates Linux kernel, Go runtime, Helm dependencies, kubectl context, and nvidia-smi accessibility. - Dynamic Accelerator Selection: Automatically detects hardware capabilities. Defaults to --accelerator auto, falling back to CPU profiles when NVIDIA hardware is unavailable or disabled. - Dry-Run Safety: Pre-renders Helm values and manifest changes before executing mutating cluster operations. - Lifecycle Management: Provides explicit commands for doctor, install, status, logs, validate, and uninstall.

2. Infrastructure Substrate (k3s-nvidia-edge)¶

Layer 1 installs and configures the Kubernetes GPU substrate in the gpu-operator namespace: - Node Feature Discovery (NFD): Labels cluster nodes with hardware attributes (e.g., feature.node.kubernetes.io/pci-10de.present=true). - NVIDIA GPU Operator: Manages the lifecycle of GPU drivers, container runtime hooks, and monitoring tools. - NVIDIA Container Toolkit: Configures containerd to support CUDA runtimes seamlessly via nvidia-container-runtime. - RuntimeClass (nvidia): Registers the NVIDIA OCI runtime handler with k3s. - NVIDIA Device Plugin: Exposes nvidia.com/gpu allocatable capacity to Kubernetes pod schedulers. - DCGM Exporter: Emits low-overhead GPU metrics (memory, utilization, temperature, power, PCIe bandwidth) on port 9400.

3. Inference & Observability Workloads (llm-observability-stack)¶

Layer 2 manages application workloads in the llm-observability-stack namespace: - Inference Engine (Ollama): Serves GGUF quantized models (e.g., Gemma 3 1B, Llama 3.2 1B, Qwen 1.8B). Supports low-VRAM profiles (offloading layers up to operational memory ceilings like 900 MiB). - User Interface (Open WebUI & Redis): Browser UI connecting to Ollama on port 11434, utilizing Redis (port 6379) for web session storage and prompt caching. - OpenTelemetry Collector & Operator: Receives GenAI traces and metrics on OTLP gRPC (4317) and HTTP (4318) ports, forwarding sanitized telemetry to Prometheus. - Kube-Prometheus-Stack: Deploys Prometheus (9090), Grafana (80), Alertmanager (9093), node-exporter (9100), and kube-state-metrics (8080) configured with custom edge LLM dashboards.

4. Verification & Evidence Plane (edge-llm-tests, gguf-observability, edge-computing-openai)¶

• edge-llm-tests: Reusable Go test harness inspecting module integrity, Helm renders, host state, and running pods.
• gguf-observability (gguf-observe): Read-only binary asserting that the active model contract (num_gpu, num_ctx, num_batch) matches expected memory ceiling parameters.
• EdgeProof (edge-computing-openai): Integrates an OpenAI Codex Agent Skill with Go verifiers to evaluate platform health and generate reproducible JSON evidence.

Detailed Data & Telemetry Flow¶

sequenceDiagram
  autonumber
  actor User as Local Operator
  participant UI as Open WebUI (8080)
  participant OLL as Ollama Engine (11434)
  participant GPU as NVIDIA GPU (CUDA)
  participant DCGM as DCGM Exporter (9400)
  participant OTEL as OTel Collector (4317/4318)
  participant PROM as Prometheus (9090)
  participant GRAF as Grafana (80)

  User->>UI: Submit Prompt ("Explain k3s GPU Operator")
  UI->>OLL: POST /api/generate (Stream=true)
  OLL->>GPU: Execute matrix operations (CUDA kernels)
  GPU-->>OLL: Return tensor output
  OLL-->>UI: Stream response tokens
  UI-->>User: Render markdown response

  par Background Monitoring
    DCGM->>GPU: Query NVML metrics
    PROM->>DCGM: Scrape /metrics (GPU memory, % utilization)
    PROM->>OLL: Scrape /metrics (Prompt tokens/sec, evaluation time)
    OLL->>OTEL: Export GenAI traces (duration, prompt/completion tokens)
    OTEL->>PROM: Forward transformed metric series
    GRAF->>PROM: Query metrics for live dashboards
  end

Kubernetes Namespace Allocation Matrix¶