Skip to content

Orientation

Welcome to the Edge Computing LLM project. This platform is a collection of repositories and automation tools designed to turn a Linux host into a compact, highly observable Large Language Model (LLM) inference platform.

It is intentionally modularized and split into distinct layers. This separation ensures that an underlying infrastructure failure (like a GPU runtime crash) is easily distinguished from an application failure (like a model failing to load).

System Overview

The following diagram outlines the high-level components and their interactions within the edge environment.

flowchart LR
    subgraph Client Space
        U[Browser / User]
        API[API Client]
    end

    subgraph Layer 2: Workload
        W[Open WebUI]
        O[Ollama Server]
        M[(GGUF Models\nPersistent Volume)]
    end

    subgraph Layer 3: Observability
        P[Prometheus]
        D[Grafana]
        T[OTel Collector]
    end

    subgraph Layer 1: Substrate
        G[NVIDIA GPU]
        SYS[System RAM/CPU]
        X[DCGM Exporter]
        K[kube-state-metrics]
    end

    U -->|UI Access| W
    API -->|REST API| O
    W -->|Inference Requests| O

    O -->|Loads Weights| M
    O -->|Schedules Compute| G
    O -->|Fallback Compute| SYS

    X -.->|Metrics| P
    K -.->|State| P
    T -.->|Traces/Metrics| P
    O -.->|OTLP| T
    P -->|Visualizes| D

[!NOTE] The default local profile utilizes ClusterIP services and local kubectl port-forward sessions to prioritize security. Model weights reside on persistent storage. GPU offloading is configured conservatively by default; system RAM carries the remaining computation to prevent Out-Of-Memory (OOM) crashes.

What You Will Build

By completing this guide, you will assemble: 1. A single-node k3s Kubernetes cluster optimized for edge devices. 2. The NVIDIA GPU Operator configured for containerized GPU access. 3. Ollama serving quantized GGUF models. 4. Open WebUI for an intuitive, ChatGPT-like user experience. 5. A comprehensive Prometheus + Grafana telemetry stack to monitor hardware and model latency.

Enhanced Learning Path

Follow these steps in order to ensure a stable deployment. Do not skip the prerequisite checks!

  1. Host Preparation: Verify the host prerequisites. (Time estimate: 10 mins)
  2. Installation: Follow the fresh-clone quickstart to deploy the base layers. (Time estimate: 15 mins)
  3. Validation: Send a safe first inference to confirm the model is serving. (Time estimate: 5 mins)
  4. Deep Dive: Learn the architecture and ownership model to understand how the components interact. (Time estimate: 20 mins)
  5. Day 2 Operations: Adopt the operations runbook for scaling and troubleshooting. (Time estimate: Ongoing)

Glossary

  • Layer 1 / substrate: k3s, containerd, and NVIDIA GPU enablement.
  • Layer 2 / workload: Ollama, Open WebUI, and the telemetry applications.
  • Profile: A Helm values.yaml overlay tuned for specific hardware (e.g., jetson, rtx-3090) or a specific model scenario.
  • Read-only validation: Automated checks that observe state without mutating it (e.g., checking if a pod is ready).
  • Evidence: Redacted logs and metrics summaries that deliberately exclude sensitive runtime data like user prompts.

[!TIP] Use edge-cli doctor before an installation and edge-cli status afterward. These commands automate a long series of manual validation checks into a consistent, fast diagnostic path.