Skip to content

Edge Computing LLM Documentation

Local-first · GPU-aware · Observable · Verifiable

Deploy Ollama GGUF models on Ubuntu and k3s, expose NVIDIA GPUs safely, observe the system end to end, and use Codex-guided verification to produce privacy-aware operational evidence.

One Platform, Clear Ownership

Layer Repository Responsibilities
Substrate k3s-nvidia-edge k3s integration, NVIDIA container runtime, GPU Operator configuration, device discovery, and infrastructure readiness checks
Workloads llm-observability-stack Ollama, Open WebUI, Prometheus, Grafana, OpenTelemetry, Redis, and resource-aware model profiles
Control Plane edge-cli Dependency-aware diagnostics, dry runs, installation, validation, status, logs, and explicit uninstall workflows
Verification edge-llm-tests / gguf-observability Read-only platform validation, runtime checks, and sanitized schema-driven evidence
Codex Operations edge-computing-openai Codex operational skill and EdgeProof verification layer for automated platform inspection and guided remediation
Documentation Edge-Computing-LLM.github.io Central documentation website published via GitHub Pages

OpenAI Build Week Project

The edge-computing-openai repository presents EdgeProof, the OpenAI-facing verification layer for the Edge-Computing-LLM platform.

EdgeProof uses Codex as the reasoning and operational guidance layer above deterministic Go checks. The workflow:

  1. Inspects Linux, k3s, NVIDIA GPU, inference, and observability state;
  2. Evaluates selected operational contracts through read-only checks;
  3. Explains which checks passed, failed, were skipped, or were not applicable;
  4. Produces privacy-aware human-readable and JSON evidence; and
  5. Proposes remediation without changing the host or cluster until the operator explicitly approves the action.

Explore EdgeProof · View Competition Repository

Documentation Guide

Get Started

Complete host prerequisites, complete a fresh-clone install, and execute your first local LLM inference call.

Concepts

Understand platform architecture, repository maps, ownership boundaries, and GPU vs. CPU accelerator modes.

Install

Step-by-step installation guides for Ubuntu, NVIDIA substrate, observability stack, and CPU fallback paths.

Operate

Operational runbooks, service access parameters, model lifecycle management, and troubleshooting workflows.

Observe

Telemetry flow details, OpenTelemetry Collector integration, Prometheus metrics, and Grafana dashboard guides.

Develop

Development setup, CLI Go code organization, test harness configuration, and adding new GGUF models.

Secure

Threat modeling, security hardening checklists, and privacy/evidence sanitization standards.

Validated Reference Environment

The documentation was reconciled against clean local clones of the platform repositories and a live single-node Ubuntu, k3s, and NVIDIA GPU deployment.

  • OS: Ubuntu 24.04.3 LTS (Linux kernel 6.17.0-40-generic)
  • k3s: v1.36.2+k3s1 (Control-plane / Worker)
  • GPU: NVIDIA GeForce 940M (1024 MiB VRAM, Driver 580.95.05, CUDA 13.0)
  • Inference Stack: Ollama 0.17.7 serving gemma3-1b-it-gguf-local with partial GPU offload
  • Observability: Prometheus, Grafana, OpenTelemetry Collector & Operator, DCGM exporter, Node Exporter, kube-state-metrics

See Validated Environment for full metrics and testing boundaries.