Glossary¶

This glossary defines key terms and acronyms used throughout the Edge Computing LLM platform documentation.

A-C¶

Air-gapped Environment A secure network environment completely disconnected from the public internet. The platform supports installation in air-gapped scenarios using pre-downloaded images.

Containerd An industry-standard container runtime. K3s uses containerd by default, which we configure to support the NVIDIA runtime wrapper.

CUDA Compute Unified Device Architecture. A parallel computing platform and API model created by NVIDIA, essential for running LLMs on GPUs.

D-G¶

DCGM (Data Center GPU Manager) A suite of tools for managing and monitoring NVIDIA GPUs. The dcgm-exporter exposes GPU metrics (temperature, utilization, VRAM usage) to Prometheus.

Edge Node A physical computer (often a small form factor server, workstation, or ruggedized device) located physically close to the data source or end user, rather than in a centralized cloud data center.

GGUF (GPT-Generated Unified Format) A binary file format used to store quantized LLM models. It allows models to be run efficiently on edge devices using CPU or GPU acceleration. It is the primary format used by Ollama.

GPU Operator A Kubernetes component developed by NVIDIA that automates the management of all NVIDIA software components needed to provision GPUs.

H-M¶

Helm A package manager for Kubernetes. The platform uses Helm charts internally to deploy complex applications like the observability stack.

Inference The process of running data through a trained machine learning model to make a prediction or generate text.

K3s A lightweight, fully compliant Kubernetes distribution built for IoT and Edge computing.

LLM (Large Language Model) A type of artificial intelligence model designed to understand and generate human language. Examples include Llama 3, Mistral, and Qwen.

N-Q¶

NetworkPolicy A Kubernetes resource that controls the traffic flow between pods, providing network isolation and security.

Ollama An open-source project that simplifies running LLMs locally. It serves as the core inference engine in our platform.

Open WebUI A user-friendly web interface for interacting with LLMs, similar to ChatGPT, that connects to the local Ollama backend.

Quantization A technique to reduce the precision of the numbers used to represent a model's weights (e.g., from 16-bit floats to 4-bit integers). This drastically reduces VRAM requirements with minimal loss in quality, crucial for edge computing.

R-Z¶

RBAC (Role-Based Access Control) A method of regulating access to computer or network resources based on the roles of individual users within an enterprise. In Kubernetes, it controls what service accounts can do.

Taint and Toleration Kubernetes concepts. A node is "tainted" to repel pods that do not have a matching "toleration". We use this to ensure only GPU-workloads run on GPU nodes if clustering multiple devices.

VRAM (Video Random Access Memory) The memory located on the graphics card. LLMs must be loaded into VRAM for fast inference. VRAM capacity is the primary constraint in edge LLM deployments.