Profiles and Service Catalog¶
This document details the configuration profiles and the comprehensive catalog of services deployed in the Edge Computing LLM platform.
Configuration Profiles¶
The platform supports three primary profiles to accommodate different edge hardware capabilities.
1. Lite Profile¶
Designed for edge nodes with limited VRAM (4GB - 8GB). - Target Hardware: Jetson Orin Nano, low-end consumer GPUs. - Models: Quantized small models (e.g., Qwen 1.5B, Phi-3 Mini). - Features: Minimal observability, no heavy metric scraping.
2. Standard Profile¶
Designed for typical edge servers and workstations (12GB - 24GB VRAM). - Target Hardware: RTX 3060/4070, A2 Tensor Core GPU. - Models: Mid-size models (e.g., Llama 3 8B, Mistral 7B). - Features: Full observability stack (Prometheus + Grafana), Open WebUI enabled.
3. Heavy Profile¶
Designed for powerful edge clusters (24GB+ VRAM). - Target Hardware: RTX 3090/4090, A100. - Models: Large models or multiple concurrent smaller models (e.g., Mixtral 8x7B, Llama 3 70B Quantized). - Features: High-frequency metrics scraping, advanced log aggregation, High Availability (HA) web interfaces.
Service Catalog¶
Namespace: gpu-operator¶
Manages the NVIDIA hardware integration.
| Service / Component | Type | Description |
|---|---|---|
nvidia-device-plugin |
DaemonSet | Exposes GPU resources (nvidia.com/gpu) to the kubelet. |
nvidia-dcgm-exporter |
DaemonSet | Collects and exposes GPU telemetry metrics. Port: 9400 |
gpu-feature-discovery |
DaemonSet | Labels nodes with GPU capabilities (memory, architecture). |
Namespace: llm-observability¶
Houses the core inference engine and monitoring tools.
| Service | Type | Internal Port | Description |
|---|---|---|---|
ollama |
ClusterIP | 11434 |
The core LLM inference engine. |
prometheus |
ClusterIP | 9090 |
Time-series database for metrics collection. |
grafana |
ClusterIP | 80 |
Visualization dashboards for GPU and LLM metrics. |
kube-state-metrics |
ClusterIP | 8080 |
Exposes cluster-level object metrics. |
Namespace: default (or app)¶
Houses the user-facing applications.
| Service | Type | Internal Port | External Port (NodePort/Ingress) | Description |
|---|---|---|---|---|
open-webui |
NodePort | 8080 |
30080 |
The primary web interface for interacting with models. |
Helm Values and Composition¶
The deployment relies on Helm values overlays to compose the final state based on the profile.
- Base Values: Located in
charts/base-values.yaml. Contains common configurations. - Profile Overlays: Located in
charts/profiles/<profile>.yaml. Overrides resources, model lists, and replicas.
Example of Profile Composition:
helm upgrade --install ollama ./ollama-chart -f base-values.yaml -f profiles/standard.yaml
This composition is handled automatically by the edge-cli apply command.