Frequently Asked Questions (FAQ)¶
Installation and Setup¶
1. What operating systems are supported?¶
Currently, the platform strictly targets Ubuntu 22.04 LTS and Ubuntu 24.04 LTS. Other Linux distributions might work but are not officially supported or tested in our CI pipeline.
2. Can I install this on a cluster without GPUs?¶
The platform is specifically designed for NVIDIA GPU acceleration. While K3s can run CPU-only, the core LLM inference engine (Ollama) and the GPU operator require an NVIDIA GPU. For testing purposes, you can run in a degraded CPU-only mode, but inference will be extremely slow.
3. How do I completely remove the platform?¶
Run the following command to remove all components, including downloaded models and persistent data:
GPU and Hardware Issues¶
4. edge-cli doctor reports "NVIDIA drivers not found". What do I do?¶
Ensure you have installed the proprietary NVIDIA drivers on the host Ubuntu system. You can install them using:
5. I have an AMD GPU or Intel ARC GPU. Will this work?¶
No. This project explicitly relies on CUDA and the NVIDIA GPU Operator. AMD (ROCm) and Intel (OpenVINO) are not supported in this architectural design.
6. The system crashes with an Out Of Memory (OOM) error during inference.¶
This indicates the model you are trying to load exceeds your physical VRAM.
- Try switching to a smaller quantization (e.g., q4_K_M instead of q8_0).
- Use the lite profile via edge-cli apply --profile lite.
- Reduce the context window (num_ctx) in the Modelfile.
Model Management¶
7. Where are the downloaded models stored?¶
Models are stored in a PersistentVolumeClaim (PVC) bound to the ollama pod. On the host, this typically maps to a directory managed by the K3s local-path provisioner (usually under /var/lib/rancher/k3s/storage).
8. How do I add a custom model?¶
Please refer to the Add Model Guide for detailed step-by-step instructions.
9. Can I run multiple models simultaneously?¶
Yes, provided you have sufficient VRAM. Ollama will load models into memory until VRAM is full, and will evict older models if necessary. The heavy profile is best suited for multi-model concurrency.
Observability and Troubleshooting¶
10. How do I access the Grafana dashboards?¶
By default, the standard and heavy profiles deploy Grafana. You can access it by port-forwarding:
http://localhost:8080 in your browser.
11. My metrics are not showing up in Grafana.¶
Check if the DCGM exporter is running and Prometheus is scraping it. Run:
Also, ensure your GPU is actively processing requests, as idle states might show zero usage.12. How can I view the raw logs for the LLM engine?¶
Use the CLI to pull logs specifically from the Ollama pod:
Security¶
13. Is the Open WebUI exposed to the internet by default?¶
No. By default, Open WebUI is exposed as a NodePort on the local network only. To expose it to the internet, you must configure an Ingress controller with TLS, as described in the Hardening Guide.
14. Are my chat prompts sent to OpenAI or NVIDIA?¶
No. All inference happens completely locally on your edge node. No data is transmitted to third-party cloud services.
15. How do I restrict who can access Open WebUI?¶
Open WebUI has built-in authentication. Upon first launch, the first user to register becomes the administrator and can manage subsequent user access.
Development¶
16. How do I run the automated tests locally?¶
Ensure you have a cluster running, then execute:
17. How do I build the CLI from source?¶
Refer to the Developer Setup Guide. In short, clone the repository and run make build.
18. Why does the project use K3s instead of standard Kubernetes (K8s)?¶
K3s is optimized for edge environments with a smaller memory footprint and simplified single-binary installation, making it ideal for resource-constrained nodes.
19. How do I change the default port for the WebUI?¶
You can override the port by creating a custom values file and passing it to edge-cli apply.
20. Where can I submit a bug report?¶
Please submit all issues with sanitized evidence logs to the edge-cli GitHub repository issue tracker.