Troubleshooting Guide¶

This document provides a decision tree and specific commands to diagnose and fix common issues in the Edge LLM platform.

1. High-Level Decision Tree¶

Start here when diagnosing an issue:

Is Kubernetes reachable? kubectl get nodes
No: Check k3s service (systemctl status k3s).
Are GPUs detected by k3s? kubectl get nodes -o jsonpath='{.items[*].status.allocatable.nvidia\.com/gpu}'
No: See Section 2: GPU Detection Issues.
Are LLM Pods Running? kubectl get pods -n llm-observability
Pending: See Section 3: Pods Pending or Evicted.
CrashLoopBackOff: See Section 4: Application Crashes.
Is the UI working but models failing?
See Section 5: Inference and VRAM Issues.

2. GPU Detection Issues¶

Symptom: nvidia.com/gpu is missing from node allocatable resources.

Diagnostic Steps: 1. Check host drivers:

nvidia-smi

If this fails, the host driver is missing or broken. Reinstall via ubuntu-drivers autoinstall.

Check GPU Operator Pods:
```
kubectl get pods -n gpu-operator
```
If nvidia-cuda-validator is crashing, check logs: kubectl logs -n gpu-operator -l app=nvidia-cuda-validator. Usually indicates a mismatch between container toolkit and host driver.
If nvidia-device-plugin is crashing, it cannot communicate with the container runtime. Verify the containerd config steps in nvidia-layer.md.

Check NFD:

kubectl logs -n gpu-operator -l app=nfd-worker | grep pci

Ensure NFD sees the PCI device for the NVIDIA GPU.

3. Pods Pending or Evicted¶

Symptom: Ollama or WebUI pods are stuck in Pending or Evicted state.

Diagnostic Steps: 1. Describe the pod:

kubectl describe pod -l app.kubernetes.io/name=ollama -n llm-observability

2. Common Causes: - Insufficient GPU: 0/1 nodes are available: 1 Insufficient nvidia.com/gpu. This means either the GPU isn't detected (see Section 2), or another pod is monopolizing it. - Disk Pressure / Evicted: The host node has less than ~10-15% free disk space. Kubelet evicts pods to save itself. Clear space and delete the Evicted pods. - RuntimeClass Missing: If using GPU profile, nvidia RuntimeClass must exist. kubectl get runtimeclass.

4. Application Crashes¶

Symptom: Pods are in CrashLoopBackOff.

Diagnostic Steps: 1. Check Application Logs:

kubectl logs -n llm-observability <POD_NAME> --previous

2. Ollama Specifics: If Ollama crashes immediately, it might be trying to load a corrupt model file from the PVC or failing to bind to the NVIDIA runtime. 3. Open WebUI Specifics: If WebUI crashes, it is usually a database migration issue or a problem connecting to the Ollama service. Check if the ollama service is resolving:

kubectl exec -it <WEBUI_POD> -n llm-observability -- ping ollama

5. Inference and VRAM Issues¶

Symptom: "Out of memory", slow generation, or empty responses.

Diagnostic Steps: 1. Check VRAM Usage:

watch -n 1 nvidia-smi

2. Check Ollama Server Logs:

kubectl logs -n llm-observability -l app.kubernetes.io/name=ollama -f

Look for CUDA error: out of memory. 3. Resolution: - Unload models: curl -X POST http://localhost:11434/api/generate -d '{"model": "<MODEL>", "keep_alive": 0}' - Use a smaller quantized model. - Reduce num_ctx in the Modelfile.

6. Observability Stack Issues¶

Symptom: Grafana shows no data, or OTel collector has errors.

Diagnostic Steps: 1. Check DCGM Exporter:

kubectl logs -n gpu-operator -l app=nvidia-dcgm-exporter

2. Check Prometheus Targets: Port-forward Prometheus (9090) and check /targets. If they are down, check the ServiceMonitors. 3. OTel Collector Logs:

kubectl logs -n llm-observability llm-stack-opentelemetry-collector-0

Look for connectivity issues to Prometheus/Grafana.