Skip to content

Operations Runbook

This runbook outlines the standard operating procedures for managing the Edge LLM platform, performing maintenance, and executing health checks.

1. Pre-Change Checklist

Before performing any upgrades, configuration changes, or scaling operations:

  1. Verify Current State: Ensure all namespaces (gpu-operator, llm-observability) are fully ready.
  2. Snapshot Storage: If possible, take a filesystem snapshot of /var/lib/rancher/k3s/storage (where PVCs are stored).
  3. Check Resource Utilization: Ensure there is sufficient disk space and memory available on the node before initiating pulls for large images/models.

2. Health Check Commands

Use these commands to perform a comprehensive health check of the platform.

Kubernetes Node Health

kubectl get nodes
Expected Output: Ready status.

GPU Operator Health

kubectl get pods -n gpu-operator | grep -v Running
Expected Output: Only Completed pods (like nvidia-cuda-validator) should appear. No CrashLoopBackOff or Error.

Verify device plugin allocation: 

kubectl get nodes -o jsonpath='{.items[*].status.allocatable.nvidia\.com/gpu}'
Expected Output: 1 (or more, depending on your GPUs).

LLM Stack Health

kubectl get pods -n llm-observability
Expected Output: All pods 1/1 or 2/2 Running.

API Health

curl -s http://localhost:11434/api/tags | jq .
Expected Output: A JSON payload listing loaded models.

3. Maintenance Workflow

Updating Helm Deployments

When modifying values.yaml or deploying an updated chart version:

helm upgrade llm-stack ./deploy/helm/llm-observability-stack -n llm-observability -f ./deploy/helm/llm-observability-stack/values.yaml -f ./deploy/helm/llm-observability-stack/profiles/gpu.yaml

Wait for rollout: 

kubectl rollout status statefulset/ollama -n llm-observability
kubectl rollout status deployment/llm-stack-open-webui -n llm-observability

Restarting Services

If a service becomes unresponsive but the pod is technically "Running":

Restart Ollama: 

kubectl rollout restart statefulset/ollama -n llm-observability

Restart WebUI: 

kubectl rollout restart deployment/llm-stack-open-webui -n llm-observability

4. Rollback Procedures

If a Helm upgrade introduces instability:

  1. Identify Revision History: 
    helm history llm-stack -n llm-observability
  2. Rollback to Previous Revision: 
    helm rollback llm-stack <REVISION_NUMBER> -n llm-observability
  3. Verify Rollback: Ensure pods recreate and reach Running state.

5. End-of-Window Verification

After completing maintenance: 1. Run the Health Check commands (Section 2). 2. Load a small model via the API to verify inference paths are open. 

curl -X POST http://localhost:11434/api/generate -d '{"model": "llama3.2:1b", "prompt": "Test", "stream": false}'
3. Open Grafana and confirm metrics are still flowing from Ollama and the GPU exporter.

6. Incident Response

Symptom: OOMKilled Pods - Action: Check dmesg -T | grep -i oom on host. Reduce model size or limit context length in Modelfile.

Symptom: Model Loading Extremely Slow - Action: Verify nvidia.com/gpu is assigned to Ollama pod. Check kubectl describe pod ollama-0 -n llm-observability. If not, it's loading into CPU RAM instead of VRAM.

Symptom: Grafana Missing GPU Metrics - Action: Restart DCGM exporter: kubectl delete pod -l app=nvidia-dcgm-exporter -n gpu-operator.