Skip to content

Storage and Upgrades

Managing persistent storage and handling upgrades are critical for maintaining a stable edge LLM platform.

1. Storage Management Overview

The platform uses k3s's built-in local-path-provisioner to dynamically provision Persistent Volume Claims (PVCs). This maps kubernetes storage directly to the host's filesystem.

Key PVCs in the platform: - Ollama Models (ollama-data): Stores all downloaded GGUF weights. Can grow very large (20GB+). - Open WebUI Database: Stores user accounts, chat histories, and settings. - Prometheus Data: Stores time-series metrics.

2. Checking Storage Usage

To check PVC capacity and usage from Kubernetes:

kubectl get pvc -n llm-observability
Expected Output: 
NAME                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
llm-stack-open-webui                Bound    pvc-1234abcd-5678-efgh-ijkl-901234567890   2Gi        RWO            local-path     10d
ollama-llm-stack-ollama-0           Bound    pvc-abcd1234-efgh-5678-ijkl-098765432109   30Gi       RWO            local-path     10d
prometheus-llm-stack-kube-prometheus-db-prometheus-llm-stack-kube-prometheus-prometheus-0 Bound pvc-xyz... 10Gi RWO local-path 10d

To find exactly where this data is stored on the host: 

sudo ls -l /var/lib/rancher/k3s/storage/

3. Expanding Storage (PVC Resizing)

If you need more space for models, you can expand the Ollama PVC (assuming the host has physical space).

  1. Edit the PVC: 
    kubectl edit pvc ollama-llm-stack-ollama-0 -n llm-observability
  2. Change the capacity: Find the spec.resources.requests.storage field and increase it (e.g., from 30Gi to 50Gi).
  3. Verify Expansion: 
    kubectl describe pvc ollama-llm-stack-ollama-0 -n llm-observability
    Note: The expansion usually requires the pod to be restarted to take effect at the filesystem level.

4. Upgrading the LLM Stack Helm Chart

When a new version of the llm-observability-stack or its dependencies is available:

  1. Update Helm Dependencies: 
    cd ./deploy/helm/llm-observability-stack
helm dependency update
  2. Apply the Upgrade: 
    helm upgrade llm-stack . \
  --namespace llm-observability \
  -f values.yaml \
  -f profiles/gpu.yaml
  3. Monitor Rollout: 
    kubectl get pods -n llm-observability -w

[!WARNING] Helm upgrades can sometimes recreate pods. While PVCs retain data, active chats or running generations will be interrupted.

5. Upgrading k3s

Upgrading the Kubernetes distribution itself should be done carefully to avoid breaking the GPU Operator.

  1. Check current version: 
    kubectl get nodes
  2. Run upgrade script: 
    curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.29.4+k3s1 sh -
  3. Verify: Ensure the GPU Operator components (especially nvidia-device-plugin-daemonset) return to Running state after the node restarts its kubelet.

6. Backup Procedures

To backup Open WebUI chats and user data, you can simply tar the directory on the host:

# Find the exact directory name first
sudo ls /var/lib/rancher/k3s/storage/ | grep open-webui
# Backup
sudo tar -czvf open-webui-backup.tar.gz /var/lib/rancher/k3s/storage/pvc-<ID>-open-webui/