Model Lifecycle Management¶

Effectively managing LLM weights is crucial on edge hardware where storage and memory are highly constrained.

1. Model Inventory¶

To see which models are currently downloaded and stored on your node's Persistent Volume:

kubectl exec -it ollama-0 -n llm-observability -- ollama list

Expected Output:

NAME            ID              SIZE    MODIFIED
llama3.2:1b     abcd1234efgh    1.3 GB  2 days ago
qwen2.5:1.5b    ijkl5678mnop    1.1 GB  4 hours ago

2. Pulling Models¶

You can download models directly from the Ollama registry without running them immediately:

kubectl exec -it ollama-0 -n llm-observability -- ollama pull gemma2:2b

[!TIP] Pulling large models (e.g., Llama 3 8B at ~5GB) can take several minutes depending on your network connection. During this time, the disk I/O on the node will increase.

3. Creating Custom Models from Modelfile¶

To create a customized version of a model (e.g., setting a system prompt or tuning parameters):

1. Create a Modelfile locally:

   cat <<EOF > Modelfile
   FROM llama3.2:1b
   SYSTEM "You are a helpful edge computing assistant."
   PARAMETER num_ctx 4096
   EOF
   

2. Copy the file into the pod:

   kubectl cp Modelfile llm-observability/ollama-0:/tmp/Modelfile
   

3. Build the model:

   kubectl exec -it ollama-0 -n llm-observability -- ollama create my-edge-assistant -f /tmp/Modelfile
   

4. Load and Unload Procedures¶

Models are automatically loaded into VRAM when an API request is received. However, on small GPUs, you may need to explicitly manage memory.

Verify Loaded Models¶

See which models are currently consuming VRAM:

kubectl exec -it ollama-0 -n llm-observability -- ollama ps

NAME            ID              SIZE    PROCESSOR       UNTIL
llama3.2:1b     abcd1234efgh    1.8 GB  100% GPU        4 minutes from now

Force Unload a Model¶

If you need to free up VRAM immediately for a different model:

curl -X POST http://localhost:11434/api/generate -d '{
  "model": "llama3.2:1b",
  "keep_alive": 0
}'

5. Storage Management¶

Models are stored on a Persistent Volume Claim (PVC) mounted to /root/.ollama inside the ollama-0 pod.

If your disk fills up, you must delete old models:

kubectl exec -it ollama-0 -n llm-observability -- ollama rm gemma2:2b

To see actual disk usage on the host (assuming local-path-provisioner):

sudo du -sh /var/lib/rancher/k3s/storage/*ollama*

6. Model Switching on Small GPUs¶

When running Open WebUI on a small GPU (e.g., 8GB-12GB), rapidly switching between chats that use different models can cause Out-Of-Memory (OOM) errors if the previous model hasn't been unloaded yet.

Best Practice: Configure Open WebUI to aggressively unload models. In the Open WebUI admin settings, navigate to the Connections > Ollama section and set the Keep Alive setting to a very short duration (e.g., 1m or 0) if you frequently switch models.