CPU Fallback Mode¶

This document explains how to configure and run the LLM platform in CPU-only mode. This is critical for environments where NVIDIA GPUs are unavailable, drivers fail, or for testing platform logic without requiring specialized hardware.

1. Overview of CPU Mode¶

In CPU mode, Ollama relies entirely on system RAM and the host CPU for model inference.

Key Differences vs GPU Mode: - Inference is significantly slower (often 10x - 50x slower depending on CPU). - VRAM limits do not apply; instead, you are limited by system RAM. - The nvidia.com/gpu resource requests are removed from Kubernetes manifests. - The RuntimeClass is standard (not nvidia).

2. Deployment Commands¶

To deploy the stack in CPU mode, we use the profiles/cpu.yaml file to override the base configurations.

Using edge-cli¶

cd /media/waqasm86/External1/Waqas-Projects/Project-Linux-Kubernetes-Nvidia/Project-Edge-Computing-LLM/edge-cli
go run main.go install --profile cpu

Using Helm Directly¶

If you have already deployed in GPU mode, you can upgrade the release:

helm upgrade --install llm-stack ./deploy/helm/llm-observability-stack \
  --namespace llm-observability \
  -f ./deploy/helm/llm-observability-stack/values.yaml \
  -f ./deploy/helm/llm-observability-stack/profiles/cpu.yaml

3. Validating CPU Fallback¶

To verify that the system is running in CPU mode, check the Ollama pod's logs and resource allocations.

Check Pod Configuration¶

kubectl get pod ollama-0 -n llm-observability -o yaml | grep -A 5 "resources:"

Expected Output:

    resources:
      limits:
        cpu: "4"
        memory: 12Gi
      requests:
        cpu: "2"
        memory: 4Gi

(Notice the absence of nvidia.com/gpu under limits/requests)

Verify Inference Engine¶

Run a test prompt and observe the CPU usage on the host.

kubectl exec -it ollama-0 -n llm-observability -- ollama run qwen2.5:0.5b "Explain kubernetes."

While generating, open another terminal and run top or htop. You should see ollama process consuming a significant amount of CPU (e.g., 400% on a 4-core machine).

4. Switching Between Modes¶

If you have resolved hardware issues and wish to switch back to GPU mode:

Verify GPU Operator: Ensure the GPU operator pods are healthy and the node shows nvidia.com/gpu capacity.

Redeploy with GPU Profile:

helm upgrade llm-stack ./deploy/helm/llm-observability-stack \
  --namespace llm-observability \
  -f ./deploy/helm/llm-observability-stack/values.yaml \
  -f ./deploy/helm/llm-observability-stack/profiles/gpu.yaml

Restart Ollama Pod: Sometimes necessary to ensure the new runtime class attaches the device properly.
```
kubectl delete pod ollama-0 -n llm-observability
```

5. Performance Considerations for CPU Mode¶

When running in CPU mode, stick to extremely small models to maintain acceptable response times: - qwen2.5:0.5b - llama3.2:1b - Models quantized heavily (e.g., Q2_K or Q3_K).

[!WARNING] Attempting to run 8B+ models on CPU will result in extremely slow generation speeds (e.g., < 1 token per second) and may consume all available system RAM, causing host instability.