Skip to content

Accelerator modes

The CLI and Helm charts distinguish between GPU-backed execution, automatic detection, and CPU fallback. Understanding these modes is critical for ensuring reliable inference performance, optimal resource utilization, and successful workload scheduling.

You can explicitly choose the execution mode using the CLI flag --accelerator <mode> when installing or upgrading. Choose explicitly when repeatability matters.

Supported Modes Overview

The Edge LLM Stack supports three primary execution modes:

  1. Auto Mode (--accelerator auto): The default behavior. Dynamically detects capabilities.
  2. NVIDIA Mode (--accelerator nvidia): Forces GPU scheduling and requires the NVIDIA stack.
  3. CPU Mode (--accelerator cpu): Bypasses GPU scheduling and relies entirely on system RAM and CPU.

Decision Flowchart

The following diagram illustrates how the auto detection logic selects the final runtime mode.

flowchart TD
    Start[User runs edge-cli install]
    Flag{Is --accelerator set?}

    Flag -- "Yes (nvidia)" --> CheckNVIDIA
    Flag -- "Yes (cpu)" --> SetCPU
    Flag -- "Yes (auto) / Default" --> ProbeHost

    ProbeHost[Probe Host Capabilities] --> CheckSMI{nvidia-smi works?}
    CheckSMI -- No --> SetCPU
    CheckSMI -- Yes --> CheckNode{Node advertises nvidia.com/gpu?}
    CheckNode -- No --> SetCPU
    CheckNode -- Yes --> CheckRuntime{nvidia RuntimeClass exists?}
    CheckRuntime -- No --> SetCPU
    CheckRuntime -- Yes --> SetNVIDIA

    CheckNVIDIA[Verify NVIDIA Contracts] --> ValidNVIDIA{Contracts valid?}
    ValidNVIDIA -- Yes --> SetNVIDIA
    ValidNVIDIA -- No --> Fail[Fail Installation]

    SetNVIDIA[Deploy with NVIDIA RuntimeClass] --> DeployGPU[Ollama Pod scheduled with GPU limits]
    SetCPU[Deploy with default RuntimeClass] --> DeployCPU[Ollama Pod scheduled on CPU only]

[!NOTE] The auto mode is convenient for a known single-node host. It probes host and cluster capabilities, then selects a supported path. It does not guarantee that an interrupted first install left every runtime component healthy. Run edge-cli doctor and inspect its decision before applying.

NVIDIA Mode (--accelerator nvidia)

NVIDIA mode is the recommended path for production-like performance. It requires all of the following contracts to be satisfied:

  1. Host Driver: The host driver responds successfully to nvidia-smi.
  2. Container Runtime: k3s and containerd have an NVIDIA-capable runtime configured (e.g., nvidia-container-runtime).
  3. RuntimeClass: A relevant RuntimeClass resource named nvidia exists in the cluster.
  4. Node Capacity: The node actively advertises nvidia.com/gpu capacity.
  5. Operator Readiness: GPU Operator components in the gpu-operator namespace are in a Ready state.
  6. Workload Profile: The llm-observability-stack Helm values request the intended runtime and resource behavior.

Verifying NVIDIA Contracts

# 1. Check host driver
nvidia-smi

# Expected output should show driver version and CUDA version:
# +-----------------------------------------------------------------------------------------+
# | NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
# |-----------------------------------------+------------------------+----------------------+

# 2. Check Node Capacity
kubectl get nodes -o jsonpath='{.items[*].status.allocatable}' | grep -o 'nvidia.com/gpu'

# Expected output:
# nvidia.com/gpu

# 3. Check RuntimeClass
kubectl get runtimeclass nvidia

# Expected output:
# NAME     HANDLER   AGE
# nvidia   nvidia    5d

[!IMPORTANT] If any of these contracts fail when --accelerator nvidia is explicitly passed, the installation will abort.

CPU Mode (--accelerator cpu)

CPU fallback avoids NVIDIA scheduling and sets Ollama to CPU execution. It is useful for: - Functional validation of the application stack. - Environments with unsupported or broken hardware. - CI/CD pipelines without physical GPUs.

[!WARNING] Latency and memory behavior differ drastically in CPU mode. A passing CPU smoke test validates API integration and chart rendering, but it does not validate GPU integration, VRAM limits, or realistic inference latency.

Mixed CPU/GPU Offload

Ollama supports mixed execution. When running in NVIDIA mode, models that are too large to fit entirely in GPU VRAM will overflow to system RAM.

This is controlled by the num_gpu parameter, which determines the number of model layers offloaded to the GPU.

  • num_gpu = 0: Pure CPU execution (effectively CPU mode).
  • num_gpu = N: N layers are processed on the GPU, the rest on the CPU.
  • num_gpu = -1 (or max): All layers are offloaded to the GPU (fails if VRAM is insufficient).

VRAM Guardrails

Ollama's num_gpu controls a number of offloaded layers, not explicit megabytes. To stay near a specific target (especially on smaller Edge devices):

  1. Unload other models: Ensure the GPU memory is clear.
  2. Record idle GPU usage: Run nvidia-smi to see baseline usage (e.g., display servers).
  3. Select a quantized model: Choose a small quantized GGUF model and use low context/batch settings.
  4. Start conservative: Start with a conservative layer count in your Modelfile or API request.
  5. Measure: Load the model once and measure using both nvidia-smi and ollama ps.
  6. Adjust: Reduce layers if total observed usage exceeds the operational guardrail.

[!TIP] MIG (Multi-Instance GPU) is the mechanism for a true hardware partition on supported enterprise GPUs. Consumer GPUs (like RTX series) without MIG cannot turn num_gpu into a strict hard VRAM cap, so leave headroom for driver spikes.