GGUF Model Profiles and Management¶

The Edge LLM platform leverages the GGUF (GPT-Generated Unified Format) for efficient inference. This document details how to configure models, tune parameters for edge hardware, and manage VRAM constraints.

1. Modelfile Syntax¶

Ollama uses a Modelfile to define model configurations, similar to a Dockerfile.

Example Modelfile for a custom system prompt:

FROM gemma2:2b
# Set temperature
PARAMETER temperature 0.3
# Set system message
SYSTEM """
You are a concise, highly technical kubernetes assistant.
Always format code in markdown blocks.
"""

To build a model from a Modelfile:

ollama create k8s-assistant -f ./Modelfile

2. Parameter Tuning for Edge Hardware¶

When deploying on resource-constrained hardware (e.g., RTX 3060 with 12GB VRAM), tuning is essential.

Key Parameters:¶

num_ctx: Context window size. Larger contexts use exponentially more VRAM. For 8B models on 12GB GPUs, limit to 4096 or 8192.
num_batch: Number of tokens to process in parallel. Lower this (e.g., 256) if you encounter Out-Of-Memory (OOM) errors during generation.
num_gpu: Number of layers to offload to GPU. Ollama usually auto-detects this, but you can force it (e.g., 999 for all layers, 0 for CPU only).

Example tuning in Modelfile:

FROM llama3.2:1b
PARAMETER num_ctx 4096
PARAMETER num_batch 256

3. VRAM Ceiling Enforcement¶

If you load a model that exceeds available VRAM, Ollama will partially offload to CPU/RAM, drastically reducing performance.

To prevent OOM crashes and ensure performance: 1. Monitor VRAM: Keep Grafana open. 2. Calculate Requirements: A Q4_K_M quantized 8B model uses ~5GB for weights + ~2GB for a 4k context window. 3. Keep Alive Settings: By default, Ollama keeps models in memory for 5 minutes. If switching between models frequently, set a shorter keep_alive to free VRAM quickly.

Via API:

{
  "model": "gemma2:2b",
  "keep_alive": "1m"
}

4. Model Catalog for Edge Deployment¶

We recommend the following models for edge GPUs (8GB - 12GB VRAM):

Model Family	Size	Recommended Quantization	VRAM Required	Best Use Case
Llama 3.2	1B	Q8_0 or fp16	~2-3 GB	Fast local tasks, JSON formatting
Llama 3.1	8B	Q4_K_M	~6-7 GB	General purpose chat, coding
Qwen 2.5	1.5B	Q8_0	~3 GB	Multilingual, code generation
Gemma 2	2B	Q6_K	~3-4 GB	Logic, reasoning tasks

5. Sequential Validation Procedure on Small GPUs¶

If you have a single GPU and want to test multiple models, use this procedure to avoid VRAM exhaustion:

Unload all models:

curl -X POST http://localhost:11434/api/generate -d '{"model": "llama3.2:1b", "keep_alive": 0}'

Load Model A: Run a test prompt.
Verify VRAM: Check nvidia-smi or Grafana.
Unload Model A: Send keep_alive: 0 request.
Load Model B: Run next test.

Shell Script for Sequential Testing¶

#!/bin/bash
MODELS=("llama3.2:1b" "qwen2.5:1.5b")
for model in "${MODELS[@]}"; do
  echo "Testing $model..."
  curl http://localhost:11434/api/generate -d "{
    \"model\": \"$model\",
    \"prompt\": \"Write a bash script to ping google.\",
    \"stream\": false
  }"
  echo -e "\nUnloading $model..."
  curl http://localhost:11434/api/generate -d "{\"model\": \"$model\", \"keep_alive\": 0}"
done