First Inference & Model Verification¶

This guide walks you through executing your first local GGUF inference request, verifying GPU offloading layer allocation, testing the Open WebUI interface, and capturing sanitized operational evidence.

1. Discover Services and Create Port Tunnels¶

First, list all services running in the llm-observability namespace to discover the exact service endpoints:

kubectl get svc -n llm-observability

Expected output:

NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
ollama                   ClusterIP   10.43.231.195   <none>        11434/TCP   8h
open-webui               ClusterIP   10.43.6.112     <none>        8080/TCP    8h
llm-observability-stack-grafana ClusterIP 10.43.169.54 <none>      80/TCP      8h

Establish local port-forwarding tunnels for Ollama (11434) and Open WebUI (8080):

# Terminal 1: Port-forward Ollama API
kubectl -n llm-observability port-forward service/ollama 11434:11434

# Terminal 2: Port-forward Open WebUI
kubectl -n llm-observability port-forward service/open-webui 8080:8080

2. Inspect Registered Models via Ollama API¶

In a separate terminal, verify that the Ollama service is reachable and inspect registered local model tags:

# Check service health
curl -s http://127.0.0.1:11434/api/tags | jq .

Expected JSON response listing available model profiles:

{
  "models": [
    {
      "name": "gemma3-1b-it-gguf-local:latest",
      "modified_at": "2026-07-20T10:00:00Z",
      "size": 815347200,
      "digest": "sha256:1a2b3c4d..."
    },
    {
      "name": "llama3-2-1b-local:latest",
      "size": 1320000000
    }
  ]
}

Check currently loaded resident models in memory:

curl -s http://127.0.0.1:11434/api/ps | jq .

3. Execute Non-Streaming and Streaming Inference¶

Non-Streaming Generation (`stream: false`)¶

Execute a simple completion request to verify prompt evaluation and response generation:

export OLLAMA_MODEL="gemma3-1b-it-gguf-local:latest"

curl -s http://127.0.0.1:11434/api/generate \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"$OLLAMA_MODEL\",
    \"prompt\": \"Explain why Kubernetes is useful for edge computing in two sentences.\",
    \"stream\": false
  }" | jq .

Expected response contains timing and token statistics:

{
  "model": "gemma3-1b-it-gguf-local:latest",
  "response": "Kubernetes provides container orchestration and automated scaling at the edge. It ensures high availability and standardized deployment across heterogeneous edge devices.",
  "done": true,
  "total_duration": 482000000,
  "load_duration": 120000000,
  "prompt_eval_count": 14,
  "eval_count": 28,
  "eval_duration": 350000000
}

Streaming Generation (`stream: true`)¶

Stream response tokens in real-time:

curl -s http://127.0.0.1:11434/api/generate \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"$OLLAMA_MODEL\",
    \"prompt\": \"List 3 advantages of GGUF quantization.\",
    \"stream\": true
  }"

4. Verify GPU Layer Offloading & VRAM Allocation¶

A running pod does not automatically guarantee full or partial GPU offloading. Verify that model layers are loaded into GPU VRAM rather than host CPU RAM.

Step 4A: Check Ollama Resident Process¶

Query Ollama's internal process list to inspect the exact processor split:

OLLAMA_POD=$(kubectl -n llm-observability get pods -l app.kubernetes.io/name=ollama -o jsonpath='{.items[0].metadata.name}')

kubectl -n llm-observability exec "$OLLAMA_POD" -- ollama ps

Expected output showing GPU offloading:

NAME                                ID           SIZE     PROCESSOR    UNTIL
gemma3-1b-it-gguf-local:latest      1a2b3c4d     815 MB   100% GPU     4 minutes from now

Step 4B: Query Host GPU VRAM (`nvidia-smi`)¶

Run nvidia-smi on the host to observe process memory allocation:

nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv

Expected output:

pid, process_name, used_gpu_memory [MiB]
14285, /usr/bin/ollama_llama_server, 784 MiB

5. Validate via `gguf-observability` Contract Verifier¶

Run the read-only GGUF runtime verifier binary to assert that operational memory ceilings are respected:

cd /media/waqasm86/External1/Waqas-Projects/Project-Linux-Kubernetes-Nvidia/Project-Edge-Computing-LLM/gguf-observability

GGUF_MODEL="gemma3-1b-it-gguf-local" \
GGUF_VRAM_CEILING_MIB=900 \
GGUF_EXPECTED_NUM_GPU=23 \
GGUF_EXPECTED_NUM_CTX=256 \
GGUF_EXPECTED_NUM_BATCH=1 \
go run ./cmd/gguf-observe -mode contract

Sanitized evidence output:

{
  "timestamp": "2026-07-20T23:55:00Z",
  "status": "PASS",
  "model_alias": "gemma3-1b-it-gguf-local",
  "vram_ceiling_mib": 900,
  "observed_vram_mib": 784,
  "num_gpu_layers": 23,
  "num_ctx": 256,
  "num_batch": 1,
  "gpu_accelerator": "NVIDIA GeForce 940M",
  "evidence_hash": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}

6. Access via Open WebUI Browser Interface¶

Open your browser and navigate to http://localhost:8080.
Create your initial local administrator user account (stored locally in Redis / SQLite).
Select gemma3-1b-it-gguf-local from the top model dropdown menu.
Test conversational inference and observe latency metrics rendered in the Grafana dashboard.