Skip to content

Observability and Application Layer Installation

This guide covers deploying the core application stack, which includes Ollama, Open WebUI, and a comprehensive observability suite (Prometheus, Grafana, OpenTelemetry).

1. Stack Components

The llm-observability-stack helm chart manages: - Ollama: The LLM inference engine. - Open WebUI: A ChatGPT-like web interface for interacting with models. - OpenTelemetry Collector: Ingests traces and metrics. - Prometheus Ecosystem: Prometheus, Alertmanager, Node Exporter, Kube State Metrics. - Grafana: Visualization dashboards for GPU and LLM metrics.

2. Helm Dependency Build Process

Our stack uses a master Helm chart (llm-observability-stack) that depends on several sub-charts. Before deploying, you must build these dependencies.

cd /media/waqasm86/External1/Waqas-Projects/Project-Linux-Kubernetes-Nvidia/Project-Edge-Computing-LLM/edge-llm-tests/deploy/helm/llm-observability-stack
helm dependency build

Expected Output: 

Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 4 charts
Downloading kube-prometheus-stack from repo https://prometheus-community.github.io/helm-charts
...

3. Profile Selection and Values Composition

The platform supports different operational modes defined via Helm values files: - values.yaml: Base configuration. - profiles/gpu.yaml: Overrides for GPU deployments (e.g., sets nvidia.com/gpu: 1 requests, configures RuntimeClass). - profiles/cpu.yaml: Overrides for CPU-only deployments (removes GPU limits, adjusts threading).

When deploying, Helm overlays these files in order.

4. Deployment Commands

We deploy the stack into the llm-observability namespace.

cd /media/waqasm86/External1/Waqas-Projects/Project-Linux-Kubernetes-Nvidia/Project-Edge-Computing-LLM/edge-cli
go run main.go install --profile gpu

Using Helm Directly

kubectl create namespace llm-observability

helm upgrade --install llm-stack ./deploy/helm/llm-observability-stack \
  --namespace llm-observability \
  -f ./deploy/helm/llm-observability-stack/values.yaml \
  -f ./deploy/helm/llm-observability-stack/profiles/gpu.yaml

5. Rollout Verification

After deploying, verify the pods are running. Initializing Ollama and Open WebUI may take a few minutes as container images are pulled.

kubectl get pods -n llm-observability

Expected Output: 

NAME                                                        READY   STATUS    RESTARTS   AGE
llm-stack-grafana-7b89f89b5c-k2p8x                          2/2     Running   0          5m
llm-stack-kube-prometheus-operator-5d4b8f5d5-9m2p4          1/1     Running   0          5m
llm-stack-kube-state-metrics-845b4c9794-x6v2m               1/1     Running   0          5m
llm-stack-open-webui-6b586d6d87-j4h9s                       1/1     Running   0          5m
llm-stack-opentelemetry-collector-0                         1/1     Running   0          5m
llm-stack-prometheus-node-exporter-4x2s9                    1/1     Running   0          5m
ollama-0                                                    1/1     Running   0          5m
alertmanager-llm-stack-kube-prometheus-alertmanager-0       2/2     Running   0          4m
prometheus-llm-stack-kube-prometheus-prometheus-0           2/2     Running   0          4m

Verifying Services

Check that all expected services and ports are exposed:

kubectl get svc -n llm-observability

Key Services: - ollama: port 11434 - llm-stack-open-webui: port 80 (target 8080) - llm-stack-grafana: port 80 - llm-stack-kube-prometheus-prometheus: port 9090 - llm-stack-opentelemetry-collector: ports 4317/4318/8888

6. ServiceMonitor and Metrics Verification

The stack uses ServiceMonitors to tell Prometheus what to scrape.

kubectl get servicemonitors -n llm-observability

To ensure Prometheus is successfully scraping Ollama, you can port-forward Prometheus and check the targets:

kubectl port-forward svc/llm-stack-kube-prometheus-prometheus -n llm-observability 9090:9090

Visit http://localhost:9090/targets in your browser. You should see ollama listed as UP.

7. Configuration Flow

graph TD
    A[Values.yaml] --> B{Helm Template}
    C[profiles/gpu.yaml] --> B
    B --> D(llm-observability namespace)
    D --> E[Ollama Pods]
    D --> F[Open WebUI]
    D --> G[Prometheus/Grafana]
    E -->|Exposes Metrics| G