Accessing Services¶

This guide provides explicit instructions and commands for accessing the various user interfaces and APIs exposed by the Edge LLM platform.

1. Port Forwarding Overview¶

Because this platform is designed for an edge deployment (often without a dedicated load balancer or ingress controller), we rely on kubectl port-forward to securely access services running inside the k3s cluster.

2. Open WebUI (User Interface)¶

Open WebUI provides a ChatGPT-like interface for interacting with your models.

Command:

kubectl port-forward svc/llm-stack-open-webui -n llm-observability 8080:80

[!NOTE] The service exposes port 80 internally, but targets port 8080 on the container. We map it to 8080 on the host.

Access: Open your browser and navigate to: http://localhost:8080

Initial Setup: The first user to register becomes the administrator.

3. Ollama (Inference API)¶

Ollama provides a REST API compatible with OpenAI's format, making it easy to integrate with other tools.

Command: The Ollama service is configured as a NodePort or ClusterIP. To ensure reliable local access:

kubectl port-forward svc/ollama -n llm-observability 11434:11434

Health Check (Expected JSON Response):

curl -s http://localhost:11434/api/tags | jq .

{
  "models": [
    {
      "name": "llama3.2:1b",
      "model": "llama3.2:1b",
      "modified_at": "2024-05-20T10:00:00.0000000Z",
      "size": 1300000000,
      "digest": "sha256:abcd...",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "llama",
        "families": ["llama"],
        "parameter_size": "1B",
        "quantization_level": "Q8_0"
      }
    }
  ]
}

4. Grafana (Observability Dashboard)¶

Grafana visualizes metrics collected from the host, GPU, and LLM inference engine.

Command:

kubectl port-forward svc/llm-stack-grafana -n llm-observability 3000:80

Access: Open your browser and navigate to: http://localhost:3000

Credentials: - Username: admin - Password: Retrieved from the kubernetes secret created during helm installation:

kubectl get secret -n llm-observability llm-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

5. Prometheus (Metrics Backend)¶

For querying raw metrics or debugging alert rules.

Command:

kubectl port-forward svc/llm-stack-kube-prometheus-prometheus -n llm-observability 9090:9090

Access: Open your browser and navigate to: http://localhost:9090

6. Accessing from Other Devices on the LAN¶

If you need to access these services from another laptop or phone on the same local network, you can bind the port-forward to all interfaces (assuming your firewall allows it):

# Example for Open WebUI
kubectl port-forward --address 0.0.0.0 svc/llm-stack-open-webui -n llm-observability 8080:80