Preparing the Ubuntu Host for the LLM Platform¶

This comprehensive guide details the necessary steps to prepare your Ubuntu host for deploying the Edge Computing LLM platform with NVIDIA GPU support.

1. System Requirements and Checks¶

Before proceeding, ensure your system meets the basic requirements.

Kernel and OS Verification¶

Verify that you are running a supported Ubuntu release (22.04 LTS or newer is recommended) and check your kernel version:

lsb_release -a
uname -r

Expected Output:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

5.15.0-101-generic

Disk Space Planning¶

You will need adequate disk space for container images, Helm charts, and large GGUF model weights. - System/Containers: ~10-15GB - Models: 5-20GB depending on your model selection (e.g., Llama 3 8B Q4 requires ~5GB).

Verify available space on your primary partition:

df -h /

Expected Output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2  234G   45G  177G  21% /

[!WARNING] Running out of disk space during model pulls or container execution will cause pods to fail or become Evicted. Ensure at least 30GB of free space is available before starting.

Swap Management¶

Kubernetes traditionally requires swap to be disabled, though k3s supports it. For optimal performance with memory-mapped LLMs, we recommend disabling swap to prevent paging VRAM/RAM to slow disks.

sudo swapoff -a

To make this permanent, edit /etc/fstab and comment out the swap line:

sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

2. NVIDIA Driver Verification¶

The NVIDIA driver must be installed on the host. We do not install the CUDA toolkit or container toolkit on the host directly; the GPU Operator will handle that inside the cluster.

Check if the driver is loaded:

nvidia-smi

Expected Output:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01     CUDA Version: 12.2     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060        Off |   00000000:01:00.0  On |                  N/A |
|  0%   45C    P8             14W /  170W |     558MiB /  12288MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

If nvidia-smi is not found, install the driver:

sudo ubuntu-drivers autoinstall
sudo reboot

3. k3s Installation and Verification¶

k3s is our chosen lightweight Kubernetes distribution.

Installation¶

Install k3s without the built-in Traefik ingress (we use node ports or explicit port forwards for simplicity) and without local-storage (if using a different provisioner, though local-path is fine for edge).

curl -sfL https://get.k3s.io | sh -s - --disable traefik --write-kubeconfig-mode 644

Verification¶

Check the node status:

kubectl get nodes

Expected Output:

NAME          STATUS   ROLES                  AGE   VERSION
ubuntu-host   Ready    control-plane,master   1m    v1.29.3+k3s1

Check the core k3s components:

kubectl get pods -n kube-system

Expected Output:

NAME                                      READY   STATUS    RESTARTS   AGE
coredns-6799fbcd5-xg9pk                   1/1     Running   0          2m
local-path-provisioner-84db5d44d9-b42lm   1/1     Running   0          2m
metrics-server-67c658944b-4qtz8           1/1     Running   0          2m

4. Installing Tooling (Helm & Go)¶

Helm Installation¶

Helm is required to deploy the GPU Operator and our custom LLM Observability Stack.

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Verify installation:

helm version

Go Installation¶

Go is required to build and run the edge-cli tool.

wget https://go.dev/dl/go1.22.2.linux-amd64.tar.gz
sudo rm -rf /usr/local/go
sudo tar -C /usr/local -xzf go1.22.2.linux-amd64.tar.gz
echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc
source ~/.bashrc

Verify installation:

go version

Expected Output:

go version go1.22.2 linux/amd64

5. Network Configuration Verification¶

Ensure your host networking allows traffic on standard k3s ports (e.g., 6443 for API server) if accessing remotely. For local edge access, ensure localhost resolution is working correctly.

ping -c 2 localhost

[!TIP] If you plan to access services from other machines on the LAN, ensure the host's UFW (Uncomplicated Firewall) is either disabled or configured to allow incoming connections to NodePorts.

6. Systemd Service Checks¶

Verify that the k3s service is active and enabled to start on boot:

systemctl status k3s

Expected Output:

● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-05-20 10:00:00 UTC; 5min ago
       Docs: https://k3s.io

You are now ready to install the NVIDIA Layer.