Guide: Adding a New Model¶

This guide outlines the end-to-end process for adding a new LLM (Large Language Model) to the Edge Computing platform.

Overview¶

Adding a model involves several steps across multiple repositories: 1. Creating a Modelfile. 2. Updating Helm values overlays. 3. Updating the observability catalog. 4. Adding tests to the matrix. 5. Documenting the model.

Step 1: Create a Modelfile¶

The edge-computing-openai repository (or the designated models repo) contains Modelfiles that define how Ollama should run the model.

1. Navigate to the models directory:

   cd edge-computing-openai/models
   

2. Create a new directory for the model (e.g., mistral-nemo).
3. Create a Modelfile:

   FROM mistral-nemo:12b-instruct-2407-q4_K_M
   
   # Set context window
   PARAMETER num_ctx 8192
   
   # Set system prompt
   SYSTEM """
   You are a helpful, secure, and precise AI assistant running on an edge device.
   Always provide factual information and decline requests that violate safety guidelines.
   """
   

Step 2: Add Values Overlay¶

Update the edge-cli profiles to include the new model in the appropriate tier (lite, standard, or heavy).

1. Open edge-cli/internal/config/profiles.go or the relevant YAML configurations.
2. Add the model to the profile configuration. For example, in profiles/standard.yaml:

   ollama:
     models:
       - name: llama3:8b
       - name: mistral-nemo:12b # New model
         preload: true
   

Step 3: Update GGUF Observability Catalog¶

To ensure the new model is tracked properly in Grafana dashboards, update the gguf-observability catalog.

1. Open gguf-observability/catalog/models.yaml.
2. Add the model metadata:

   - id: mistral-nemo-12b
     name: Mistral Nemo 12B
     family: mistral
     quantization: q4_K_M
     vram_required_mb: 8500
     recommended_profile: standard
   

Step 4: Update the Test Matrix¶

Ensure the new model is validated during CI/CD.

1. Open edge-llm-tests/matrix/models.json.
2. Add the model entry:

   {
     "name": "mistral-nemo:12b",
     "profile": "standard",
     "tests": ["load", "inference", "vram_limit"]
   }
   

Step 5: Testing the Model Locally¶

Before committing, verify the model works on your local edge cluster:

1. Apply the updated profile:

   edge-cli apply --profile standard
   

2. Wait for the model to pull and load:

   kubectl logs -n llm-observability deployment/ollama -f
   

3. Run inference via the CLI or Open WebUI:

   curl -X POST http://localhost:11434/api/generate -d '{
     "model": "mistral-nemo:12b",
     "prompt": "Explain edge computing briefly.",
     "stream": false
   }'
   

Step 6: Documentation¶

Update the user-facing documentation to announce the new model.

1. Update docs/reference/profiles-services.md to list the model under the correct profile.
2. Add any specific quirks or requirements (e.g., "Requires minimum 10GB VRAM") to docs/faq.md.

Once all steps are complete, submit PRs to the respective repositories.