Skip to content

Testing and Evidence Collection

The edge-llm-tests repository provides a comprehensive test suite to validate the functionality, performance, and security of the Edge Computing LLM platform. It also generates sanitized evidence for compliance and auditing.

Test Modes

The test suite can be executed in three primary modes:

1. Repository Mode

Tests individual components locally without requiring a full Kubernetes cluster. Useful for unit tests, configuration validation, and pre-commit checks.

cd edge-llm-tests
./run-tests.sh --mode repository

2. Cluster Mode

Runs integration and end-to-end tests against a live, deployed cluster. Validates Kubernetes resources, network policies, GPU allocation, and service connectivity.

./run-tests.sh --mode cluster

[!IMPORTANT] Cluster mode requires a valid KUBECONFIG pointing to a cluster with the platform fully deployed (edge-cli apply completed).

3. All Mode

Runs both repository and cluster tests sequentially.

./run-tests.sh --mode all

Environment Variables

The test suite relies on environment variables, particularly for testing specific model profiles (GGUF):

  • TEST_GGUF_PROFILE: Specifies the profile to test (e.g., lite, standard, heavy).
  • TEST_OLLAMA_ENDPOINT: Overrides the default Ollama service endpoint.
  • TEST_TIMEOUT: Sets the maximum duration for long-running tests (default: 300s).

Example: 

TEST_GGUF_PROFILE=standard ./run-tests.sh --mode cluster --suite models

Running Focused Test Suites

You can run specific test suites to narrow down failures or validate specific areas:

# Run only GPU validation tests
./run-tests.sh --mode cluster --suite gpu

# Run observability stack tests (Prometheus, Grafana)
./run-tests.sh --mode cluster --suite observability

# Run application tests (Open WebUI connectivity)
./run-tests.sh --mode cluster --suite webui

Sanitized Evidence Output

When tests complete, they generate evidence artifacts in the evidence/ directory. For privacy and security, this output is sanitized to remove sensitive information (e.g., real IP addresses, personal usernames, raw API keys).

Evidence Schema (JSON)

{
  "timestamp": "2023-10-27T10:00:00Z",
  "environment": "k3s-edge",
  "profile": "standard",
  "tests": [
    {
      "name": "gpu-operator-ready",
      "status": "PASS",
      "duration_ms": 150,
      "output_sanitized": "daemonset/nvidia-device-plugin-daemonset is ready"
    },
    {
      "name": "ollama-model-load",
      "status": "PASS",
      "duration_ms": 4500,
      "details": {
        "model": "llama3:8b",
        "vram_usage_mb": 4800
      }
    }
  ],
  "summary": {
    "total": 45,
    "passed": 45,
    "failed": 0,
    "skipped": 0
  }
}

Interpreting Results

  • PASS: The component meets all expected criteria.
  • FAIL: A critical assertion failed. Review the output_sanitized field for details.
  • SKIP: The test was skipped, often because a prerequisite was not met (e.g., trying to run a heavy profile test on a 8GB VRAM GPU).

Troubleshooting Test Failures

  1. Check Pod Logs: If a cluster test fails, use kubectl logs in the relevant namespace (gpu-operator or llm-observability).
  2. Verify Resources: Ensure the node has sufficient CPU/Memory/VRAM. Run edge-cli doctor.
  3. Network Policies: If connectivity tests fail, check if Calico or standard NetworkPolicies are blocking traffic.