# Operations

### Requirements

`llama-cpp-python` is installed as a **pre-built wheel** (not compiled from source). `requirements.txt` specifies the **Vulkan** variant via `--extra-index-url https://abetlen.github.io/llama-cpp-python/whl/vulkan`. Other backends are available by changing the index URL:

<table id="bkmrk-variant-table"><tbody><tr><th>Backend</th><th>Index URL suffix</th></tr><tr><td>cpu</td><td>`.../whl/cpu`</td></tr><tr><td>vulkan **(default)**</td><td>`.../whl/vulkan`</td></tr><tr><td>cuda</td><td>`.../whl/cuda`</td></tr><tr><td>rocm</td><td>`.../whl/rocm`</td></tr><tr><td>metal</td><td>`.../whl/metal`</td></tr><tr><td>sycl</td><td>`.../whl/sycl`</td></tr></tbody></table>

Docker image installs `libvulkan1`. GPU access (`/dev/dri`) is commented out in `compose.yaml` by default — uncomment for hardware acceleration. Falls back to CPU without GPU.

The x86-64-v2 baseline or equivalent is required by NumPy's pre-built wheels (see [NumPy SIMD build options](https://numpy.org/doc/stable/reference/simd/build-options.html)), usually configurable in virtual machines. CPUs without these instructions can still run the orchestrator by **recompiling NumPy from source** with reduced SIMD flags (`NPY_DISABLE_CPU_FEATURES`), or by using a distro that ships a compatible build.

---

### Installation

#### Using Docker

The Python orchestrator runs in Docker. Quick start:

```bash
cd plugins/agent
docker compose up -d --build
```

Verify: `curl http://localhost:8120/docs` should show the Swagger UI.

**Port** `127.0.0.1:8120:8120` — bound to localhost only, **MUST NOT** be publicly exposed.

[Here](https://usermanual.vtenext.com/books/developers/page/install-docker-for-kitt) a quick snippet to install docker.

#### Environment Variables

<table id="bkmrk-env-table"><tbody><tr><th>Variable</th><th>Default</th></tr><tr><td>`HOST` (inside the container)</td><td>`0.0.0.0`</td></tr><tr><td>`PORT`</td><td>`8120`</td></tr><tr><td>`EMBED_MODEL`</td><td>`nomic-ai/nomic-embed-text-v2-moe-GGUF:Q8_0`</td></tr><tr><td>`RERANK_MODEL`</td><td>`ms-marco-MiniLM-L-12-v2`</td></tr><tr><td>`HF_CACHE_DIR`</td><td>`/app/hf_cache`</td></tr><tr><td>`DOCUMENTS_DIR`</td><td>`/app/docs`</td></tr><tr><td>`VECTORS_DIR`</td><td>`/app/vectors`</td></tr></tbody></table>

In case of multiple orchestrators on the same machine it's highly recommended to point `HF_CACHE_DIR` to the same directory.

#### Without Docker

This setup is technically possible but it is **unsupported** by our support subscription service. The script assumes Ubuntu 24.04 LTS but **it's experimental and not guaranteed to work**.

<details id="bkmrk-script-%23%21%2Fbin%2Fbash-s"><summary>script</summary>

```bash
#!/bin/bash
set -euo pipefail

# install host dependencies
pkgs=(
  python3
  python3-pip
  python3.12-venv
  python-is-python3
  libgomp1
  libvulkan1
)
apt install "${pkgs[@]}"

# make virtual env
VTE="/var/www/html/vte"
AGENT="$VTE/plugins/agent"
cd "$AGENT"
python -m venv venv
source venv/bin/activate

# install python libraries
pip install --upgrade pip setuptools wheel
pip install -r requirements.txt

# install systemd service
SERVICE="vte-agent.service"
cat > "/etc/systemd/system/$SERVICE" <<UNIT
[Unit]
Description=vtenext AI agent orchestrator
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
Restart=unless-stopped
RestartSec=3

User=www-data
Group=www-data

Environment=PYTHONUNBUFFERED=1
Environment=PYTHONDONTWRITEBYTECODE=1
Environment=PYTHONPATH=${AGENT}/src
Environment=HF_CACHE_DIR=${AGENT}/hf_cache
Environment=HOST=127.0.0.1
Environment=PORT=8120

WorkingDirectory=${AGENT}
ExecStart=${AGENT}/venv/bin/python -m vte_agent

[Install]
WantedBy=default.target
UNIT

systemctl daemon-reload
systemctl enable --now "$SERVICE"

# verify
systemctl --no-pager --full status "$SERVICE"
curl -fsS "http://127.0.0.1:8120/docs" >/dev/null && echo "OK"
```

</details>---

### Troubleshooting

- **Slow startup:** Embedding model downloads from HuggingFace on first container start — cached in `HF_CACHE_DIR` afterwards.
- **Empty docs:** `/rag/build` raises `RuntimeError` if `docs/<agent_id>/` is empty.
- **GPU not used:** Uncomment `/dev/dri` in `compose.yaml`. Falls back to CPU otherwise.
- **CPU compat:** Verify with `/lib64/ld-linux-x86-64.so.2 --help`