DEVLOG.md

# Tabby-Tavern Development Log

Engineering history for the containerized local AI lab. Public tree is sanitized — no live keys, no weights, no chat logs.

## Hardware baseline

- GPU: NVIDIA GeForce RTX 4070 (12 GB)
- Environment: WSL2 Ubuntu + Docker Compose with NVIDIA GPU passthrough
- Primary inference path (v3 UNIFIED): Qwen3.5-9B GGUF on llama.cpp CUDA (`qwen` service)
- Alternate inference (profile): TabbyAPI EXL3 + Ollama GGUF
- Character frontend: SillyTavern (PNG character cards)
- Tooling: MCPO + FastMCP; optional dockroot diagnostics
- Coding pack: former Basecamp volumes on the same network

## v3.0.0 UNIFIED — Taproot + Basecamp folded in (2026-08-25)

Hermes session `20260825_002829` merged the running Taproot Qwen stack and Basecamp coding pack into `/home/jpanasuk/tabby-tavern/docker-compose.yml`.

- Compose project: `tabby-tavern`
- Network: `tabby-tavern_ai-network`
- Default: 15 services. Qwen and Postgres healthy. SearXNG / MCPO / code-server / Qdrant / Meilisearch HTTP 200.
- Primary endpoint: `http://qwen:8080/v1` in-network, `http://localhost:1234/v1` on the host. Model alias `Qwen3.5-9B`.
- TabbyAPI + Ollama: `--profile alternate-inference` (VRAM contention).
- dockroot: `--profile diagnostics`, socket read-only, no host port.
- Basecamp named volumes preserved (`basecamp_*`).
- Public tree sanitizes host model paths to `./models/...` and drops `external: true` so a stranger can `up` without those volume names pre-created.
- Sell sheet rewritten from this live graph — not the old 6-service pitch and not the fake 21-service pitch.

## Week 1 — Core integration

- Consolidated TabbyAPI, SillyTavern, Open WebUI, Ollama, SearXNG into one compose file
- Adopted EXL3 weights for faster VRAM load vs earlier experiments
- Fixed cross-container TabbyAPI auth/whitelist failures that blocked peer services
- Added persistent volume mounts for configs and data directories

## Week 2 — Model bring-up & GPU tuning

- Loaded Llama-3.1-8B-Instruct EXL3 (6.0 bpw class) into `tabby_models/`
- Tuned container env: flash attention / KV cache flags, `shm_size: 16g`, CUDA device ordering
- Documented start/stop operational loop (`docker compose down && up -d`)

## Week 3 — Public packaging

- Created sanitized publish tree (no weights, no user DBs, no tokens)
- Added SECURITY.md and example TabbyAPI config
- Mirrored narrative to GitHub portfolio rebuild (Aug 2026)
- Early public revisions accidentally included lab convenience keys — treat those as burned (see SECURITY.md)

## Week 4 — WSL2 fresh install + MCPO (Aug 2026)

Full from-scratch rebuild on clean WSL2 (same RTX 4070). Every fix is in the README WSL2 field notes.

### Issues hit and resolved

- **EXL2 dropped from TabbyAPI.** `ghcr.io/theroyallab/tabbyapi:latest` no longer loads EXL2. Switched to `turboderp/Llama-3.1-8B-Instruct-exl3` 6.0bpw.
- **NVIDIA Container Toolkit repo URL.** `deb ... noble main` → apt `Malformed entry (Component)`. Flat repo line with `amd64 /` works.
- **libcuda.so missing in container.** Triton/exllamav3 link step needs `-lcuda`. Dockerfile now symlinks `/usr/local/cuda-12.8/compat/libcuda.so`.
- **Port conflict with host Ollama.** Docker Ollama remapped to host **11435**.
- **Open WebUI only wired to Ollama.** Added `OPENAI_API_BASE_URL=http://tabbyapi:5000/v1` so the UI can use EXL3 and GGUF.
- **SillyTavern browser launch in Docker.** `browserLaunch.enabled: false`. Keep `basicAuthMode: true` or ST will refuse to start on 0.0.0.0.
- **SearXNG JSON format.** Default was HTML-only; added `json` for MCPO + Open WebUI RAG.
- **12 GB VRAM tuning.** `cache_8bit: true`, `max_seq_len: 8192`, `cache_size: 8192`. ~8 GB used by 6.0bpw EXL3, ~4 GB left for a small GGUF.

### MCPO MCP server — built and integrated

- FastMCP server at `mcp-servers/server.py` (not raw JSON-RPC)
- Seven tools via MCPO OpenAPI at `/host-master/`:
  - `list_tabbyapi_models`
  - `tabbyapi_chat`
  - `list_ollama_models`
  - `ollama_pull_model`
  - `ollama_chat`
  - `get_stack_status`
  - `searxng_search`
- MCPO config uses `/app/.venv/bin/python3` because that venv has `mcp`
- Empty `mcpServers` crashes MCPO — always ship at least one entry
- Companion `tavern_mcp.py` is the connectivity toolkit (status / self-check / wire)

### Character cards

- SillyTavern PNG character cards are first-class in this tree (`cards/`)
- Default Seraphina card + expression sprites copied from the lab ST data dir (no chats, no secrets)
- README documents import into a fresh SillyTavern

### Compose shape vs earlier public tree

- **In:** `mcpo` service (port 8001), Open WebUI dual-backend env, Ollama host 11435, character cards
- **Out of core compose:** SearXNG Redis (SearXNG runs fine without it; JSON search still works)
- Optional `docker-compose.starter.yml` overlay for extra coding tools (not required)

### Final verified state (24/24 checks)

- Six containers running, no restart loops
- TabbyAPI: EXL3 model loaded, chat ~53 tok/s processing
- Ollama: `llama3.1:8b` pulled
- Open WebUI: healthy, connected to TabbyAPI and Ollama
- SillyTavern: running with basic auth; cards importable
- SearXNG: serving with JSON format
- MCPO: 7 tool endpoints live
- GPU: 8 GB / 12 GB VRAM on RTX 4070

## v2.0.0 public release (2026-08-25)

- Hugging Face **model card** (this README YAML) brought in sync with the live lab
- GitHub first tagged release (`v2.0.0`)
- Public `mcpo/config.json` no longer contains a live TabbyAPI key (placeholder only)
- Cards, MCPO, WSL2 Dockerfile fix, and dual-backend Open WebUI are in the published compose

## Open follow-ups

- One-command bootstrap that builds the TabbyAPI image + prints next model download
- Healthcheck targets in compose
- Optional Traefik/Caddy reverse-proxy profile for LAN HTTPS
- Keep Redis as an optional compose profile if SearXNG cache becomes a bottleneck