Tabby-Tavern Development Log

Engineering history for the containerized local AI lab. Public tree is sanitized — no live keys, no weights, no chat logs.

Hardware baseline

GPU: NVIDIA GeForce RTX 4070 (12 GB)
Environment: WSL2 Ubuntu + Docker Compose with NVIDIA GPU passthrough
Primary inference path (v3 UNIFIED): Qwen3.5-9B GGUF on llama.cpp CUDA (qwen service)
Alternate inference (profile): TabbyAPI EXL3 + Ollama GGUF
Character frontend: SillyTavern (PNG character cards)
Tooling: MCPO + FastMCP; optional dockroot diagnostics
Coding pack: former Basecamp volumes on the same network

v3.0.0 UNIFIED — Taproot + Basecamp folded in (2026-08-25)

Hermes session 20260825_002829 merged the running Taproot Qwen stack and Basecamp coding pack into /home/jpanasuk/tabby-tavern/docker-compose.yml.

Compose project: tabby-tavern
Network: tabby-tavern_ai-network
Default: 15 services. Qwen and Postgres healthy. SearXNG / MCPO / code-server / Qdrant / Meilisearch HTTP 200.
Primary endpoint: http://qwen:8080/v1 in-network, http://localhost:1234/v1 on the host. Model alias Qwen3.5-9B.
TabbyAPI + Ollama: --profile alternate-inference (VRAM contention).
dockroot: --profile diagnostics, socket read-only, no host port.
Basecamp named volumes preserved (basecamp_*).
Public tree sanitizes host model paths to ./models/... and drops external: true so a stranger can up without those volume names pre-created.
Sell sheet rewritten from this live graph — not the old 6-service pitch and not the fake 21-service pitch.

Week 1 — Core integration

Consolidated TabbyAPI, SillyTavern, Open WebUI, Ollama, SearXNG into one compose file
Adopted EXL3 weights for faster VRAM load vs earlier experiments
Fixed cross-container TabbyAPI auth/whitelist failures that blocked peer services
Added persistent volume mounts for configs and data directories

Week 2 — Model bring-up & GPU tuning

Loaded Llama-3.1-8B-Instruct EXL3 (6.0 bpw class) into tabby_models/
Tuned container env: flash attention / KV cache flags, shm_size: 16g, CUDA device ordering
Documented start/stop operational loop (docker compose down && up -d)

Week 3 — Public packaging

Created sanitized publish tree (no weights, no user DBs, no tokens)
Added SECURITY.md and example TabbyAPI config
Mirrored narrative to GitHub portfolio rebuild (Aug 2026)
Early public revisions accidentally included lab convenience keys — treat those as burned (see SECURITY.md)

Week 4 — WSL2 fresh install + MCPO (Aug 2026)

Full from-scratch rebuild on clean WSL2 (same RTX 4070). Every fix is in the README WSL2 field notes.

Issues hit and resolved

EXL2 dropped from TabbyAPI. ghcr.io/theroyallab/tabbyapi:latest no longer loads EXL2. Switched to turboderp/Llama-3.1-8B-Instruct-exl3 6.0bpw.
NVIDIA Container Toolkit repo URL. deb ... noble main → apt Malformed entry (Component). Flat repo line with amd64 / works.
libcuda.so missing in container. Triton/exllamav3 link step needs -lcuda. Dockerfile now symlinks /usr/local/cuda-12.8/compat/libcuda.so.
Port conflict with host Ollama. Docker Ollama remapped to host 11435.
Open WebUI only wired to Ollama. Added OPENAI_API_BASE_URL=http://tabbyapi:5000/v1 so the UI can use EXL3 and GGUF.
SillyTavern browser launch in Docker. browserLaunch.enabled: false. Keep basicAuthMode: true or ST will refuse to start on 0.0.0.0.
SearXNG JSON format. Default was HTML-only; added json for MCPO + Open WebUI RAG.
12 GB VRAM tuning. cache_8bit: true, max_seq_len: 8192, cache_size: 8192. ~8 GB used by 6.0bpw EXL3, ~4 GB left for a small GGUF.

MCPO MCP server — built and integrated

FastMCP server at mcp-servers/server.py (not raw JSON-RPC)
Seven tools via MCPO OpenAPI at /host-master/:
- list_tabbyapi_models
- tabbyapi_chat
- list_ollama_models
- ollama_pull_model
- ollama_chat
- get_stack_status
- searxng_search
MCPO config uses /app/.venv/bin/python3 because that venv has mcp
Empty mcpServers crashes MCPO — always ship at least one entry
Companion tavern_mcp.py is the connectivity toolkit (status / self-check / wire)

Character cards

SillyTavern PNG character cards are first-class in this tree (cards/)
Default Seraphina card + expression sprites copied from the lab ST data dir (no chats, no secrets)
README documents import into a fresh SillyTavern

Compose shape vs earlier public tree

In: mcpo service (port 8001), Open WebUI dual-backend env, Ollama host 11435, character cards
Out of core compose: SearXNG Redis (SearXNG runs fine without it; JSON search still works)
Optional docker-compose.starter.yml overlay for extra coding tools (not required)

Final verified state (24/24 checks)

Six containers running, no restart loops
TabbyAPI: EXL3 model loaded, chat ~53 tok/s processing
Ollama: llama3.1:8b pulled
Open WebUI: healthy, connected to TabbyAPI and Ollama
SillyTavern: running with basic auth; cards importable
SearXNG: serving with JSON format
MCPO: 7 tool endpoints live
GPU: 8 GB / 12 GB VRAM on RTX 4070

v2.0.0 public release (2026-08-25)

Hugging Face model card (this README YAML) brought in sync with the live lab
GitHub first tagged release (v2.0.0)
Public mcpo/config.json no longer contains a live TabbyAPI key (placeholder only)
Cards, MCPO, WSL2 Dockerfile fix, and dual-backend Open WebUI are in the published compose

Open follow-ups

One-command bootstrap that builds the TabbyAPI image + prints next model download
Healthcheck targets in compose
Optional Traefik/Caddy reverse-proxy profile for LAN HTTPS
Keep Redis as an optional compose profile if SearXNG cache becomes a bottleneck