# Codette Multi-Adapter Inference + Chat System — Implementation Plan

## Overview

Build three things inside `codette-training-lab`:

1. **HF Upload Scripts + Model Cards** — publish each trained adapter to HuggingFace
2. **Multi-Adapter Inference Engine** — loads Llama 3.1 8B + dynamically switches between 8 LoRA adapters
3. **Gradio Real-Time Chat App** — interactive UI to test any adapter with streaming responses, deployable to HF Spaces

---

## Architecture

```
codette-training-lab/
├── inference/                    ← NEW
│   ├── __init__.py
│   ├── model_loader.py          ← Core: loads base model + all adapters via PEFT
│   ├── multi_adapter_engine.py  ← Orchestrates multi-perspective generation
│   └── chat_app.py              ← Gradio UI with streaming chat
├── scripts/
│   ├── upload_adapters.py       ← NEW: push adapters to HF Hub
│   └── model_card_template.md   ← NEW: model card for each adapter
└── app.py                       ← NEW: HF Spaces entry point (launches chat_app)
```

---

## Part 1: HF Upload Scripts + Model Cards (2 files)

### `scripts/upload_adapters.py`
- Scans `adapters/` directory for trained adapter folders
- For each adapter: creates an HF repo `Raiff1982/codette-{adapter_name}`, uploads safetensors + adapter_config.json + tokenizer
- Generates a model card from template with correct metadata (base_model, datasets, pipeline_tag, etc.)
- Supports `--adapter newton` to upload one or `--all` to upload all 8

### `scripts/model_card_template.md`
- Standard HF model card with YAML frontmatter
- Fields: base_model, datasets, tags, pipeline_tag, license
- Sections: description, intended use, training details, how to use

---

## Part 2: Multi-Adapter Inference Engine (2 files)

### `inference/model_loader.py` — `CodetteModelLoader`
- Loads `meta-llama/Llama-3.1-8B-Instruct` in 4-bit QLoRA (same config as training)
- Uses PEFT's `PeftModel.from_pretrained()` to load the first adapter
- Uses `model.load_adapter("path", adapter_name="name")` for each additional adapter
- Exposes `set_active_adapter(name)` to switch between loaded adapters at runtime
- Manages tokenizer (Llama 3.1 chat template with `apply_chat_template`)
- GPU memory footprint: ~5GB base + ~20MB per adapter = ~5.2GB total (fits A10G/T4/consumer GPUs)

### `inference/multi_adapter_engine.py` — `CodetteEngine`
- Takes a `CodetteModelLoader` instance
- **Single-perspective mode**: user picks one adapter, generates with it
- **Multi-perspective mode**: runs the query through N selected adapters, collects responses, synthesizes
- **Synthesis**: combines multiple adapter responses into one unified answer (using the multi_perspective adapter or a template)
- Streaming support via `TextIteratorStreamer` for real-time token output
- Generation params: temperature, top_p, max_tokens, repetition_penalty — all configurable per adapter from `adapter_registry.yaml`

---

## Part 3: Gradio Chat Interface (2 files)

### `inference/chat_app.py` — `create_chat_app()`
- **Chat Tab**: streaming chatbot with adapter selector dropdown
  - Dropdown: "Newton", "DaVinci", "Empathy", "Philosophy", "Quantum", "RC-XI", "Multi-Perspective", "Systems", "All (synthesized)"
  - Slider controls: temperature, max tokens, top_p
  - Streaming output token-by-token
  - Chat history with system/user/assistant roles
- **Compare Tab**: side-by-side adapter comparison
  - Select 2-4 adapters, send same prompt, see responses side by side
  - Quality scores from ReasoningMetrics displayed per response
- **Status Tab**: model info, loaded adapters, GPU memory, adapter configs
- Theme: `gr.themes.Soft()` matching existing Codette aesthetic

### `app.py` (project root) — HF Spaces entry point
- Minimal: imports and launches `create_chat_app()`
- Loads adapters from HF Hub (for Spaces) or local `adapters/` directory
- Configurable via env vars: `CODETTE_ADAPTER_SOURCE=hub|local`, `HF_TOKEN`, `ADAPTER_NAMES`

---

## Key Design Decisions

1. **PEFT multi-adapter** — PEFT natively supports loading multiple LoRA adapters on one base model and switching with `set_adapter()`. No need to load 8 separate models.

2. **Streaming** — `TextIteratorStreamer` from transformers, threaded generation, yielded to Gradio chatbot for real-time display.

3. **Chat template** — Llama 3.1 uses `<|begin_of_text|><|start_header_id|>system<|end_header_id|>...` format. We use `tokenizer.apply_chat_template()` which handles this automatically.

4. **System prompts from registry** — Each adapter's system prompt comes from `adapter_registry.yaml`, injected as the system message in chat.

5. **HF Spaces compatible** — The app.py + requirements.txt are structured so deploying to a HF Space with GPU runtime works out of the box.

---

## File Count: 7 new files

| File | Purpose | ~Lines |
|------|---------|--------|
| `inference/__init__.py` | Package exports | 10 |
| `inference/model_loader.py` | Load base + adapters | 200 |
| `inference/multi_adapter_engine.py` | Generation orchestration | 250 |
| `inference/chat_app.py` | Gradio UI | 350 |
| `app.py` | HF Spaces entry point | 50 |
| `scripts/upload_adapters.py` | Push to HF Hub | 180 |
| `scripts/model_card_template.md` | Model card template | 80 |

**Total: ~1,120 lines of new code**

---

## Execution Order

1. Upload scripts + model cards (so adapters are on HF when chat loads)
2. Model loader (core inference)
3. Multi-adapter engine (orchestration)
4. Chat app + entry point (UI)
5. Test locally, then deploy to HF Space