ARCHITECTURE.md

# 🏗️ DocGenie Architecture & Dependency Resolution

## 📦 Package Structure

```
docgenie/                          ← Root monorepo
├── docgenie/                      ← Core package (importable)
│   ├── __init__.py               
│   ├── generation/               ← Used by API
│   │   ├── pipeline_01/
│   │   │   └── claude_batching.py  ← ClaudeBatchedClient
│   │   ├── pipeline_03/
│   │   ├── pipeline_04/
│   │   └── utils/
│   ├── evaluation/
│   └── utils/
│
├── api/                          ← API Service (imports docgenie.*)
│   ├── main.py                   from docgenie import ENV
│   ├── worker.py                 from docgenie.generation.pipeline_01...
│   ├── utils.py                  from docgenie.generation...
│   └── requirements.txt          Extra: Redis, Supabase, Google
│
├── handwriting_service/          ← GPU Service (NO docgenie imports!)
│   ├── main.py                   ✓ Self-contained
│   ├── inference.py              ✓ No external deps
│   └── models.py
│
└── WordStylist/                  ← Model code (used by handwriting)
```

## 🔗 Dependency Graph

```
┌─────────────────────────────────────────────────────────────┐
│                     API Service                              │
│  ┌──────────────────────────────────────────────────────┐   │
│  │ api/main.py                                          │   │
│  │ ↓ imports                                             │   │
│  │ api/utils.py (call_claude_api_direct)               │   │
│  └──────────────────────────────────────────────────────┘   │
│                                                              │
│  ┌──────────────────────────────────────────────────────┐   │
│  │ api/worker.py                                        │   │
│  │ ↓ imports                                             │   │
│  │ from docgenie.generation.pipeline_01.claude_batching │   │
│  │ from docgenie.generation.constants                   │   │
│  │ from docgenie.generation.pipeline_03_process_response│   │
│  │ from docgenie.generation.pipeline_04_render_pdf...   │   │
│  │ from docgenie import ENV                             │   │
│  └──────────────────────────────────────────────────────┘   │
│                           ↓                                  │
│                    REQUIRES                                  │
│  ┌──────────────────────────────────────────────────────┐   │
│  │          docgenie/ package                           │   │
│  │          (entire generation module)                   │   │
│  └──────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│              Handwriting Service                             │
│  ┌──────────────────────────────────────────────────────┐   │
│  │ handwriting_service/main.py                          │   │
│  │ ↓ imports                                             │   │
│  │ from handwriting_service.inference import ...        │   │
│  │ from handwriting_service.models import ...           │   │
│  └──────────────────────────────────────────────────────┘   │
│                           ↓                                  │
│                    REQUIRES                                  │
│  ┌──────────────────────────────────────────────────────┐   │
│  │          WordStylist/ model                          │   │
│  │          (diffusion model code)                       │   │
│  └──────────────────────────────────────────────────────┘   │
│                                                              │
│  ✓ NO docgenie imports - completely independent!            │
└─────────────────────────────────────────────────────────────┘
```

## 🐳 Docker Build Strategy

### ❌ What Doesn't Work

```dockerfile
# ❌ WRONG: Can't copy just api/ folder
FROM python:3.11
COPY api/ /app/api/              # Missing docgenie package!
RUN pip install -r requirements.txt
CMD ["uvicorn", "main:app"]     # ImportError: No module named 'docgenie'
```

### ✅ What Works

```dockerfile
# ✅ CORRECT: Copy entire monorepo
FROM python:3.11
WORKDIR /app

# Copy everything
COPY . .

# Install docgenie as package
RUN pip install -e .             # Makes docgenie.* importable

# Install API requirements
RUN pip install -r api/requirements.txt

WORKDIR /app/api
CMD ["uvicorn", "main:app"]     # ✓ docgenie imports work!
```

## 🚢 Deployment Strategy Comparison

### Option 1: Separate Deployments (❌ Won't Work)

```
API Deployment:
├── api/ folder only
└── ❌ Missing docgenie package → ImportError

Handwriting Deployment:
├── handwriting_service/ folder
└── WordStylist/
```

**Problem:** API can't find docgenie imports!

### Option 2: Monorepo Deployment (✅ Works)

```
API Deployment:
├── docgenie/ package (core)
├── api/ service (imports docgenie)
├── setup.py
└── requirements.txt

Handwriting Deployment:
├── handwriting_service/
└── WordStylist/
```

**Solution:** Deploy entire repo for API, standalone for handwriting!

## 📁 File Structure in Containers

### API Container (Railway/EC2)
```
/app/
├── docgenie/              ← Installed as Python package
│   ├── __init__.py
│   ├── generation/
│   └── utils/
├── api/                   ← Working directory
│   ├── main.py
│   ├── worker.py
│   └── utils.py
├── setup.py
└── pyproject.toml

Python can import:
✓ from docgenie.generation.pipeline_01 import ...
✓ from docgenie import ENV
```

### Handwriting Container (RunPod)
```
/app/
├── handwriting_service/
│   ├── main.py           ← No docgenie imports!
│   ├── inference.py
│   └── models.py
└── WordStylist/          ← Model code
    ├── ldm/
    └── wordstylist_inference.py

Python can import:
✓ from handwriting_service.inference import ...
✓ No docgenie dependencies needed!
```

## 🎯 Import Resolution Flow

### API Service Import Chain

1. **FastAPI starts:**
   ```python
   uvicorn main:app
   ```

2. **main.py imports utils:**
   ```python
   from api.utils import call_claude_api_direct
   ```

3. **utils.py imports docgenie:**
   ```python
   from docgenie.generation.pipeline_01.claude_batching import ClaudeBatchedClient
   ```

4. **Python looks for docgenie:**
   - Checks sys.path
   - Finds `/app` (where `pip install -e .` installed it)
   - Loads `docgenie/__init__.py`
   - ✓ Import succeeds!

### Handwriting Service Import Chain

1. **FastAPI starts:**
   ```python
   uvicorn main:app
   ```

2. **main.py imports local modules:**
   ```python
   from handwriting_service.inference import HandwritingGenerator
   ```

3. **inference.py imports WordStylist:**
   ```python
   sys.path.insert(0, str(Path(__file__).parent.parent / "WordStylist"))
   from ldm.models.diffusion.ddpm import LatentDiffusion
   ```

4. **Python loads local modules:**
   - No external package dependencies
   - ✓ Completely self-contained!

## 🔍 Verifying Imports

### Test API Imports
```bash
# Inside API container
python3 -c "from docgenie.generation.pipeline_01.claude_batching import ClaudeBatchedClient; print('✓ Import works!')"
```

### Test Handwriting Imports
```bash
# Inside handwriting container
python3 -c "from handwriting_service.inference import HandwritingGenerator; print('✓ Import works!')"
```

## 💡 Key Insights

1. **API needs monorepo:** Must deploy entire `docgenie/` folder structure
2. **Handwriting is independent:** Can deploy just `handwriting_service/` + `WordStylist/`
3. **Docker layer caching:** Install docgenie package first, then API requirements
4. **Working directory matters:** Set WORKDIR to /app/api for API service
5. **Python package installation:** `pip install -e .` makes docgenie importable globally

## 📊 Deployment Size Comparison

| Deployment | Size | Contents |
|------------|------|----------|
| API (Railway) | ~2GB | Python 3.11 + docgenie + API deps + Playwright |
| Worker (Railway) | ~2GB | Same as API (shares image) |
| Handwriting (RunPod) | ~8GB | CUDA 11.8 + PyTorch + Diffusers + WordStylist |

**Total:** ~12GB (but cached independently)

## ✅ Checklist for Successful Deployment

- [ ] Dockerfile copies **entire monorepo** for API
- [ ] `pip install -e .` runs before API requirements
- [ ] WORKDIR set to /app/api for runtime
- [ ] Handwriting Dockerfile copies only handwriting_service/ + WordStylist/
- [ ] .dockerignore excludes data/ folders (too large)
- [ ] Environment variables set in Railway/EC2
- [ ] Redis URL points to Upstash
- [ ] HANDWRITING_SERVICE_URL points to RunPod endpoint

## 🎉 Result

```
✓ API can import from docgenie package
✓ Worker can use ClaudeBatchedClient
✓ Handwriting service runs independently
✓ All services communicate via HTTP
✓ No more ImportError!
```