README.md

---
language:
- multilingual
- en
- zh
- fr
- de
- es
- ja
- ko
- pt
- it
- ru
- ar
- hi
- tr
- pl
- nl
- sv
- da
- fi
- no
- cs
- ro
- hu
tags:
- text-to-speech
- executorch
- on-device
- android
- voice-cloning
- chatterbox
license: apache-2.0
---

# Chatterbox Multilingual TTS — ExecuTorch Models

Pre-exported `.pte` model files for running [Resemble AI's Chatterbox Multilingual TTS](https://github.com/resemble-ai/chatterbox) fully on-device using [ExecuTorch](https://pytorch.org/executorch/).

**📦 Code & export scripts:** [acul3/chatterbox-executorch](https://github.com/acul3/chatterbox-executorch) on GitHub

---

## What's Here

9 ExecuTorch `.pte` files covering the complete TTS pipeline — from text input to 24kHz waveform — with zero PyTorch runtime required:

| File | Size | Backend | Precision | Stage |
|------|------|---------|-----------|-------|
| `voice_encoder.pte` | 7 MB | portable | FP32 | Speaker embedding |
| `xvector_encoder.pte` | 27 MB | portable | FP32 | X-vector conditioning |
| `t3_cond_speech_emb.pte` | 49 MB | portable | FP32 | Speech token embedding |
| `t3_cond_enc.pte` | 18 MB | portable | FP32 | Text/conditioning encoder |
| `t3_prefill.pte` | 1010 MB | XNNPACK | **FP16** | T3 Transformer prefill |
| `t3_decode.pte` | 1002 MB | XNNPACK | **FP16** | T3 Transformer decode |
| `s3gen_encoder.pte` | 178 MB | portable | FP32 | S3Gen Conformer encoder |
| `cfm_step.pte` | 274 MB | XNNPACK | FP32 | CFM flow matching step |
| `hifigan.pte` | 84 MB | XNNPACK | FP32 | HiFiGAN vocoder |
| **Total** | **~2.6 GB** | | | |

---

## Quick Download

```python
from huggingface_hub import snapshot_download

snapshot_download(
    "acul3/chatterbox-executorch",
    local_dir="et_models",
    repo_type="model"
)
```

---

## Pipeline Overview

```
Text → MTLTokenizer → text tokens
Reference Audio → VoiceEncoder + CAMPPlus → speaker conditioning
                          ↓
              T3 Prefill (LlamaModel, conditioned)
                          ↓
              T3 Decode (autoregressive, ~100 tokens)
                          ↓
              S3Gen Encoder (Conformer)
                          ↓
              CFM Step × 2 (flow matching)
                          ↓
              HiFiGAN (vocoder, chunked)
                          ↓
              24kHz PCM waveform 🎵
```

---

## Key Technical Notes

- **T3 Decode** uses a manually unrolled 30-layer Llama forward pass with static KV cache (`torch.where` writes) — bypasses HF `DynamicCache` for `torch.export` compatibility
- **HiFiGAN** uses a manual real-valued DFT (cosine/sine matrix multiply) — replaces `torch.stft`/`torch.istft` which XNNPACK doesn't support
- **T3 models** are FP16 (XNNPACK half-precision kernels) — ~half the size of FP32 with near-identical quality
- **Fixed shapes:** CFM expects `T_MEL=2200`, HiFiGAN expects `T_MEL=300` (use chunked processing for longer audio)

---

## Usage

See the GitHub repo for full inference code: [acul3/chatterbox-executorch](https://github.com/acul3/chatterbox-executorch)

```bash
# Clone code
git clone https://github.com/acul3/chatterbox-executorch.git
cd chatterbox-executorch

# Download models (this repo)
python -c "
from huggingface_hub import snapshot_download
snapshot_download('acul3/chatterbox-executorch', local_dir='et_models', repo_type='model')
"

# Run full PTE inference
python test_true_full_pte.py
```

---

## Android Integration

These models are designed for Android deployment via the [ExecuTorch Android SDK](https://pytorch.org/executorch/stable/android-setup.html). Load with:

```kotlin
val module = Module.load(context.filesDir.path + "/t3_prefill.pte")
```

With QNN/NPU delegation on a Snapdragon device, expect **10–50× speedup** over the CPU timings below.

## Performance (Jetson AGX Orin, CPU only)

| Stage | Time |
|-------|------|
| Voice encoding | ~1s |
| T3 prefill | ~22s |
| T3 decode (~100 tokens) | ~800s total (~8s/token) |
| S3Gen encoder | ~2s |
| CFM (2 steps) | ~40s |
| HiFiGAN | ~10s/chunk |

---

## License

Model weights are derived from [Resemble AI's Chatterbox](https://github.com/resemble-ai/chatterbox). The export pipeline code is MIT licensed. Please refer to the original [Chatterbox license](https://github.com/resemble-ai/chatterbox/blob/main/LICENSE) for model weights usage terms.