card
Browse files
README.md
ADDED
|
@@ -0,0 +1,64 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
- zh
|
| 6 |
+
pipeline_tag: text-to-speech
|
| 7 |
+
tags:
|
| 8 |
+
- text-to-speech
|
| 9 |
+
- tts
|
| 10 |
+
- core-ai
|
| 11 |
+
- on-device
|
| 12 |
+
- ios
|
| 13 |
+
- voxcpm
|
| 14 |
+
base_model: openbmb/VoxCPM2
|
| 15 |
+
---
|
| 16 |
+
|
| 17 |
+
# VoxCPM2 2B — Core AI (on-device, 48 kHz)
|
| 18 |
+
|
| 19 |
+
[OpenBMB **VoxCPM2** (2B)](https://huggingface.co/openbmb/VoxCPM2) converted to **Apple Core AI**, running
|
| 20 |
+
fully **on-device** on iPhone (A19 Pro / iPhone 17 Pro) and Mac — no network. The 2B, 48 kHz successor to
|
| 21 |
+
[VoxCPM-0.5B-CoreAI](https://huggingface.co/mlboydaisuke/VoxCPM-0.5B-CoreAI).
|
| 22 |
+
|
| 23 |
+
A tokenizer-free diffusion TTS: a **MiniCPM4 28-layer** text-semantic LM + an **8-layer residual** acoustic
|
| 24 |
+
LM drive a **12-layer LocDiT** flow-matching diffusion head, decoded by a **48 kHz AudioVAE**. Five Core AI
|
| 25 |
+
bundles + a few host-side projections.
|
| 26 |
+
|
| 27 |
+
## What's inside
|
| 28 |
+
|
| 29 |
+
| dir | contents |
|
| 30 |
+
|---|---|
|
| 31 |
+
| `macos/` | JIT `.aimodel` bundles (Mac): int8 base/res decode + prefill, fp16 feat_decoder / feat_encoder / vocoder |
|
| 32 |
+
| `ios/` | AOT `.aimodelc` bundles (iOS `h18p`, GPU): same five + the two int8 prefill bundles |
|
| 33 |
+
| `voxcpm2_host_glue/` | embed table + projections / FSQ-512 / stop-head / fusion (`.bin` + manifest) |
|
| 34 |
+
| `tokenizer/` | the VoxCPM2 tokenizer (Llama fast) |
|
| 35 |
+
|
| 36 |
+
The backbone LMs are **weight-only int8** (the size driver); the diffusion + VAE stay **fp16** (the
|
| 37 |
+
continuous-feedback path is quant-sensitive — same split mlx-community uses).
|
| 38 |
+
|
| 39 |
+
## On-device numbers (iPhone 17 Pro, int8 + prefill + streaming)
|
| 40 |
+
|
| 41 |
+
- **RTF 1.19**, **first-audio 0.65 s**, 48 kHz, ~4.9 GB resident (increased-memory entitlement).
|
| 42 |
+
- Streaming starts after the first ~0.65 s; the 2B is ~4× the 0.5B, so RTF sits just above realtime.
|
| 43 |
+
|
| 44 |
+
## Use it
|
| 45 |
+
|
| 46 |
+
Runs through **[coreai-kit](https://github.com/john-rocky/coreai-kit)** `VoxCPM2TTS`, wired into the
|
| 47 |
+
**[coreai-model-zoo](https://github.com/john-rocky/coreai-model-zoo)** `coreai-audio` app ("Voice 2B" tab).
|
| 48 |
+
Conversion + gates + export scripts: `coreai-model-zoo/conversion/voxcpm/` (`*_v2.py`).
|
| 49 |
+
|
| 50 |
+
```swift
|
| 51 |
+
let tts = try await VoxCPM2TTS(paths: .standard(artifactsRoot: root, lm: .int8))
|
| 52 |
+
let wav = try await tts.synthesize("On device speech synthesis, running entirely on your iPhone.") // 48 kHz Float PCM
|
| 53 |
+
```
|
| 54 |
+
|
| 55 |
+
## Verification
|
| 56 |
+
|
| 57 |
+
Reimplemented in exportable Core AI overlays and gated end-to-end against the official model: backbone /
|
| 58 |
+
feat_decoder / feat_encoder **cos 1.0**, full chain **magspec 0.996**; every exported bundle engine-gated
|
| 59 |
+
**cos ≥ 0.9999**.
|
| 60 |
+
|
| 61 |
+
## License
|
| 62 |
+
|
| 63 |
+
Apache-2.0 (commercial OK), inherited from [openbmb/VoxCPM2](https://huggingface.co/openbmb/VoxCPM2).
|
| 64 |
+
Not affiliated with OpenBMB or Apple. Community port.
|