LuxTTS CoreML: gpu/ane/6bit/long decoder graphs + 48kHz vocoder (mobius PR #75)

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+gpu/FmDecoder.mlmodelc/model.mil filter=lfs diff=lfs merge=lfs -text
+gpu-6bit/FmDecoder.mlmodelc/model.mil filter=lfs diff=lfs merge=lfs -text
+gpu-long/FmDecoder.mlmodelc/model.mil filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,116 @@
+---
+license: apache-2.0
+library_name: coreml
+tags:
+- tts
+- text-to-speech
+- coreml
+- ane
+- apple
+- on-device
+- voice-cloning
+language:
+- en
+pipeline_tag: text-to-speech
+base_model:
+- YatharthS/LuxTTS
+base_model_relation: quantized
+---
+
+# LuxTTS CoreML
+
+CoreML conversion of [YatharthS/LuxTTS](https://huggingface.co/YatharthS/LuxTTS) for on-device
+inference on Apple platforms.
+
+LuxTTS is a 48 kHz zero-shot voice-cloning TTS (~123M params): give it a short reference clip
+plus its transcript, and it speaks new text in that voice. It is based on
+[ZipVoice](https://github.com/k2-fsa/ZipVoice) (k2-fsa, Xiaomi) — a flow-matching Zipformer
+decoder run with a 4-step solver — with a dual-head Vocos vocoder from LinaCodec (ysharma3501).
+
+## Models
+
+| Path | Model | Size | Notes |
+|------|-------|------|-------|
+| `gpu/TextEncoder.mlmodelc` | Text encoder (256 tokens) | 9.5 MB | fp16 |
+| `gpu/FmDecoder.mlmodelc` | Flow-matching decoder, 1024-frame bucket | 256 MB | fp16, original graph — **macOS GPU path** |
+| `gpu-6bit/FmDecoder.mlmodelc` | 6-bit palettized decoder, 1024-frame bucket | 115 MB | full GPU speed, -0.45 dB, transcript-transparent |
+| `gpu-long/TextEncoder.mlmodelc` | Text encoder (512 tokens) | 13 MB | fp16 |
+| `gpu-long/FmDecoder.mlmodelc` | Decoder, 2048-frame bucket | 349 MB | long-form macOS (~16.8 s per bucket) |
+| `ane/TextEncoder.mlmodelc` | Text encoder (256 tokens) | 9.5 MB | fp16 |
+| `ane/FmDecoder.mlmodelc` | ANE-canonical decoder, 1024-frame bucket | 313 MB | 100% ANE placement — **iPhone path** |
+| `vocoder/Vocoder282.mlmodelc` | Vocos vocoder, 282 gen frames (~3.0 s) | 33 MB | ISTFT + 24k→48k resample + crossover in-graph |
+| `vocoder/Vocoder555.mlmodelc` | Vocos vocoder, 555 gen frames (~5.9 s) | 33 MB | matches the full 1024-frame bucket |
+| `tokens.txt` / `config.json` | Phoneme token table + model config | — | from upstream LuxTTS |
+
+### Two decoder graphs — pick by compute unit
+
+- **`gpu/`** — the original graph layout. Fastest on Mac GPU. **Do not run it on the ANE**:
+  the seq-first rel-pos attention path loses precision there and produces corrupted audio.
+- **`ane/`** — ANE-canonical rewrite (channels-first `(1,C,1,S)`, 1×1-conv projections,
+  constant rel-pos basis, split depthwise kernels). 100% ANE op placement, zero CPU fallback.
+  This is the iPhone path; it is slower than the original graph on GPU, so keep both.
+
+All decoder inputs/outputs are identical across graphs
+(`t, x, text_condition, speech_condition, guidance_scale, padding_mask → v`).
+
+## Performance (M5 Pro, macOS)
+
+| path | fm step | core RTFx (full bucket) | steady footprint |
+|------|---------|-------------------------|------------------|
+| `gpu/` fp16, GPU | 14.7 ms | 92x (~5.9 s gen) | 996 MB |
+| `gpu-6bit/`, GPU | 15.0 ms | ~92x | 419 MB |
+| `ane/` fp16, ANE | 54.6 ms | 27x | **25.5 MB** |
+
+- Vocoder: 2.34 ms on GPU (282 frames); end-to-end ~64 ms for 3 s of audio (~47x realtime).
+- Quality vs PyTorch reference: log-mel cosine 0.99925 (GPU path); whisper-base transcripts
+  word-identical for all shipped variants.
+- Footprints are `phys_footprint` from a Swift host; the ANE graph keeps weights and
+  activations in ANE-managed memory, hence the 25.5 MB jetsam-visible footprint.
+
+## Fixed-shape buckets
+
+CoreML graphs are compiled at fixed shapes. Frame budget covers prompt + generated speech
+(86.13 frames/s):
+
+- `gpu/`, `ane/`: 1024 frames (~5.9 s generated after a 5 s prompt), 256 text tokens
+- `gpu-long/`: 2048 frames (~16.8 s generated), 512 text tokens
+- vocoder: fixed at 282 or 555 generated frames
+
+Chunk long text at sentence boundaries for the 1024-frame bucket on iPhone.
+
+## Usage notes
+
+The CoreML graphs cover the text encoder, flow-matching decoder, and vocoder (ISTFT,
+24k→48k resample, and the Linkwitz crossover are all in-graph). The host must provide:
+
+- Phoneme tokenization against `tokens.txt` (espeak-compatible token set) — the frontend is
+  phoneme-token based, not raw text.
+- Duration expansion of encoder output into the frame grid (upstream `pad_labels` keeps S+1
+  token rows; the pad slot's embedding fills remainder frames).
+- The 4-step anchor-Euler flow solver with `t_shift = 0.5` and `guidance_scale = 3.0`.
+- Prompt trimming at a **speech pause** (VAD boundary), not a fixed duration — a prompt
+  hard-cut mid-phrase makes the model elide sentence-initial words.
+- `speed` default **1.0** (upstream's `generate()` default of 1.3 clips sentence onsets).
+
+Swift integration in [FluidAudio](https://github.com/FluidInference/FluidAudio) is upcoming.
+
+## Known limitations
+
+- Fixed-shape buckets only (no enumerated/flexible shapes yet).
+- An int8-linear variant exists but crashes MPSGraph on macOS 26.5 GPU — not shipped.
+- The ANE graph is ~0.5 dB softer than the GPU graph (fp16 accumulation over 16 layers ×
+  4 solver steps); transcripts are unaffected.
+
+## Attribution & license
+
+- [LuxTTS](https://huggingface.co/YatharthS/LuxTTS) by YatharthS — Apache-2.0.
+- Based on [ZipVoice](https://github.com/k2-fsa/ZipVoice) by k2-fsa (Xiaomi) — Apache-2.0.
+- Vocoder from LinaCodec (ysharma3501).
+- CoreML conversion by [FluidInference](https://huggingface.co/FluidInference); conversion
+  source lives in the mobius repo (`models/tts/zipvoice`, PR #75).
+
+## References
+
+- https://huggingface.co/YatharthS/LuxTTS
+- https://github.com/k2-fsa/ZipVoice
+- https://github.com/FluidInference/FluidAudio

ane/FmDecoder.mlmodelc/analytics/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2ffcfdf275e23501930c09512a40bd22a231b76f80c07e147c62c0ee3608bb8
+size 243

ane/FmDecoder.mlmodelc/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b3e3f177caf2a08427eb5c4f02934abc75cb958753eda65a1a7f59448c293b54
+size 504

ane/FmDecoder.mlmodelc/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4c50bb5d66cb021efb932d98945ae52ea97670167e453703812d7cab43f08f46
+size 326937480

ane/TextEncoder.mlmodelc/analytics/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:11fe573ed7a0a909c6c6391b33369b8ffafb7fb6ccad0e2f551d49f75b42d648
+size 243

ane/TextEncoder.mlmodelc/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:14f0113bcd36a9b3cdcb44399eb27620621ecbc0a42af7d7d574629db5f80f48
+size 408

ane/TextEncoder.mlmodelc/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3f390f15b56cb218ee362ab340981c694da490640d8cb8514f89c9551848dde6
+size 8486088

config.json

@@ -0,0 +1,26 @@
+{
+  "model" : {
+    "fm_decoder_downsampling_factor" : [1,2,4,2,1],
+    "fm_decoder_num_layers" : [2,2,4,4,4],
+    "fm_decoder_cnn_module_kernel" : [31,15,7,15,31],
+    "fm_decoder_feedforward_dim" : 1536,
+    "fm_decoder_num_heads" : 4,
+    "fm_decoder_dim" : 512,
+    "text_encoder_num_layers" : 4,
+    "text_encoder_feedforward_dim" : 512,
+    "text_encoder_cnn_module_kernel" : 9,
+    "text_encoder_num_heads" : 4,
+    "text_encoder_dim" : 192,
+    "query_head_dim" : 32,
+    "value_head_dim" : 12,
+    "pos_head_dim" : 4,
+    "pos_dim" : 48,
+    "time_embed_dim" : 192,
+    "text_embed_dim" : 192,
+    "feat_dim": 100
+  },
+  "feature" : {
+    "sampling_rate": 24000,
+    "type": "vocos"
+  }
+}

gpu-6bit/FmDecoder.mlmodelc/analytics/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f2d8ab4200977512e88eec487d3cec5bab7d6d52091bc0fa9a424a92da38fe40
+size 243

gpu-6bit/FmDecoder.mlmodelc/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8140f24275134402260c84cabf5e0877518b3ba9f579cfb455fc41e5e4511966
+size 504

gpu-6bit/FmDecoder.mlmodelc/model.mil

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f6dd2da40d9349ff7e399c2679ebb9f7be19f800fcab4ed429f73a95e904b03a
+size 30658794

gpu-6bit/FmDecoder.mlmodelc/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6d4399f0b2f76e46f342ca0cc404569a03d5b57286d9fd20cec706d6da693ab1
+size 89520712

gpu-long/FmDecoder.mlmodelc/analytics/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:779a05fc8245f2c1cca9822f8a16a77e7cafc1848a54152d52833db9a7573a3e
+size 243

gpu-long/FmDecoder.mlmodelc/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7efe79a4908f20b5dac5653c20822166c5b1d4192fe74e001869937025a426f0
+size 504

gpu-long/FmDecoder.mlmodelc/model.mil

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:62fc68e54b6c84680b3682c5761169eeca94b45d766f85cefc3ffa6b7706c87f
+size 127949488

gpu-long/FmDecoder.mlmodelc/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eae0f3d2d0641886c81dc874ea9bc461fb5353811bbe7ef9ccbe2b5190225be9
+size 238321352

gpu-long/TextEncoder.mlmodelc/analytics/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dbfc4b26648418dddd03f319e9e14d7023adf114601acde8bda9ffbc13a8ce84
+size 243

gpu-long/TextEncoder.mlmodelc/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cec8fd8d7a079d90b22c08ada7fc465221dd97c980a2c59a2a85e20c6610d50f
+size 408

gpu-long/TextEncoder.mlmodelc/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad4dab7ffc6af12f492b8c0e8493588a75491338f3858cd7205721976570d2dc
+size 8551624

gpu/FmDecoder.mlmodelc/analytics/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1ac8703694344fb99d45b7a99ef10734a215c043e4ca02426e3fa37c60e2cef3
+size 243

gpu/FmDecoder.mlmodelc/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d944ef10ac85db1ee2bb459289fb17319dedfe9893745cffabc45530a51b2496
+size 504

gpu/FmDecoder.mlmodelc/model.mil

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1979761db1b3ef75784bf47c3f273e72d980a449572918b8f0dc8123e21d380b
+size 30583551

gpu/FmDecoder.mlmodelc/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c633c48857ea341a8fb6d6b49167dbd79f2c50fb1ab025a4aa56573ed31c242e
+size 237665992

gpu/TextEncoder.mlmodelc/analytics/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:11fe573ed7a0a909c6c6391b33369b8ffafb7fb6ccad0e2f551d49f75b42d648
+size 243

gpu/TextEncoder.mlmodelc/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1bc2506cd274455bb1e447bb42ed578374d1e9cdd40cbafdc53908e4ca2e27c6
+size 408

gpu/TextEncoder.mlmodelc/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3f390f15b56cb218ee362ab340981c694da490640d8cb8514f89c9551848dde6
+size 8486088

tokens.txt

@@ -0,0 +1,360 @@
+_	0
+^	1
+$	2
+ 	3
+!	4
+'	5
+(	6
+)	7
+,	8
+-	9
+.	10
+:	11
+;	12
+?	13
+a	14
+b	15
+c	16
+d	17
+e	18
+f	19
+h	20
+i	21
+j	22
+k	23
+l	24
+m	25
+n	26
+o	27
+p	28
+q	29
+r	30
+s	31
+t	32
+u	33
+v	34
+w	35
+x	36
+y	37
+z	38
+æ	39
+ç	40
+ð	41
+ø	42
+ħ	43
+ŋ	44
+œ	45
+ǀ	46
+ǁ	47
+ǂ	48
+ǃ	49
+ɐ	50
+ɑ	51
+ɒ	52
+ɓ	53
+ɔ	54
+ɕ	55
+ɖ	56
+ɗ	57
+ɘ	58
+ə	59
+ɚ	60
+ɛ	61
+ɜ	62
+ɞ	63
+ɟ	64
+ɠ	65
+ɡ	66
+ɢ	67
+ɣ	68
+ɤ	69
+ɥ	70
+ɦ	71
+ɧ	72
+ɨ	73
+ɪ	74
+ɫ	75
+ɬ	76
+ɭ	77
+ɮ	78
+ɯ	79
+ɰ	80
+ɱ	81
+ɲ	82
+ɳ	83
+ɴ	84
+ɵ	85
+ɶ	86
+ɸ	87
+ɹ	88
+ɺ	89
+ɻ	90
+ɽ	91
+ɾ	92
+ʀ	93
+ʁ	94
+ʂ	95
+ʃ	96
+ʄ	97
+ʈ	98
+ʉ	99
+ʊ	100
+ʋ	101
+ʌ	102
+ʍ	103
+ʎ	104
+ʏ	105
+ʐ	106
+ʑ	107
+ʒ	108
+ʔ	109
+ʕ	110
+ʘ	111
+ʙ	112
+ʛ	113
+ʜ	114
+ʝ	115
+ʟ	116
+ʡ	117
+ʢ	118
+ʲ	119
+ˈ	120
+ˌ	121
+ː	122
+ˑ	123
+˞	124
+β	125
+θ	126
+χ	127
+ᵻ	128
+ⱱ	129
+0	130
+1	131
+2	132
+3	133
+4	134
+5	135
+6	136
+7	137
+8	138
+9	139
+̧	140
+̃	141
+̪	142
+̯	143
+̩	144
+ʰ	145
+ˤ	146
+ε	147
+↓	148
+#	149
+"	150
+↑	151
+̺	152
+̻	153
+g	154
+ʦ	155
+X	156
+̝	157
+̊	158
+a1	159
+a2	160
+a3	161
+a4	162
+a5	163
+ai1	164
+ai2	165
+ai3	166
+ai4	167
+ai5	168
+an1	169
+an2	170
+an3	171
+an4	172
+an5	173
+ang1	174
+ang2	175
+ang3	176
+ang4	177
+ang5	178
+ao1	179
+ao2	180
+ao3	181
+ao4	182
+ao5	183
+b0	184
+c0	185
+ch0	186
+d0	187
+e1	188
+e2	189
+e3	190
+e4	191
+e5	192
+ei1	193
+ei2	194
+ei3	195
+ei4	196
+ei5	197
+en1	198
+en2	199
+en3	200
+en4	201
+en5	202
+eng1	203
+eng2	204
+eng3	205
+eng4	206
+eng5	207
+er2	208
+er3	209
+er4	210
+er5	211
+f0	212
+g0	213
+g2	214
+g3	215
+g4	216
+g5	217
+h0	218
+i1	219
+i2	220
+i3	221
+i4	222
+i5	223
+ia1	224
+ia2	225
+ia3	226
+ia4	227
+ia5	228
+ian1	229
+ian2	230
+ian3	231
+ian4	232
+ian5	233
+iang1	234
+iang2	235
+iang3	236
+iang4	237
+iang5	238
+iao1	239
+iao2	240
+iao3	241
+iao4	242
+iao5	243
+ie1	244
+ie2	245
+ie3	246
+ie4	247
+ie5	248
+in1	249
+in2	250
+in3	251
+in4	252
+in5	253
+ing1	254
+ing2	255
+ing3	256
+ing4	257
+ing5	258
+iong1	259
+iong2	260
+iong3	261
+iong4	262
+iu1	263
+iu2	264
+iu3	265
+iu4	266
+iu5	267
+j0	268
+k0	269
+l0	270
+m0	271
+m1	272
+m2	273
+m4	274
+m5	275
+n0	276
+n2	277
+n3	278
+n4	279
+n5	280
+ng5	281
+o1	282
+o2	283
+o3	284
+o4	285
+o5	286
+ong1	287
+ong2	288
+ong3	289
+ong4	290
+ong5	291
+ou1	292
+ou2	293
+ou3	294
+ou4	295
+ou5	296
+p0	297
+q0	298
+r0	299
+s0	300
+sh0	301
+t0	302
+u1	303
+u2	304
+u3	305
+u4	306
+u5	307
+ua1	308
+ua2	309
+ua3	310
+ua4	311
+uai1	312
+uai2	313
+uai3	314
+uai4	315
+uai5	316
+uan1	317
+uan2	318
+uan3	319
+uan4	320
+uan5	321
+uang1	322
+uang2	323
+uang3	324
+uang4	325
+uang5	326
+ue1	327
+ue2	328
+ue3	329
+ue4	330
+ui1	331
+ui2	332
+ui3	333
+ui4	334
+ui5	335
+un1	336
+un2	337
+un3	338
+un4	339
+un5	340
+uo1	341
+uo2	342
+uo3	343
+uo4	344
+uo5	345
+v2	346
+v3	347
+v4	348
+ve3	349
+ve4	350
+w0	351
+x0	352
+y0	353
+z0	354
+zh0	355
+ê1	356
+ê2	357
+ê3	358
+ê4	359

vocoder/Vocoder282.mlmodelc/analytics/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2bde9c2ef71dafba21664999d41cc53f6fe399b500f1eee97f9ee76286862d79
+size 243

vocoder/Vocoder282.mlmodelc/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d15e4be675e39f6c9645d7776697bf08bd77e33bb595e59a958b8d2de8e0d747
+size 370

vocoder/Vocoder282.mlmodelc/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:def01cc84d75317b66ee1f17881a3d74b9036d03d9a4277cc25baab37f9076ef
+size 34407998

vocoder/Vocoder555.mlmodelc/analytics/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd28319df22fb7a28a4d1069b1217d4efcd1729972fc0fec0ed1e3a6e70f7bb0
+size 243

vocoder/Vocoder555.mlmodelc/coremldata.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d086b412be9577366d7b83fd544957ccc894f662457a9be56b10641a5942eee
+size 370

vocoder/Vocoder555.mlmodelc/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c35b601e5d0cd95759338e2cff79995fbb515352333cbbcb4a8f75d0d0e4ce7a
+size 34827326