| --- |
| license: apache-2.0 |
| tags: |
| - coreml |
| - forced-alignment |
| - wav2vec2 |
| - ctc |
| - apple-neural-engine |
| library_name: coremlit |
| --- |
| |
| # alignkit-coreml — fp16-safe CoreML conversion of wav2vec2-base-960h for forced alignment |
|
|
| A CoreML conversion of **facebook/wav2vec2-base-960h** (CTC head, 29-class character |
| vocabulary) with an **fp16-survivable log-probability tail**, for word-level forced |
| alignment. Consumed by the `alignkit` crate of the |
| [coremlit](https://github.com/findit-studio/coremlit) workspace. |
|
|
| ## Why this conversion exists |
|
|
| The stock conversion's tail is a decomposed `softmax → log(eps = 2^-149)`. That epsilon is |
| below fp16's smallest subnormal (`2^-24`), so on the Apple Neural Engine the guard rounds |
| to zero and `log(0)` saturates to **≈ −45440**, corrupting **16.7% of emission cells** and |
| shifting word timings by up to **881 ms** — silently, since the values are still finite and |
| negative. (A naive `torch.log_softmax` re-export reproduces the defect under coremltools |
| 9.0.) This conversion uses an explicitly **fused `x − logsumexp(x)`** tail: the emissions |
| are exact log-probabilities on CPU and bounded (min > −100) on every placement. |
|
|
| ## Contents |
|
|
| - `base960h_aligner.{mlpackage,mlmodelc}` — encoder weights **byte-identical** to the |
| established conversion (`weight.bin` sha256 `de51193f…`); only the tail graph changed. |
| - `CHECKSUMS.sha256` — per-file digests. |
|
|
| Input: 16 kHz mono f32, up to a 960,000-sample (60 s) window. Output: `[1, T, 29]` |
| log-probabilities, 320-sample hop. |
|
|
| ## Placement guidance (measured) |
|
|
| - **CPU: exact** log-probabilities (per-frame `logsumexp ≈ 0`) and the fastest placement |
| at this model size — the recommended default. |
| - **GPU / ANE: numerically safe with this conversion** (emissions bounded, no saturation), |
| but the wav2vec2 encoder itself is fp16-sensitive (group-norm): expect ~98% frame-argmax |
| agreement vs fp32, i.e. small word-timing differences. Consumers that require exact |
| timings should stay on CPU; the crate's runtime guard rejects any corrupted emission |
| matrix regardless of placement. |
|
|
| ## Verification (summary) |
|
|
| MIL-level fp16-guard audit clean (27 guard sites, every effective floor ≥ `2^-24`); |
| emissions min > −100 on ANE (−23.81) and GPU (−28.36) with real speech; word-timing parity |
| vs an independent ONNX implementation: median **0.0 ms** on unpadded 60 s audio (367/372 |
| boundaries within one 20 ms frame). Toolchain pinned: coremltools 9.0, torch/torchaudio |
| 2.5.1, python 3.11.15. |
|
|
| ## Upstream provenance & licensing |
|
|
| Weights: [facebook/wav2vec2-base-960h](https://huggingface.co/facebook/wav2vec2-base-960h) |
| (Apache-2.0), via torchaudio's `WAV2VEC2_ASR_BASE_960H` pipeline. This is a derivative |
| conversion: identical weights, corrected graph numerics. |
|
|
