Model card: hfa-english-fullweak-v2, mark v1 superseded
Browse files
README.md
CHANGED
|
@@ -26,7 +26,8 @@ name and verifies the pinned SHA256 before it puts anything on disk.
|
|
| 26 |
| File | Family | Size | SHA256 |
|
| 27 |
|---|---|---:|---|
|
| 28 |
| `hfa-mandarin-fullweak-v1.ckpt` | HubertFA | 54.2 MB | `d38f7d5839e2cd93e8dec5b77b6bd3ed28aa1318d5ed069c2cd81c899dd3c057` |
|
| 29 |
-
| `hfa-english-fullweak-
|
|
|
|
| 30 |
| `nll-hubert-ls960-v1.pt` | breath head (NLL) for the English aligner | 9.9 MB | `03587e7a476aea7f137904e6034420d0a4b255d12949e85ec16b20077ae29ce8` |
|
| 31 |
|
| 32 |
## `hfa-mandarin-fullweak-v1`
|
|
@@ -93,9 +94,69 @@ aligner = HubertFATorchAligner() # resolves to this checkpoint
|
|
| 93 |
results = aligner.align([{"id": "x", "audio_path": "a.wav", "text": "中文歌词"}])
|
| 94 |
```
|
| 95 |
|
| 96 |
-
## `hfa-english-fullweak-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 97 |
|
| 98 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 99 |
front-end** — same HuBERT-base recipe as the Mandarin model, so only the
|
| 100 |
encoder path and the phoneme inventory differ.
|
| 101 |
|
|
@@ -165,7 +226,7 @@ A **breath (non-lexical) head**, not an aligner: a 2.45 M-parameter CVNT that
|
|
| 165 |
labels each 10 ms frame `None` / `AP` from the HuBERT units the aligner has
|
| 166 |
already computed, and whose breaths are spliced into the finished alignment as
|
| 167 |
`AP` without touching any other boundary. It rides on
|
| 168 |
-
`hfa-english-fullweak-
|
| 169 |
for any HubertFA aligner whose front-end is `hubert-base-ls960`.
|
| 170 |
|
| 171 |
- **Why it exists**: a breath head is bound to the SSL front-end it was
|
|
@@ -205,7 +266,7 @@ the per-singer annotation probe:
|
|
| 205 |
```python
|
| 206 |
from modules.api import HubertFATorchAligner
|
| 207 |
|
| 208 |
-
aligner = HubertFATorchAligner(ckpt="hfa-english-fullweak-
|
| 209 |
ap_detector="nll") # resolves to this head
|
| 210 |
```
|
| 211 |
|
|
|
|
| 26 |
| File | Family | Size | SHA256 |
|
| 27 |
|---|---|---:|---|
|
| 28 |
| `hfa-mandarin-fullweak-v1.ckpt` | HubertFA | 54.2 MB | `d38f7d5839e2cd93e8dec5b77b6bd3ed28aa1318d5ed069c2cd81c899dd3c057` |
|
| 29 |
+
| `hfa-english-fullweak-v2.ckpt` | HubertFA | 54.1 MB | `878de0899cb188c05537934a6f55bfd7aaabcbde1c411aba30d1e6f50a995712` |
|
| 30 |
+
| `hfa-english-fullweak-v1.ckpt` | HubertFA (superseded by v2) | 54.1 MB | `c1b2baa939a87b3ed832c97e18eba9fcb27214b39f50ff346c062601bd11c513` |
|
| 31 |
| `nll-hubert-ls960-v1.pt` | breath head (NLL) for the English aligner | 9.9 MB | `03587e7a476aea7f137904e6034420d0a4b255d12949e85ec16b20077ae29ce8` |
|
| 32 |
|
| 33 |
## `hfa-mandarin-fullweak-v1`
|
|
|
|
| 94 |
results = aligner.align([{"id": "x", "audio_path": "a.wav", "text": "中文歌词"}])
|
| 95 |
```
|
| 96 |
|
| 97 |
+
## `hfa-english-fullweak-v2`
|
| 98 |
+
|
| 99 |
+
The English model: the same alignment head over a **frozen
|
| 100 |
+
`facebook/hubert-base-ls960` front-end**, trained with the recipe of
|
| 101 |
+
`hfa-english-fullweak-v1` below on data whose only change is *where the clips
|
| 102 |
+
start*.
|
| 103 |
+
|
| 104 |
+
- **Recipe**: `configs/hfa/train_en_fullweak_v6.yaml` (identical to v1's
|
| 105 |
+
except the run name), data `configs/hfa/binarize_en_fullweak_v6.yaml`.
|
| 106 |
+
- **Training data**: 21.4 h full-label across 13 singers (as v1; Project-AIdol
|
| 107 |
+
and NUS-48E re-cut so that a silence goes whole to one neighbouring segment
|
| 108 |
+
instead of being split) + 1845 h weak-label English singing selected by
|
| 109 |
+
`singmos >= 3.5` only, so the corpus's silence-cut `sub` segments (37 %) are
|
| 110 |
+
in the pool alongside the lyric-timed `short` ones.
|
| 111 |
+
- **Step 56000**, selected by CBER on held-out singers.
|
| 112 |
+
- **Phoneme inventory / G2P**: as v1 (`english-arpabet-v1`, G2pEn); the
|
| 113 |
+
vocabulary file is byte-identical, so the two are drop-in for each other.
|
| 114 |
+
|
| 115 |
+
### Why v2 exists
|
| 116 |
+
|
| 117 |
+
v1 puts a spurious sub-millisecond `SP` at the head of about a third of clips
|
| 118 |
+
that start on the voice. The cause was traced to its weak-label pool: a
|
| 119 |
+
`wer` filter silently restricted it to lyric-timed segments, three quarters of
|
| 120 |
+
which start mid-voice, and CTC training on those teaches the shared phone
|
| 121 |
+
channels to read onset frames as "not yet a phone". Mixing in silence-cut
|
| 122 |
+
segments removes it: **32 % → 5.3 %** of voice-initial GTSinger-en clips
|
| 123 |
+
(`bench/tools/probe_leading_sp.py`; `docs/experiments/leading_sp_probe.md`).
|
| 124 |
|
| 125 |
+
### Results
|
| 126 |
+
|
| 127 |
+
Same four held-out NUS-48E singers as v1 (201 clips, 0.53 h):
|
| 128 |
+
|
| 129 |
+
| System | CBER ↓ | VER20 ↓ | VER50 ↓ | mIoU ↑ |
|
| 130 |
+
|---|---:|---:|---:|---:|
|
| 131 |
+
| **`hfa-english-fullweak-v2`** | 0.1951 | 0.1327 | 0.0634 | 0.7823 |
|
| 132 |
+
| `hfa-english-fullweak-v1` | 0.1975 | 0.1318 | 0.0616 | 0.7821 |
|
| 133 |
+
| Same recipe, full label only (21.4 h) | **0.1940** | **0.1300** | 0.0617 | **0.7833** |
|
| 134 |
+
|
| 135 |
+
A tie with v1 on alignment quality; the caveat below about the weak label
|
| 136 |
+
buying nothing measurable on clean studio audio still applies.
|
| 137 |
+
|
| 138 |
+
### Batched inference **is** bit-reproducible
|
| 139 |
+
|
| 140 |
+
Unlike v1, this checkpoint **passes** the toolkit's batch-invariance gate:
|
| 141 |
+
`bench/tools/validate_hfa_batch.py` on the same 48 items, batch 8, measures a
|
| 142 |
+
maximum boundary shift of **0.002 ms** against a 1.0 ms tolerance.
|
| 143 |
+
|
| 144 |
+
### Runtime dependency and usage
|
| 145 |
+
|
| 146 |
+
As v1: [`facebook/hubert-base-ls960`](https://huggingface.co/facebook/hubert-base-ls960)
|
| 147 |
+
must be present, and the model is named explicitly:
|
| 148 |
+
|
| 149 |
+
```python
|
| 150 |
+
from modules.api import HubertFATorchAligner
|
| 151 |
+
|
| 152 |
+
aligner = HubertFATorchAligner(ckpt="hfa-english-fullweak-v2", g2p="G2pEn")
|
| 153 |
+
results = aligner.align([{"id": "x", "audio_path": "a.wav", "text": "english lyrics"}])
|
| 154 |
+
```
|
| 155 |
+
|
| 156 |
+
## `hfa-english-fullweak-v1` (superseded)
|
| 157 |
+
|
| 158 |
+
Kept because the `CrawlSinger-en` corpus's `align_conf` column was written by
|
| 159 |
+
it; use `hfa-english-fullweak-v2` for new work. The same alignment head over a **frozen `facebook/hubert-base-ls960`
|
| 160 |
front-end** — same HuBERT-base recipe as the Mandarin model, so only the
|
| 161 |
encoder path and the phoneme inventory differ.
|
| 162 |
|
|
|
|
| 226 |
labels each 10 ms frame `None` / `AP` from the HuBERT units the aligner has
|
| 227 |
already computed, and whose breaths are spliced into the finished alignment as
|
| 228 |
`AP` without touching any other boundary. It rides on
|
| 229 |
+
`hfa-english-fullweak-v2` (or v1) as `ap_detector="nll"` and is selected automatically
|
| 230 |
for any HubertFA aligner whose front-end is `hubert-base-ls960`.
|
| 231 |
|
| 232 |
- **Why it exists**: a breath head is bound to the SSL front-end it was
|
|
|
|
| 266 |
```python
|
| 267 |
from modules.api import HubertFATorchAligner
|
| 268 |
|
| 269 |
+
aligner = HubertFATorchAligner(ckpt="hfa-english-fullweak-v2", g2p="G2pEn",
|
| 270 |
ap_detector="nll") # resolves to this head
|
| 271 |
```
|
| 272 |
|