pymaster commited on
Commit
ded3fd4
·
verified ·
1 Parent(s): 04a20b8

Model card: hfa-english-fullweak-v2, mark v1 superseded

Browse files
Files changed (1) hide show
  1. README.md +66 -5
README.md CHANGED
@@ -26,7 +26,8 @@ name and verifies the pinned SHA256 before it puts anything on disk.
26
  | File | Family | Size | SHA256 |
27
  |---|---|---:|---|
28
  | `hfa-mandarin-fullweak-v1.ckpt` | HubertFA | 54.2 MB | `d38f7d5839e2cd93e8dec5b77b6bd3ed28aa1318d5ed069c2cd81c899dd3c057` |
29
- | `hfa-english-fullweak-v1.ckpt` | HubertFA | 54.1 MB | `c1b2baa939a87b3ed832c97e18eba9fcb27214b39f50ff346c062601bd11c513` |
 
30
  | `nll-hubert-ls960-v1.pt` | breath head (NLL) for the English aligner | 9.9 MB | `03587e7a476aea7f137904e6034420d0a4b255d12949e85ec16b20077ae29ce8` |
31
 
32
  ## `hfa-mandarin-fullweak-v1`
@@ -93,9 +94,69 @@ aligner = HubertFATorchAligner() # resolves to this checkpoint
93
  results = aligner.align([{"id": "x", "audio_path": "a.wav", "text": "中文歌词"}])
94
  ```
95
 
96
- ## `hfa-english-fullweak-v1`
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
97
 
98
- The same alignment head over a **frozen `facebook/hubert-base-ls960`
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
99
  front-end** — same HuBERT-base recipe as the Mandarin model, so only the
100
  encoder path and the phoneme inventory differ.
101
 
@@ -165,7 +226,7 @@ A **breath (non-lexical) head**, not an aligner: a 2.45 M-parameter CVNT that
165
  labels each 10 ms frame `None` / `AP` from the HuBERT units the aligner has
166
  already computed, and whose breaths are spliced into the finished alignment as
167
  `AP` without touching any other boundary. It rides on
168
- `hfa-english-fullweak-v1` as `ap_detector="nll"` and is selected automatically
169
  for any HubertFA aligner whose front-end is `hubert-base-ls960`.
170
 
171
  - **Why it exists**: a breath head is bound to the SSL front-end it was
@@ -205,7 +266,7 @@ the per-singer annotation probe:
205
  ```python
206
  from modules.api import HubertFATorchAligner
207
 
208
- aligner = HubertFATorchAligner(ckpt="hfa-english-fullweak-v1", g2p="G2pEn",
209
  ap_detector="nll") # resolves to this head
210
  ```
211
 
 
26
  | File | Family | Size | SHA256 |
27
  |---|---|---:|---|
28
  | `hfa-mandarin-fullweak-v1.ckpt` | HubertFA | 54.2 MB | `d38f7d5839e2cd93e8dec5b77b6bd3ed28aa1318d5ed069c2cd81c899dd3c057` |
29
+ | `hfa-english-fullweak-v2.ckpt` | HubertFA | 54.1 MB | `878de0899cb188c05537934a6f55bfd7aaabcbde1c411aba30d1e6f50a995712` |
30
+ | `hfa-english-fullweak-v1.ckpt` | HubertFA (superseded by v2) | 54.1 MB | `c1b2baa939a87b3ed832c97e18eba9fcb27214b39f50ff346c062601bd11c513` |
31
  | `nll-hubert-ls960-v1.pt` | breath head (NLL) for the English aligner | 9.9 MB | `03587e7a476aea7f137904e6034420d0a4b255d12949e85ec16b20077ae29ce8` |
32
 
33
  ## `hfa-mandarin-fullweak-v1`
 
94
  results = aligner.align([{"id": "x", "audio_path": "a.wav", "text": "中文歌词"}])
95
  ```
96
 
97
+ ## `hfa-english-fullweak-v2`
98
+
99
+ The English model: the same alignment head over a **frozen
100
+ `facebook/hubert-base-ls960` front-end**, trained with the recipe of
101
+ `hfa-english-fullweak-v1` below on data whose only change is *where the clips
102
+ start*.
103
+
104
+ - **Recipe**: `configs/hfa/train_en_fullweak_v6.yaml` (identical to v1's
105
+ except the run name), data `configs/hfa/binarize_en_fullweak_v6.yaml`.
106
+ - **Training data**: 21.4 h full-label across 13 singers (as v1; Project-AIdol
107
+ and NUS-48E re-cut so that a silence goes whole to one neighbouring segment
108
+ instead of being split) + 1845 h weak-label English singing selected by
109
+ `singmos >= 3.5` only, so the corpus's silence-cut `sub` segments (37 %) are
110
+ in the pool alongside the lyric-timed `short` ones.
111
+ - **Step 56000**, selected by CBER on held-out singers.
112
+ - **Phoneme inventory / G2P**: as v1 (`english-arpabet-v1`, G2pEn); the
113
+ vocabulary file is byte-identical, so the two are drop-in for each other.
114
+
115
+ ### Why v2 exists
116
+
117
+ v1 puts a spurious sub-millisecond `SP` at the head of about a third of clips
118
+ that start on the voice. The cause was traced to its weak-label pool: a
119
+ `wer` filter silently restricted it to lyric-timed segments, three quarters of
120
+ which start mid-voice, and CTC training on those teaches the shared phone
121
+ channels to read onset frames as "not yet a phone". Mixing in silence-cut
122
+ segments removes it: **32 % → 5.3 %** of voice-initial GTSinger-en clips
123
+ (`bench/tools/probe_leading_sp.py`; `docs/experiments/leading_sp_probe.md`).
124
 
125
+ ### Results
126
+
127
+ Same four held-out NUS-48E singers as v1 (201 clips, 0.53 h):
128
+
129
+ | System | CBER ↓ | VER20 ↓ | VER50 ↓ | mIoU ↑ |
130
+ |---|---:|---:|---:|---:|
131
+ | **`hfa-english-fullweak-v2`** | 0.1951 | 0.1327 | 0.0634 | 0.7823 |
132
+ | `hfa-english-fullweak-v1` | 0.1975 | 0.1318 | 0.0616 | 0.7821 |
133
+ | Same recipe, full label only (21.4 h) | **0.1940** | **0.1300** | 0.0617 | **0.7833** |
134
+
135
+ A tie with v1 on alignment quality; the caveat below about the weak label
136
+ buying nothing measurable on clean studio audio still applies.
137
+
138
+ ### Batched inference **is** bit-reproducible
139
+
140
+ Unlike v1, this checkpoint **passes** the toolkit's batch-invariance gate:
141
+ `bench/tools/validate_hfa_batch.py` on the same 48 items, batch 8, measures a
142
+ maximum boundary shift of **0.002 ms** against a 1.0 ms tolerance.
143
+
144
+ ### Runtime dependency and usage
145
+
146
+ As v1: [`facebook/hubert-base-ls960`](https://huggingface.co/facebook/hubert-base-ls960)
147
+ must be present, and the model is named explicitly:
148
+
149
+ ```python
150
+ from modules.api import HubertFATorchAligner
151
+
152
+ aligner = HubertFATorchAligner(ckpt="hfa-english-fullweak-v2", g2p="G2pEn")
153
+ results = aligner.align([{"id": "x", "audio_path": "a.wav", "text": "english lyrics"}])
154
+ ```
155
+
156
+ ## `hfa-english-fullweak-v1` (superseded)
157
+
158
+ Kept because the `CrawlSinger-en` corpus's `align_conf` column was written by
159
+ it; use `hfa-english-fullweak-v2` for new work. The same alignment head over a **frozen `facebook/hubert-base-ls960`
160
  front-end** — same HuBERT-base recipe as the Mandarin model, so only the
161
  encoder path and the phoneme inventory differ.
162
 
 
226
  labels each 10 ms frame `None` / `AP` from the HuBERT units the aligner has
227
  already computed, and whose breaths are spliced into the finished alignment as
228
  `AP` without touching any other boundary. It rides on
229
+ `hfa-english-fullweak-v2` (or v1) as `ap_detector="nll"` and is selected automatically
230
  for any HubertFA aligner whose front-end is `hubert-base-ls960`.
231
 
232
  - **Why it exists**: a breath head is bound to the SSL front-end it was
 
266
  ```python
267
  from modules.api import HubertFATorchAligner
268
 
269
+ aligner = HubertFATorchAligner(ckpt="hfa-english-fullweak-v2", g2p="G2pEn",
270
  ap_detector="nll") # resolves to this head
271
  ```
272