Publish weights repackaged for browser WebGPU inference

README.md

@@ -0,0 +1,121 @@
+---
+license: mit
+base_model: Aratako/MioCodec-25Hz-24kHz
+base_model_relation: adapter
+language:
+  - ja
+  - en
+tags:
+  - speaker-embedding
+  - webgpu
+  - browser
+  - voice-cloning
+---
+
+# MioCodec's speaker encoder, in one file for the browser
+
+The global-embedding path of
+[Aratako/MioCodec-25Hz-24kHz](https://huggingface.co/Aratako/MioCodec-25Hz-24kHz)'s
+encoder — reference audio in, a 128-dimensional speaker embedding out —
+collected into a single f32 safetensors so a browser can fetch it in one
+request. Built for [voxshot](https://github.com/m96-chan/voxshot).
+
+**This is a repackaging, not a new model.** No weight here was trained; every
+tensor is copied from one of the two sources below.
+
+| | |
+| --- | ---: |
+| `encoder-weights.safetensors` | 117,303,232 B, 99 tensors, 29,322,838 params |
+| `encoder-weights.json` | 8,905 B — tensor index and provenance |
+
+## Why it exists
+
+The weights for this one path live in **two different places**: the
+`GlobalEncoder` (50 tensors) is in MioCodec's own checkpoint, while the WavLM
+front end it runs on top of is a torchaudio download. A browser should not
+have to know that, nor fetch a 378 MB `.pth` to use 4 MB of it. So the path is
+collected once, ahead of time, into one file.
+
+Only the **global** branch is included. MioCodec's encoder also produces
+content tokens — that half is not here, and is not needed for speaker
+embedding.
+
+## What the path is
+
+```
+24 kHz mono
+  → symmetric zero pad
+  → resample 24k→16k          (torchaudio's polyphase filter, precomputed
+                               here as a conv1d kernel [2, 1, 23], stride 3)
+  → WavLM feature extractor    (7 strided convs, GroupNorm on the first, GELU)
+  → LayerNorm + Linear 512→768
+  → positional conv            (grouped k=128, weight-norm already folded)
+  → transformer layers 1 and 2 (12 heads; bucketed, gated relative-position
+                               bias — the global branch reads only these two)
+  → mean of the two layers
+  → ConvNeXt backbone, 4 blocks, 768→384
+  → attentive statistics pooling → Linear 768→128 → LayerNorm
+  → 128-dim embedding
+```
+
+Two details a reimplementation gets wrong and this file cannot tell you, so
+they are written down here:
+
+- torchaudio's Base+ WavLM is **post-norm**, so an encoder-level LayerNorm runs
+  between the positional-conv residual and layer 1.
+- the pooling softmax is over **time**, not channels.
+
+## Naming
+
+WavLM tensors keep torchaudio's own names under a `wavlm.` prefix, the
+`GlobalEncoder` tensors keep MioCodec's names under `global_encoder.`, and the
+precomputed resampler is `resample.kernel`. `encoder-weights.json` lists every
+tensor with its shape.
+
+The positional convolution's weight normalization is **already folded** into
+its weight — there is no separate `weight_g` / `weight_v` to combine.
+
+## Provenance
+
+Produced by [`spike/miocodec/export_encoder_weights.py`](https://github.com/m96-chan/voxshot/blob/main/spike/miocodec/export_encoder_weights.py) from:
+
+```
+Aratako/MioCodec-25Hz-24kHz  model.safetensors
+  523,087,956 bytes
+  sha256 60483759cde136451d53ff5a2f7e283c015758ab540de9037b5d4519366a7705
+
+torchaudio WAVLM_BASE_PLUS  wavlm_base_plus.pth
+  377,604,347 bytes
+  sha256 136a3e720c04f2c77bf7a4dc6a3868b14d5a2c145a988114b733cb1a8428be98
+```
+
+Both digests are recorded in `encoder-weights.json` and cross-checked at export
+time against the golden this port is verified with, so the weights and the
+checkpoints they came from cannot drift apart unnoticed.
+
+## Accuracy
+
+The TypeScript port that consumes this file was checked stage by stage against
+the reference encoder running under torch, on two real reference clips and one
+synthetic input. Every stage agrees to 1e-4 relative to its own peak, except
+four ConvNeXt stages on one clip which reach 1e-4 in the reference's *own*
+f32-versus-f64 comparison — conditioning of that clip, not port error. The
+final embedding agrees to 7.3e-7.
+
+## License and attribution
+
+MIT, which both sources carry.
+
+- **MioCodec**: [Aratako/MioCodec-25Hz-24kHz](https://huggingface.co/Aratako/MioCodec-25Hz-24kHz) by Chihiro Arata, MIT.
+- **WavLM Base+**: originally published by the authors of *WavLM* (Chen et al., 2022) under the MIT License and redistributed by torchaudio under the same license.
+  [Source](https://github.com/microsoft/unilm/tree/65f15af2a307ebb64cfb25adf54375b002e6fe8d/wavlm#pre-trained-models) ·
+  [License](https://github.com/microsoft/unilm/blob/65f15af2a307ebb64cfb25adf54375b002e6fe8d/LICENSE)
+
+```bibtex
+@article{chen2022wavlm,
+  title   = {WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing},
+  author  = {Chen, Sanyuan and Wang, Chengyi and Chen, Zhengyang and others},
+  journal = {IEEE Journal of Selected Topics in Signal Processing},
+  year    = {2022}
+}
+```

encoder-weights.json

@@ -0,0 +1,381 @@
+{
+  "repo_id": "Aratako/MioCodec-25Hz-24kHz",
+  "torch": "2.10.0+cu128",
+  "sources": {
+    "miocodec": {
+      "file": "model.safetensors",
+      "bytes": 523087956,
+      "sha256": "60483759cde136451d53ff5a2f7e283c015758ab540de9037b5d4519366a7705"
+    },
+    "wavlm": {
+      "file": "wavlm_base_plus.pth",
+      "bytes": 377604347,
+      "sha256": "136a3e720c04f2c77bf7a4dc6a3868b14d5a2c145a988114b733cb1a8428be98"
+    }
+  },
+  "resample": {
+    "orig": 3,
+    "new": 2,
+    "width": 10
+  },
+  "tensors": {
+    "resample.kernel": [
+      2,
+      1,
+      23
+    ],
+    "wavlm.feature_extractor.conv_layers.0.conv.weight": [
+      512,
+      1,
+      10
+    ],
+    "wavlm.feature_extractor.conv_layers.0.layer_norm.weight": [
+      512
+    ],
+    "wavlm.feature_extractor.conv_layers.0.layer_norm.bias": [
+      512
+    ],
+    "wavlm.feature_extractor.conv_layers.1.conv.weight": [
+      512,
+      512,
+      3
+    ],
+    "wavlm.feature_extractor.conv_layers.2.conv.weight": [
+      512,
+      512,
+      3
+    ],
+    "wavlm.feature_extractor.conv_layers.3.conv.weight": [
+      512,
+      512,
+      3
+    ],
+    "wavlm.feature_extractor.conv_layers.4.conv.weight": [
+      512,
+      512,
+      3
+    ],
+    "wavlm.feature_extractor.conv_layers.5.conv.weight": [
+      512,
+      512,
+      2
+    ],
+    "wavlm.feature_extractor.conv_layers.6.conv.weight": [
+      512,
+      512,
+      2
+    ],
+    "wavlm.encoder.feature_projection.layer_norm.weight": [
+      512
+    ],
+    "wavlm.encoder.feature_projection.layer_norm.bias": [
+      512
+    ],
+    "wavlm.encoder.feature_projection.projection.weight": [
+      768,
+      512
+    ],
+    "wavlm.encoder.feature_projection.projection.bias": [
+      768
+    ],
+    "wavlm.encoder.transformer.pos_conv_embed.conv.weight": [
+      768,
+      48,
+      128
+    ],
+    "wavlm.encoder.transformer.pos_conv_embed.conv.bias": [
+      768
+    ],
+    "wavlm.encoder.transformer.layer_norm.weight": [
+      768
+    ],
+    "wavlm.encoder.transformer.layer_norm.bias": [
+      768
+    ],
+    "wavlm.encoder.transformer.layers.0.attention.attention.in_proj_weight": [
+      2304,
+      768
+    ],
+    "wavlm.encoder.transformer.layers.0.attention.attention.in_proj_bias": [
+      2304
+    ],
+    "wavlm.encoder.transformer.layers.0.attention.attention.out_proj.weight": [
+      768,
+      768
+    ],
+    "wavlm.encoder.transformer.layers.0.attention.attention.out_proj.bias": [
+      768
+    ],
+    "wavlm.encoder.transformer.layers.0.attention.gru_rel_pos_linear.weight": [
+      8,
+      64
+    ],
+    "wavlm.encoder.transformer.layers.0.attention.gru_rel_pos_linear.bias": [
+      8
+    ],
+    "wavlm.encoder.transformer.layers.0.attention.gru_rel_pos_const": [
+      1,
+      12,
+      1,
+      1
+    ],
+    "wavlm.encoder.transformer.layers.0.attention.rel_attn_embed.weight": [
+      320,
+      12
+    ],
+    "wavlm.encoder.transformer.layers.0.layer_norm.weight": [
+      768
+    ],
+    "wavlm.encoder.transformer.layers.0.layer_norm.bias": [
+      768
+    ],
+    "wavlm.encoder.transformer.layers.0.final_layer_norm.weight": [
+      768
+    ],
+    "wavlm.encoder.transformer.layers.0.final_layer_norm.bias": [
+      768
+    ],
+    "wavlm.encoder.transformer.layers.0.feed_forward.intermediate_dense.weight": [
+      3072,
+      768
+    ],
+    "wavlm.encoder.transformer.layers.0.feed_forward.intermediate_dense.bias": [
+      3072
+    ],
+    "wavlm.encoder.transformer.layers.0.feed_forward.output_dense.weight": [
+      768,
+      3072
+    ],
+    "wavlm.encoder.transformer.layers.0.feed_forward.output_dense.bias": [
+      768
+    ],
+    "wavlm.encoder.transformer.layers.1.attention.attention.in_proj_weight": [
+      2304,
+      768
+    ],
+    "wavlm.encoder.transformer.layers.1.attention.attention.in_proj_bias": [
+      2304
+    ],
+    "wavlm.encoder.transformer.layers.1.attention.attention.out_proj.weight": [
+      768,
+      768
+    ],
+    "wavlm.encoder.transformer.layers.1.attention.attention.out_proj.bias": [
+      768
+    ],
+    "wavlm.encoder.transformer.layers.1.attention.gru_rel_pos_linear.weight": [
+      8,
+      64
+    ],
+    "wavlm.encoder.transformer.layers.1.attention.gru_rel_pos_linear.bias": [
+      8
+    ],
+    "wavlm.encoder.transformer.layers.1.attention.gru_rel_pos_const": [
+      1,
+      12,
+      1,
+      1
+    ],
+    "wavlm.encoder.transformer.layers.1.layer_norm.weight": [
+      768
+    ],
+    "wavlm.encoder.transformer.layers.1.layer_norm.bias": [
+      768
+    ],
+    "wavlm.encoder.transformer.layers.1.final_layer_norm.weight": [
+      768
+    ],
+    "wavlm.encoder.transformer.layers.1.final_layer_norm.bias": [
+      768
+    ],
+    "wavlm.encoder.transformer.layers.1.feed_forward.intermediate_dense.weight": [
+      3072,
+      768
+    ],
+    "wavlm.encoder.transformer.layers.1.feed_forward.intermediate_dense.bias": [
+      3072
+    ],
+    "wavlm.encoder.transformer.layers.1.feed_forward.output_dense.weight": [
+      768,
+      3072
+    ],
+    "wavlm.encoder.transformer.layers.1.feed_forward.output_dense.bias": [
+      768
+    ],
+    "global_encoder.backbone.embed.weight": [
+      384,
+      768,
+      7
+    ],
+    "global_encoder.backbone.embed.bias": [
+      384
+    ],
+    "global_encoder.backbone.norm.weight": [
+      384
+    ],
+    "global_encoder.backbone.norm.bias": [
+      384
+    ],
+    "global_encoder.backbone.convnext.0.gamma": [
+      384
+    ],
+    "global_encoder.backbone.convnext.0.dwconv.weight": [
+      384,
+      1,
+      7
+    ],
+    "global_encoder.backbone.convnext.0.dwconv.bias": [
+      384
+    ],
+    "global_encoder.backbone.convnext.0.norm.weight": [
+      384
+    ],
+    "global_encoder.backbone.convnext.0.norm.bias": [
+      384
+    ],
+    "global_encoder.backbone.convnext.0.pwconv1.weight": [
+      1152,
+      384
+    ],
+    "global_encoder.backbone.convnext.0.pwconv1.bias": [
+      1152
+    ],
+    "global_encoder.backbone.convnext.0.pwconv2.weight": [
+      384,
+      1152
+    ],
+    "global_encoder.backbone.convnext.0.pwconv2.bias": [
+      384
+    ],
+    "global_encoder.backbone.convnext.1.gamma": [
+      384
+    ],
+    "global_encoder.backbone.convnext.1.dwconv.weight": [
+      384,
+      1,
+      7
+    ],
+    "global_encoder.backbone.convnext.1.dwconv.bias": [
+      384
+    ],
+    "global_encoder.backbone.convnext.1.norm.weight": [
+      384
+    ],
+    "global_encoder.backbone.convnext.1.norm.bias": [
+      384
+    ],
+    "global_encoder.backbone.convnext.1.pwconv1.weight": [
+      1152,
+      384
+    ],
+    "global_encoder.backbone.convnext.1.pwconv1.bias": [
+      1152
+    ],
+    "global_encoder.backbone.convnext.1.pwconv2.weight": [
+      384,
+      1152
+    ],
+    "global_encoder.backbone.convnext.1.pwconv2.bias": [
+      384
+    ],
+    "global_encoder.backbone.convnext.2.gamma": [
+      384
+    ],
+    "global_encoder.backbone.convnext.2.dwconv.weight": [
+      384,
+      1,
+      7
+    ],
+    "global_encoder.backbone.convnext.2.dwconv.bias": [
+      384
+    ],
+    "global_encoder.backbone.convnext.2.norm.weight": [
+      384
+    ],
+    "global_encoder.backbone.convnext.2.norm.bias": [
+      384
+    ],
+    "global_encoder.backbone.convnext.2.pwconv1.weight": [
+      1152,
+      384
+    ],
+    "global_encoder.backbone.convnext.2.pwconv1.bias": [
+      1152
+    ],
+    "global_encoder.backbone.convnext.2.pwconv2.weight": [
+      384,
+      1152
+    ],
+    "global_encoder.backbone.convnext.2.pwconv2.bias": [
+      384
+    ],
+    "global_encoder.backbone.convnext.3.gamma": [
+      384
+    ],
+    "global_encoder.backbone.convnext.3.dwconv.weight": [
+      384,
+      1,
+      7
+    ],
+    "global_encoder.backbone.convnext.3.dwconv.bias": [
+      384
+    ],
+    "global_encoder.backbone.convnext.3.norm.weight": [
+      384
+    ],
+    "global_encoder.backbone.convnext.3.norm.bias": [
+      384
+    ],
+    "global_encoder.backbone.convnext.3.pwconv1.weight": [
+      1152,
+      384
+    ],
+    "global_encoder.backbone.convnext.3.pwconv1.bias": [
+      1152
+    ],
+    "global_encoder.backbone.convnext.3.pwconv2.weight": [
+      384,
+      1152
+    ],
+    "global_encoder.backbone.convnext.3.pwconv2.bias": [
+      384
+    ],
+    "global_encoder.backbone.final_layer_norm.weight": [
+      384
+    ],
+    "global_encoder.backbone.final_layer_norm.bias": [
+      384
+    ],
+    "global_encoder.pooling.attn.0.weight": [
+      128,
+      384,
+      1
+    ],
+    "global_encoder.pooling.attn.0.bias": [
+      128
+    ],
+    "global_encoder.pooling.attn.2.weight": [
+      384,
+      128,
+      1
+    ],
+    "global_encoder.pooling.attn.2.bias": [
+      384
+    ],
+    "global_encoder.pooling.proj.weight": [
+      128,
+      768
+    ],
+    "global_encoder.pooling.proj.bias": [
+      128
+    ],
+    "global_encoder.pooling.norm.weight": [
+      128
+    ],
+    "global_encoder.pooling.norm.bias": [
+      128
+    ]
+  },
+  "bytes": 117303232,
+  "sha256": "a35b4eaec345dde0e9abb2a7aa2c8c66e7a39d4ab53afce584568b5a616a0abd"
+}

encoder-weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a35b4eaec345dde0e9abb2a7aa2c8c66e7a39d4ab53afce584568b5a616a0abd
+size 117303232