update README
Browse files
README.md
CHANGED
|
@@ -1,3 +1,108 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
| 4 |
+
---
|
| 5 |
+
license: apache-2.0
|
| 6 |
+
---
|
| 7 |
+
|
| 8 |
+
# SphereVAE
|
| 9 |
+
|
| 10 |
+
SphereVAE: Hyperspherical Latent Autoencoders for Robust Autoregressive Speech Representation Modeling
|
| 11 |
+
|
| 12 |
+
[arXiv](https://arxiv.org/pdf/2609.09903v1) · [Demo](https://haoyuzhang3.github.io/SphereVAE_Demo/) · [GitHub](https://github.com/ASLP-lab/SphereVAE)
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
## Model
|
| 17 |
+
|
| 18 |
+
SphereVAE combines a SEANet audio codec with causal Transformers and a
|
| 19 |
+
Power-spherical latent space for robust speech representation modeling.
|
| 20 |
+
|
| 21 |
+
## Installation
|
| 22 |
+
|
| 23 |
+
Use Python 3.10 or newer. Install a PyTorch and torchaudio pair appropriate for
|
| 24 |
+
your machine, then install the dependencies:
|
| 25 |
+
|
| 26 |
+
```bash
|
| 27 |
+
python -m pip install -r requirements.txt
|
| 28 |
+
```
|
| 29 |
+
|
| 30 |
+
GPU training uses Accelerate. Configure it for your machine before launching:
|
| 31 |
+
|
| 32 |
+
```bash
|
| 33 |
+
accelerate config
|
| 34 |
+
```
|
| 35 |
+
|
| 36 |
+
## Training data
|
| 37 |
+
|
| 38 |
+
Place a Kaldi-style SCP file at `data/train.scp`, or update
|
| 39 |
+
`data.train_scp_path` in `configs/config_Sphere_VAE.yaml`:
|
| 40 |
+
|
| 41 |
+
```text
|
| 42 |
+
utt_0001 /path/to/audio_0001.flac
|
| 43 |
+
utt_0002 /path/to/audio_0002.wav
|
| 44 |
+
```
|
| 45 |
+
|
| 46 |
+
Each line contains an utterance ID and an audio path. Audio is processed as
|
| 47 |
+
mono 24 kHz input and prepared as training segments by the dataset pipeline.
|
| 48 |
+
|
| 49 |
+
## Training
|
| 50 |
+
|
| 51 |
+
```bash
|
| 52 |
+
CUDA_VISIBLE_DEVICES=0 accelerate launch train.py -c configs/config_Sphere_VAE.yaml
|
| 53 |
+
```
|
| 54 |
+
|
| 55 |
+
Adjust training settings in `configs/config_Sphere_VAE.yaml`. Checkpoints and
|
| 56 |
+
TensorBoard logs are written to the configured experiment directory. For
|
| 57 |
+
multi-GPU training, configure Accelerate and set the visible devices as needed.
|
| 58 |
+
|
| 59 |
+
## Inference
|
| 60 |
+
|
| 61 |
+
```bash
|
| 62 |
+
python infer.py \
|
| 63 |
+
--checkpoint exps/Sphere_VAE/G_400000.pth \
|
| 64 |
+
--input data/test \
|
| 65 |
+
--output output/reconstructed \
|
| 66 |
+
--device cuda:0
|
| 67 |
+
```
|
| 68 |
+
|
| 69 |
+
`--input` accepts an audio file or directory. Supported extensions are WAV,
|
| 70 |
+
FLAC, MP3, OGG, and M4A. Use `--config` to select another configuration.
|
| 71 |
+
Inference uses posterior sampling, so repeated runs can produce different
|
| 72 |
+
reconstructions. Output is trimmed to the input length.
|
| 73 |
+
|
| 74 |
+
## Layout
|
| 75 |
+
|
| 76 |
+
```text
|
| 77 |
+
models/model_Sphere_VAE.py Sphere_VAE model and builder
|
| 78 |
+
modules/ SEANet, Transformer, spherical distribution, utilities
|
| 79 |
+
train.py Training entrypoint
|
| 80 |
+
infer.py Audio reconstruction entrypoint
|
| 81 |
+
dataset.py SCP-listed audio dataset
|
| 82 |
+
configs/ Sphere_VAE configuration
|
| 83 |
+
discriminators/ STFT discriminator
|
| 84 |
+
losses/ Spectral and adversarial losses
|
| 85 |
+
utils/ Configuration, checkpoints, and compilation utilities
|
| 86 |
+
```
|
| 87 |
+
|
| 88 |
+
## Citation
|
| 89 |
+
|
| 90 |
+
```bibtex
|
| 91 |
+
@misc{zhang2026spherevaehypersphericallatentautoencoders,
|
| 92 |
+
title={SphereVAE: Hyperspherical Latent Autoencoders for Robust Autoregressive Speech Representation Modeling},
|
| 93 |
+
author={Haoyu Zhang and Jingbin Hu and Hanke Xie and Qirui Zhan and Wenhao Li and Ziyu Zhang and Xiaming Ren and Yue Li and Xunyu Zhu and Zhipeng Chen and Lei Xie},
|
| 94 |
+
year={2026},
|
| 95 |
+
eprint={2609.09903},
|
| 96 |
+
archivePrefix={arXiv},
|
| 97 |
+
primaryClass={eess.AS},
|
| 98 |
+
url={https://arxiv.org/abs/2609.09903}
|
| 99 |
+
}
|
| 100 |
+
```
|
| 101 |
+
|
| 102 |
+
## Acknowledgments
|
| 103 |
+
|
| 104 |
+
Parts of the code are adapted from Kyutai Mimi and Meta AudioCraft/EnCodec,
|
| 105 |
+
as indicated by the original notices retained in the source files. This
|
| 106 |
+
export does not assign a new license to that third-party code. The source
|
| 107 |
+
checkout did not include the root license files referenced by those notices;
|
| 108 |
+
applicable upstream license texts still need to be supplied before release.
|