notmax123/Fish-Audio-S2-Pro-He

Fish Audio S2-Pro — Hebrew (LoRA + atomic IPA tokens)

A Hebrew adapter for fishaudio/s2-pro. It keeps the base model's multilingual ability and voice cloning intact, and adds native Hebrew synthesis driven by IPA rather than nikud.

This repo contains only the adapter (~67M parameters) plus the extended tokenizer. You still need the S2-Pro base weights and codec.

What's here

File	What it is
hebrew_lora_step2200.safetensors	LoRA deltas + the trained ipa_embeddings table (67M params, bf16)
hebrew_lora_step2200.ckpt	Same weights as a Lightning checkpoint, with optimizer state — use this to resume training
config.json	S2-Pro config extended with num_ipa_tokens: 26, ipa_token_start: 155774
ipa_token_map.json	IPA symbol → atomic token (e.g. ʃ → <ipa_u0283>)
ipa_embeddings.pt	Initial IPA embedding table (mean of the symbol's BPE pieces); the trained one lives in the adapter
tokenizer/	S2-Pro tokenizer extended 155,774 → 155,800 tokens
samples/	Generated audio (see below)

Quick start

Code lives in the fork the adapter was trained with:

git clone https://github.com/maxmelichov/fish-speech
cd fish-speech && uv sync --python 3.12 --extra cu129
pip install renikud-plus                 # Hebrew grapheme-to-phoneme

bash tools/hebrew/setup_hebrew.sh        # base weights + this adapter + IPA checkpoint

python tools/hebrew/infer_hebrew.py \
    --text "שלום, מה שלומך היום?" \
    --lora-checkpoint checkpoints/hebrew/hebrew_lora_step2200.safetensors \
    --output out.wav

Add --ref-audio my_voice.wav --ref-text "..." to clone a voice.

infer_hebrew.py runs plain unvocalized Hebrew through RenikudPlus G2P, maps the IPA to the atomic tokens, and chunks long inputs on sentence boundaries. --lora-scale scales the delta (0.0 = pure base model) if you want to dial the adaptation down.

Fine-tuning on your own Hebrew data is one command — a directory per speaker of *.wav plus sibling .lab transcripts:

AUDIO_ROOT=my_audio tools/hebrew/run_hebrew_pipeline.sh

See tools/hebrew/README.md for the full guide.

How it works

Atomic IPA tokens. S2-Pro's BPE splits IPA into pieces that collide with English orthography — Hebrew י phonemized as j was read as the English letter jay. So each of the 26 Hebrew IPA symbols gets a dedicated input-only token (<ipa_j>, <ipa_u0283>, …) in a separate trainable nn.Embedding, initialized to the mean of the symbol's original BPE pieces. The output vocabulary is untouched — these tokens are never predicted, only read.

What trains. LoRA r=32, α=16 on attention + mlp, plus the IPA embedding table — 66.9M parameters total: 60.2M in the slow transformer, 6.7M in the fast transformer, 0.03M IPA embeddings. Frozen are the direct interfaces to codebook space — fast_embeddings, fast_output, and the tied slow embeddings/output — which is what keeps timbre close to the base model. Note α/r = 0.5, not the usual 2.0; see Caveats. The residual-codebook loss is down-weighted to 0.3 (Qwen3-TTS's sub-talker coefficient) so the gradient stays on the text→semantic mapping.

Training. 279,476 Hebrew utterances (~10 speakers, WER ≤ 0.1), reference- conditioned on a same-speaker utterance 80% of the time so training prompts match the exact generate_long() inference format. bf16, lr 5e-5 constant with 100-step warmup, effective batch 12, 2200 optimizer steps.

Upstream bug fixed along the way

S2-Pro sets scale_codebook_embeddings=True. At inference, forward_generate() divides semantic-position embeddings by sqrt(num_codebooks + 1) = 3.317; the training path in embed() did not. Every fine-tune therefore learned against embeddings 3.3× larger than the ones it would see at generation time. Teacher- forced CE looked fine while free-running generation collapsed after the first word — the classic symptom in fishaudio/fish-speech issues #1136 (Japanese gibberish), #682 (Hindi noise) and #814.

Five Hebrew runs collapsed the same way before this was found. After the fix (train and inference embeddings verified bit-identical):

	sample RMS	energy decay over the utterance
before	0.008 – 0.022	0.07×
after	0.171 – 0.205	1.02×
base model reference	0.181	—

The fix is in fish_speech/models/text2semantic/llama.py in the fork above and applies to any S2-Pro fine-tune, Hebrew or not.

Samples

samples/ contains, all generated with this adapter:

• 01_podcast_2hosts_63s.wav — 63s two-host Hebrew conversation, cloned voices
• 03_longform_15s.wav — multi-sentence long-form
• 04_yod_BASE.wav / 05_yod_LORA.wav — the י → English jay failure, before and after atomic IPA tokens
• 07_clone_LORA_ranlevi.wav — voice clone from a real Hebrew speaker reference

Caveats — this checkpoint is early, not final

• Undertrained, and stopped by hand. 2,200 optimizer steps ≈ 53k utterances seen, under 20% of one epoch over the 279k-row set. Train loss was still falling (3.66 → 2.70 base CE) and val loss was still improving monotonically at every checkpoint (2.934 → 2.844 → 2.820 → 2.807). Nothing had plateaued; the run was simply halted.
• α/r = 0.5 is a workaround for a bug that no longer exists. The unusual scaling was chosen empirically because at α=64 the delta destroyed free-running generation — which we now know was the embedding-scale bug above, not the LoRA strength. That rationale is obsolete post-fix, and the standard α = 2r was never re-tried. It may well be better.
• The pitch homogenization noted below is the symptom you would expect from putting LoRA on the fast transformer at all. Freezing fast_layers entirely is the obvious next experiment.

Known limitations

• Emotion tags ([whisper], [excited], …) do not work — and this is not a regression from the LoRA. Measured on the base model in English: plain / whisper / shouting produced RMS 0.0655 / 0.0652 / 0.0689, i.e. no response at all. The released S2-Pro weights simply lack the tag alignment.
• Pitch is not cloned. Timbre transfers well (4/4 by ear), but neither base nor LoRA reproduces the reference's F0 (base mean |err| 23 Hz, LoRA 20 Hz). The LoRA homogenizes pitch somewhat: spread across speakers drops from 66 Hz to 26 Hz.
• Trained on read/broadcast-style Hebrew; conversational and heavily accented speech are out of distribution.
• Hebrew input must go through G2P. Feeding nikud or bare Hebrew script directly to the model is out of distribution — use infer_hebrew.py, which handles it.

License

Inherits the base model's license (CC BY-NC-SA 4.0). Non-commercial. Do not use it to clone a voice you do not have permission to clone.