PolyWhisper: Per-Language LoRA Experts on Shared Whisper-Base

5 languages · 6 experts · 1 frozen 74M encoder · 0 encoder retraining

PolyWhisper is a family of tiny language-specific LoRA experts on a frozen Whisper-Base encoder. It tackles a problem vanilla Whisper-Base cannot even begin to solve: producing correct-script text for Indian languages.

The script-confusion finding

Vanilla Whisper-Base doesn't just perform worse on Indic languages — it emits the wrong script entirely (Urdu-Arabic for Telugu/Bengali/Marathi, 0–0.5% correct-script rate). The same encoder produces near-perfect Devanagari for Hindi but collapses to Arabic for the other Indic scripts.

Per-language LoRA experts on the shared frozen encoder fix this completely.

Architecture

Audio (16kHz) → Whisper-Base Encoder [frozen, 74M]
                         ↓
          Per-language LoRA adapter [rank-16, ~0.8M each]
                         ↓
          Transcription (correct-script, correct language)

• Backbone: openai/whisper-base — frozen encoder + frozen base decoder
• Experts: rank-16 LoRA on decoder cross-attention K/V projections
• Router (Hinglish only): 33K-param MLP on decoder hidden states, per-token en/hi selection

Results

Hinglish code-switch (v5 router, 3,129 utterances)

System	WER	FuzzyWER	CER	Hallucinations
PolyWhisper v5	58.8%	57.3%	57.9%	13
Vanilla Whisper-Base	66.6%	63.3%	67.5%	279
Static 50/50 mix	72.1%	70.4%	71.1%	655

Indic experts (FLEURS, ortho-normalized, script-matched, 256 decode tokens)

Lang	Vanilla script-match	Expert script-match	Pure expert WER/CER
Hindi	6.2%	99.8%	38.5 / 14.4
Tamil	96.4%	99.3%	73.9 / 25.6
Telugu	0.0%	92.6%	82.7 / 32.7
Bengali	0.0%	77.1%	84.9 / 54.3
Marathi	0.5%	99.8%	65.0 / 22.9

Vanilla scores for te/bn/mr are unreportable (0–0.5% correct-script output). Experts restore correct-script and cut CER 10–22 points. Residual WER is the 39M-parameter ceiling.

Files

Hinglish router

File	What
en_router_best_v5.pt	English LoRA expert (rank-8 decoder)
hi_router_best_v5.pt	Hindi LoRA expert with encoder LoRA
router_best_v5.pt	Per-token router (33K params)
hi_best_v5.pt	Hindi pure expert (encoder + decoder LoRA)

Indic experts

File	What
ta_best_ta.pt	Tamil expert (rank-16 decoder)
te_best_te_v2.pt	Telugu expert (rank-16 decoder)
bn_best_bn_v2.pt	Bengali expert (rank-16 decoder)
mr_best_mr.pt	Marathi expert (rank-16 decoder)

Evaluation

File	What
fleurs_normalized_results.json	Ortho-normalized scores for all 5 languages
eval_*_fleurs.json	Per-sample results (rescorable)
eval_*_samples.json	Per-sample Hinglish results

Usage (Indic expert example)

import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration
from peft import PeftModel

proc = WhisperProcessor.from_pretrained("openai/whisper-base")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-base")
model = PeftModel.from_pretrained(model, "eulogik/polywhisper", "ta_best_ta.pt")
model.eval()

audio = ...  # 16kHz mono
feats = proc.feature_extractor([audio], sampling_rate=16000, return_tensors="pt").input_features
out = model.generate(feats, max_new_tokens=256, language="ta", task="transcribe")
print(proc.decode(out[0], skip_special_tokens=True))

Training

• 3 epochs, LR 1e-4, rank-16 LoRA on decoder cross-attention K/V
• Data: IndicVoices-ST (~19–20K samples, gated) + FLEURS (test)
• Trained on M4 Mac Mini 16GB (Apple MPS, fp16)
• Training time: ~3h per language, fully automated (train_v3.py --langs ta)

Key finding for the paper

Language-specific LoRA experts on a shared frozen encoder fix Whisper-Base's script-confusion problem for 5 Indian languages. Vanilla outputs Urdu-Arabic for Telugu/Bengali/Marathi (0% correct-script); experts restore 77–100% correct-script output. Hindi expert reaches 38.5% WER / 14.4% CER — near-usable for edge deployment.

License

MIT. © 2026 Eulogik. Whisper is OpenAI's model. Data from IndicVoices-ST (CC-BY) and FLEURS (CC-BY 4.0).

Citation

@misc{eulogik2026polywhisper,
  title={PolyWhisper: Per-Language LoRA Experts on Shared Encoder for Indic ASR},
  author={Eulogik},
  year={2026},
  howpublished={\url{https://huggingface.co/eulogik/polywhisper}},
  note={MIT licensed; 5 Indian languages + Hinglish router; script-confusion fix}
}