Persona-ASR

Bilingual (Kazakh–English) target-speaker ASR for overlapping speech. Given a multi-talker mixture and a short enrollment utterance, the system transcribes only the enrolled speaker, and emits <no_target> instead of a transcript when that speaker is absent.

• Paper: R. Meiramov, T. Rakhimzhanova, A. Taibassarov, Z. Makhataeva, H. A. Varol. Persona-ASR: Bilingual Target-Speaker Speech Recognition for Kazakh–English Overlapping Speech. Machine Learning and Knowledge Extraction, 8(8), 246, 2026. doi:10.3390/make8080246
• Code: github.com/IS2AI/Persona_ASR
• Data: KazMix-3 · PersonaMix

Files

File	Description
asr_backbone.pt	ASR backbone: a frozen ECAPA-TDNN speaker embedding modulates a WavLM-Base-Plus encoder through FiLM, feeding language-specific CTC heads (English, Kazakh) and a frame-level VAD head. 118.67 M parameters.
presence_gate.pt	Target-presence gate used in the paper: enrollment–mixture frame matching with attentive-statistics pooling, trained on the frozen backbone. 197,891 parameters. The validation-calibrated decision thresholds are stored in the checkpoint (τ_EN = 0.497, τ_KK = 0.609).
presence_gate_dualpath_ablation.pt	Earlier two-path gate (matching + speaker-conditioned attention, 567,813 parameters), kept for the gate-design comparison in Section 5.4. It is not the published model.
config.json	Backbone configuration.
manifests/libri3mix/	The English target-speaker manifests (train / validation / test) used in the paper, built on official Libri3Mix. Needed to reproduce the English results exactly.

Correction (September 2026). Until this update, presence_gate.pt in this repository was the two-path gate (now presence_gate_dualpath_ablation.pt), which does not correspond to the published results. If you downloaded the gate before September 2026, please download it again.

Results

Three-speaker test sets (Table 3 of the paper). Raw and gated WER are computed on target-present samples; detection metrics on target-present and target-absent samples.

Test set	Raw WER (%) ↓	Gated WER (%) ↓	BAcc (%) ↑	F1 (%) ↑
English (Libri3Mix-100h)	29.41	35.45	81.33	86.92
Kazakh (KazMix3-100h)	43.49	50.77	86.28	88.78
Overall	36.06	42.68	83.81	87.83

Training

The backbone is trained from scratch on target-present Libri3Mix (English) and KazMix-3 (Kazakh) mixtures with L = 0.7·CTC + 0.3·VAD for 100 epochs. The gate is trained on the frozen backbone with binary cross-entropy and a ranking loss for 40 epochs, on target-present and target-absent trials. Exact commands are in the GitHub repository.

Citation

@article{meiramov2026personaasr,
  author  = {Meiramov, Rakhat and Rakhimzhanova, Tomiris and Taibassarov, Adil and Makhataeva, Zhanat and Varol, Huseyin Atakan},
  title   = {Persona-ASR: Bilingual Target-Speaker Speech Recognition for Kazakh--English Overlapping Speech},
  journal = {Machine Learning and Knowledge Extraction},
  year    = {2026},
  volume  = {8},
  number  = {8},
  pages   = {246},
  doi     = {10.3390/make8080246}
}

Funding

This research is funded by the Committee of Science of the Ministry of Science and Higher Education of the Republic of Kazakhstan (Grant No. BR24993001).

License

CC BY 4.0.