TinyAya — Turkish ⇄ Hindi Speech-to-Speech Translation

Can a text-pretrained multilingual model be taught to translate speech to speech — for a language pair with almost no parallel speech data?

That is what this organization holds: a LoRA-adapted Cohere2 backbone driving a frozen Moshi depth decoder over Mimi audio codes, with a text "inner-monologue" running alongside the audio stream. Everything is here — the text corpus, the synthetic speech, the encoded tokens, the model, and every training checkpoint.

What v0.3 actually achieved

The full-corpus run covered 76,250 steps / 2.07 epochs on a Cloud TPU v6e-16, reaching a best validation composite of 2.8199 at step 76,000. Read honestly, capability arrived in a clear order — language identity → text translation → audio synthesis — and the run bought the first two:

The model learns the translation mapping; intelligible audio synthesis is the frontier. The gap between 3.7/9.6 and the 92/87 ceiling is not a broken pipeline — that ceiling was measured through the same codec and ASR — it is the frozen depth decoder. BLASER-2.0 sees more meaning in the generated speech than a transcript recovers, which says the audio carries real signal that is simply not yet intelligible.

We publish the number next to its ceiling every time. A low score with no ceiling beside it is unreadable.

Start here

How the data becomes the model

tr-hi-parallel-text          text triples (en pivot -> tr / hi)
        |  TTS
tr-hi-parallel-speech-v2     synthetic speech + round-trip ASR quality signals
        |  Mimi encode
tr-hi-mimi-encoded           8 codebooks @ 12.5 Hz + word-level alignments
        |  Stage-2 training
tr-hi-s2st-v0.3              the released model

Also in this organization

• Model history — v0.1 and v0.2 are kept as honest history, including the disclosed mistakes: v0.2 trained on the wrong dataset via a launcher default, and both trained audio-only because a filename check wrongly concluded the corpus had no text alignments. It ships 840,426 of them.
• Evaluation sets — fleurs-tr-hi-mimi-encoded (real human speech; acoustic shift only — its texts overlap training and it must never be called held-out text), lahaja-eval (Hindi, dialect-labelled), cv-tr-eval (Turkish).
• Codec research — dualcodec-hindi-25hz and dualcodec-turkish-25hz: can a neural codec fine-tuned on a low-resource language beat the stock multilingual one? Directly relevant, since codec quality is what caps v0.3.

Caveats worth knowing before you use this

Training speech is entirely synthetic TTS, and the model is measurably distribution-bound — on real human speech the text score collapses from ~25 to ~8 chrF++. Translation references are machine-translated, so they carry their own ceiling. This is a research artifact for low-resource S2ST, not a production translator.

Licensing

Released weights are CC-BY-NC-4.0 — non-commercial, inherited from CohereLabs/tiny-aya-base. The training and evaluation code is Apache-2.0. Moshi/Mimi components are CC-BY-4.0. Corpus text derives from FLORES (CC BY-SA 4.0 — share-alike propagates), OPUS-100 and TTS output; check each dataset card before redistributing.

Compute for the v0.3 run was provided by Google's TPU Research Cloud (TRC).