TATlit — Tatar (Cyrillic) language model, 478M, trained from scratch
A 478M-parameter base language model for Tatar (tt), trained from scratch. Not instruction-tuned.
Model details
- Architecture: decoder-only transformer (Llama-style: RoPE, SwiGLU, RMSNorm, grouped-query attention).
- Parameters: ~478M.
- Context length: 4096 tokens.
- Tokenizer: 48K SentencePiece-Unigram, shared across the Cyrillic-script Kypchak language pool (Tatar fertility ≈ 4.2–4.5 characters/token).
- Training: two stages. (1) Broad Kypchak pretraining with target-capped sampling — no language sampled more often than Tatar. (2) Tatar-centric continual pretraining, register-weighted toward literary text, with an anti-memorization regime (goldfish loss, dropout, weight decay).
Intended use
Research base model for Tatar text: language modeling and morphological / grammaticality evaluation. It is a base model with no chat or instruction tuning and is not intended for knowledge or reasoning tasks.
Evaluation
Byte-level bits-per-byte (BPB, lower is better) and accuracy (higher is better). The held-out set is literary and periodical Tatar excluded from training.
| Metric | Set | Score |
|---|---|---|
| Byte-BPB ↓ | Held-out literary | 0.732 (best) |
| Byte-BPB ↓ | Held-out periodical | 0.701 (best) |
| Byte-BPB ↓ | FLORES-200 tat_Cyrl |
0.740 (best) |
| Byte-BPB ↓ | BOUQuET (Tatar) | 0.777 (best) |
| Byte-BPB ↓ | UD Tatar-NMCTT | 0.502 |
| Accuracy ↑ | TatBLiMP (minimal pairs) | 0.972 (best) |
| Acc_norm ↑ | TUMLU-mini (Tatar) | 0.255 |
(best) = best score among the Tatar language models benchmarked here (this model, tweety-7b-tatar-v24a, goldfish-125M, mGPT-1.3B).
Additional checks: verbatim memorization on seen text 0.033; no code-switching into sibling languages in generation (0 / 180 samples).
Limitations
Base model, Tatar-focused. Knowledge and reasoning performance is near chance by design (see TUMLU-mini). Output may be factually incorrect. No safety or instruction tuning.
Training data
Register-cleaned Tatar text with related Cyrillic-script Kypchak languages (Bashkir, Kazakh, Kyrgyz, and a thin transfer layer of Karakalpak, Kumyk, Karachay-Balkar, Crimean Tatar, Nogai). Deduplicated; held-out evaluation sets removed from training.
Authors
Ilshat Saetov, Dmitry Gaynullin
© 2026 Ilshat Saetov, Dmitry Gaynullin. Licensed under CC BY-NC-SA 4.0.