Acknowledge license to accept the repository
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
By submitting this form, you agree to the License Agreement and acknowledge that the information you provide will be collected, used, and shared in accordance with Cohere's Privacy Policy. You'll receive email updates about Cohere Labs and Cohere research, events, products and services. You can unsubscribe at any time.
Log in or Sign Up to review the conditions and access this model content.
Model Card for Tiny Aya Base 32K
Model Summary
Cohere Labs Tiny Aya Base 32K is an open-weights research release of a pretrained 3.35 billion parameter model optimized for efficient, strong, and balanced multilingual representation across 70+ languages, including many lower-resourced ones. This variant provides a 32K context window and is designed to support downstream adaptation, instruction tuning and long-context research.
This is a base pretrained model and has not been instruction-tuned or preference-aligned.
This checkpoint is used as the base model for Tiny Aya L2-Thinker and Tiny Aya En-Thinker.
Developed by: Cohere and Cohere Labs
- Point of Contact: Cohere Labs
- License: CC-BY-NC, requires also adhering to Cohere Lab's Acceptable Use Policy
- Model: tiny-aya-base-32K
- Model Size: 3.35B
- Context Length: 32K (input + output)
For the 8K release, see tiny-aya-base. For the broader family, see tiny-aya-global and the Tiny Aya collection.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "CohereLabs/tiny-aya-base-32K"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
torch_dtype="auto",
)
prompt = "The capital of Spain is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=128,
do_sample=True,
temperature=0.1,
top_p=0.9,
top_k=50,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
The model can also be used with the Transformers pipeline abstraction:
from transformers import pipeline
generator = pipeline(
"text-generation",
model="CohereLabs/tiny-aya-base-32K",
torch_dtype="auto",
device_map="auto",
)
output = generator(
"The capital of Spain is",
max_new_tokens=128,
do_sample=True,
temperature=0.1,
top_p=0.9,
repetition_penalty=1.1,
)
print(output[0]["generated_text"])
The combined prompt and generated sequence must fit within the 32K context window.
Model Details
Input: Text only.
Output: Model generates text.
Model Architecture: Auto-regressive transformer in the Tiny Aya / Cohere family, using interleaved sliding-window and global-attention layers with rotary positional embeddings.
Languages covered: The model was trained on 70+ languages, with a focus on English, Dutch, French, Italian, Portuguese, Romanian, Spanish, Czech, Polish, Ukrainian, Russian, Greek, German, Danish, Swedish, Norwegian, Catalan, Galician, Welsh, Irish, Basque, Croatian, Latvian, Lithuanian, Slovak, Slovenian, Estonian, Finnish, Hungarian, Serbian, Bulgarian, Arabic, Persian, Urdu, Turkish, Maltese, Hebrew, Hindi, Marathi, Bengali, Gujarati, Punjabi, Tamil, Telugu, Nepali, Tagalog, Malay, Indonesian, Vietnamese, Javanese, Khmer, Thai, Lao, Chinese, Burmese, Japanese, Korean, Amharic, Hausa, Igbo, Malagasy, Shona, Swahili, Wolof, Xhosa, Yoruba, and Zulu.
Context Length: Tiny Aya Base 32K supports a context length of 32K tokens across input and output.
Usage and Limitations
Intended Usage
Tiny Aya Base 32K is intended for continued pretraining, instruction tuning, and research on multilingual and long-context language modeling. Potential downstream applications include multilingual text generation, summarization, translation, cross-lingual tasks, and adaptation for lower-resource languages.
Because this is a base model, prompts should use text completion rather than a chat template. Additional post-training is recommended before deploying it as a conversational assistant.
Strengths
Tiny Aya provides strong open-ended generation across broad language coverage, including many lower-resource languages. Shared training signals across language families and scripts support multilingual generation, translation, summarization, and cross-lingual adaptation. This release supports substantially longer sequences than the 8K base release.
Limitations
Base-model behavior. This checkpoint is not instruction-tuned or preference-aligned and may not reliably follow instructions or behave like a conversational assistant.
Reasoning tasks. The model's strongest performance is on open-ended generation. Chain-of-thought reasoning tasks such as multilingual math are comparatively weaker.
Factual knowledge. Outputs may contain incorrect or outdated statements, particularly in lower-resource languages with thinner training data coverage.
Uneven resource distribution. High-resource languages benefit from richer training signals and tend to show more consistent quality. The lowest-resource languages may show greater variability, and culturally specific nuance, sarcasm, or figurative language may be less reliable.
Long-context use. A 32K context window does not guarantee uniform recall or reasoning quality at every position. Long inputs also require substantially more memory and compute.
Model Card Contact
For errors or additional questions about details in this model card, contact labs@cohere.com.
Terms of Use
This model is governed by a CC-BY-NC License (Non-Commercial) and also requires adhering to Cohere Lab's Acceptable Use Policy. If you are interested in commercial use, please contact Cohere’s Sales team.
- Downloads last month
- -