openmodelai/Galactic-2.5-Mini

Galactic 2.5 Mini

SpaceModel · ~50M parameters · Experimental RU/EN checkpoint

Not a usable chat assistant. These are genuine weights after only 300 optimizer steps from random initialization on a tiny Russian/English corpus. Test generations are largely incoherent, contain repeated newlines and mix languages. Russian and English describe the training data and tokenizer, not demonstrated proficiency. Do not advertise this checkpoint as a fully trained bilingual model.

Overview

Field	Value
Name	Galactic 2.5 Mini
Developer / brand	SpaceModel (user-specified designation)
Unique trainable parameters	49,999,872
Architecture	GPT-2-compatible decoder-only Transformer
Layers / hidden size / heads	12 / 512 / 8
Feed-forward size	2,048
Vocabulary	23,258, newly trained byte-level BPE
Position embeddings	Learned absolute, 512 positions
Training sequence length	128; longer contexts were not trained
Embeddings	Input/output weights tied
Initialization	Random; no pretrained checkpoint used
Training / export precision	FP32 / FP16 safetensors
Training task	Causal next-token prediction
Chat template / instruction tuning	None
Release status	Short-run technical proof of training only

SpaceModel is the requested developer label for this project; corporate identity and ownership of a Hugging Face namespace were not verified. The code and local CPU training run were produced with AI assistance. “2.5” is a requested product name, not an assertion about prior releases.

Actual training run

• Hardware available: 2 CPU cores, approximately 2 GiB system RAM; no GPU.
• Seed: 42.
• Optimizer: AdamW, peak LR 5e-4, betas (0.9, 0.95), weight decay 0.1, gradient norm clipping 1.0.
• LR schedule: 20-step warmup multiplied by cosine decay to approximately 10% of peak.
• Steps: 300. Batch size: 2. Sequence length: 128.
• 76,800 input-token presentations, corresponding to 76,200 shifted prediction targets. These are not unique-token counts.
• One RU window and one EN window in every batch; random sampling with replacement.
• Training-loop time: 250.69 seconds, excluding data preparation, initial/final validation, export and verification.
• PyTorch 2.5.1+cpu; Transformers 4.46.3.
• No optimizer state or RNG state is shipped; restarting from the weights uses a fresh optimizer.

Small held-out loss check (not a benchmark)

The final 5% by characters of each work, split at a newline, was held out before tokenizer training. Eight fixed random batches of two 128-token windows per language were evaluated before and after training. Overlap between validation windows is possible. Metrics below are FP32 before FP16 export.

Language	Initial cross-entropy (nats/token)	Final cross-entropy (nats/token)
Russian	10.1326	7.2389
English	10.1002	6.2290

A falling loss shows that optimization ran; it does not demonstrate useful generation, reasoning, instruction following, language fluency or safety. No external benchmarks were run. Full logs are in training_log.jsonl and the run report is in training_report.json.

Data and provenance

Five Project Gutenberg UTF-8 texts:

Language	Work	Author / editor	Source
RU	Духовные оды	Gavriil Derzhavin	https://www.gutenberg.org/ebooks/14741
RU	1001 задача для умственного счета	Sergei Rachinskii	https://www.gutenberg.org/ebooks/16527
RU	Московия в представлении иностранцев XVI-XVII в.	Pavel Apostol; editor Georgii Lukomskii	https://www.gutenberg.org/ebooks/30774
EN	Alice’s Adventures in Wonderland	Lewis Carroll	https://www.gutenberg.org/ebooks/11
EN	Pride and Prejudice	Jane Austen	https://www.gutenberg.org/ebooks/1342

Prepared token pools: RU training 82,978, RU validation 5,546; EN training 199,642, EN validation 10,912. The run sampled only a small fraction of the available windows, with replacement. The raw corpus is not balanced in size; training batches are balanced by language.

Only the exterior Gutenberg header/license footer was stripped. Editorial material and contents may remain. Russian material includes historical orthography and very different domains from modern dialogue. This is not a modern, curated bilingual pretraining corpus. Texts are listed by Gutenberg as public domain in the USA; check applicable local copyright and source terms. Raw texts are not bundled. Raw-file SHA-256 hashes and download URLs are recorded in data_sources.json.

Load with Transformers

Install torch, transformers==4.46.3, tokenizers==0.20.3, and safetensors.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# Local folder containing config.json and model.safetensors,
# or replace this with YOUR_NAMESPACE/Galactic-2.5-Mini after upload.
model_id = "./model"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.float32, trust_remote_code=False
).eval()

inputs = tokenizer("Однажды вечером", return_tensors="pt")
with torch.inference_mode():
    output = model.generate(
        **inputs, max_new_tokens=48, do_sample=True,
        temperature=0.8, top_p=0.9, repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )
print(tokenizer.decode(output[0], skip_special_tokens=True))

This code demonstrates loading and generation, not output quality. No custom remote code is required. For shorter, actually seen positional ranges, keep prompt plus generated text within 128 tokens; the configured hard limit is 512.

Verification

The exported model.safetensors was loaded offline through AutoModelForCausalLM and tested on one Russian and one English prompt. Parameter count, finite FP16 weights, finite FP32 logits, tokenizer round trips and generation execution passed. The resulting texts were incoherent. Exact unedited samples, file size and SHA-256 are in verification.json.

• Weight file: 100,014,488 bytes.
• Weight SHA-256: c5203f2624e8f39c502f643698ca8691d6a156a5ca4c7a30ad8e7be738795391.

Training code

training/ contains the data-preparation, training, inference and upload scripts, requirements and a Russian usage guide. The original downloadable project uses model/ as its export directory; the published Hub repository places the contents of that directory at the root.

From a clone of the Hub repository, to reproduce into a separate output directory:

pip install -r training/requirements.txt
cd training
python prepare_data.py
python train.py --steps 300 --output reproduced

To initialize further optimization from the Hub checkpoint after preparing the same tokenizer-compatible data:

# Inside training/; parent directory is the published model.
python train.py --resume .. --output continued --steps 1000

Review the guide before changing data. Never retrain the tokenizer on different texts and pair the new token IDs with these existing weights. For substantial training, expand and audit the corpus, add checkpointing, use appropriate GPU resources, and perform independent evaluations. Repeating this tiny corpus is not a substitute for pretraining.

Limitations and intended use

For educational inspection, pipeline testing and experimentation with further pretraining only. Not suitable for deployed assistants, factual answers, medical/legal advice, translation, reasoning, arithmetic, moderation or safety-critical decisions. The checkpoint can emit biased, offensive or nonsensical content; it has no safety alignment. Historical source texts may contain obsolete or objectionable views. Training-data memorization has not been evaluated.

License

The original project code, tokenizer, configuration and model weights are offered under MIT; see LICENSE. Source works and third-party software retain their own legal status and terms. No claim of rights over third-party source texts is made.

Кратко по-русски

Galactic 2.5 Mini, SpaceModel, 49 999 872 параметра. Веса действительно обучались с нуля, но всего 300 шагов на CPU. Это сильно недообученный экспериментальный checkpoint, а не рабочая русско-английская чат-модель. Проверенные примеры генерации бессвязны. Для полезного качества нужны существенно больше данных и вычислений. Не скрывайте этот статус при публикации.