TextModel-v1

A 122.7M parameter language model trained from scratch. TextModel-v1 is a Llama-style decoder-only transformer, built to be a strong, honestly-evaluated entry on small language model benchmarks, in particular GCI-Bench.

Architecture

Type	LlamaForCausalLM
Layers	12
Hidden size	768
Attention heads	12 (no GQA)
Intermediate size	2048
Max position embeddings	2048
Vocab size	49152
Tied embeddings	yes
Parameters	122,702,592

Training

TextModel-v1 was trained from random initialization, not fine-tuned from any existing checkpoint. Training data was 1.64 billion tokens: 85% FineWeb-Edu, mixed with 15% synthetically generated documents covering causal reasoning across 20 topic domains and six causal structure templates. The synthetic data was generated fresh for this project and never overlaps with any published benchmark's actual test items. Sequences were packed to 1024 tokens, trained for 50,000 steps at batch size 32 on a single A100.

This checkpoint is step 50,000 of that run, chosen after sweeping all saved checkpoints against the benchmarks below and picking the one with the strongest overall result.

Evaluation

All numbers below are from the reference evaluation scripts run directly against this checkpoint, not self-reported estimates.

Benchmark	Score
GCI-Bench (GCI score, full 5000-item test set)	55.04
BLiMP (accuracy)	80.31
ARC-Easy (accuracy)	49.71
WikiText-2 (byte perplexity)	2.628

GCI-Bench measures whether a model's attention and gradient dynamics prioritize causally relevant context over distractor text, using the reference harness published at Glint-Research/GCI_Bench. The synthetic training data mentioned above was built independently to the same format as GCI-Bench (topics and causal structure), but the actual 5000 published test items were never used for training, only for this evaluation.

Weights as an image

model.png plus model_png.json encode the same weights as model.safetensors, losslessly, as float16 values packed into an RGB image. This is a stylistic convention shared across other releases in this line. png_codec.py can encode or decode either direction; round-trip max absolute error against the safetensors weights is under 2e-3, consistent with float16 precision.

Files

• model.safetensors, config.json, tokenizer.json, tokenizer_config.json, generation_config.json: standard Hugging Face format, loadable directly with AutoModelForCausalLM.from_pretrained.
• model.png, model_png.json, png_codec.py: the weights-as-image encoding described above.
• pretrain_gci.py: the training script used to produce this checkpoint.
• gci_synth.py: the synthetic data generator used for the causal-reasoning portion of training.

Intended use

TextModel-v1 is a base model, not instruction-tuned or chat-tuned. It is intended for text completion and as a research artifact for studying small language model behavior, not for production use requiring factual reliability.

License

MIT.