Malama Base

Malama Base is a 0.96B-parameter hybrid language model trained from scratch on 100B tokens of web text, code and math. It is the pretrained base for the Malama models: it has had no midtraining, instruction tuning or reinforcement learning, and it continues text rather than following instructions.

The goal of the Malama line is a small model that is competitive for its size and training compute in one narrow task, code autocomplete. This base model was trained with about 5.8e20 FLOPs (6 Γ— parameters Γ— tokens).

Model

Architecture Decoder-only hybrid: Kimi Delta Attention (KDA) linear-attention layers with full attention every fourth layer
Layers 24 (pattern KDA, KDA, KDA, full attention; 18 KDA and 6 full-attention layers)
Width 1,536 model dimension, 4,096 MLP hidden
Heads 12 KDA heads; 12 query and 4 key/value heads (grouped-query attention) in full-attention layers; head dimension 128
Position encoding RoPE, base 1,000,000 (full-attention layers)
Context 4,096 tokens
Vocabulary 154,880 (GLM tokenizer, 154,856 tokens used; EOS id 154820)
Parameters 959.8M during training, including a one-block multi-token-prediction (MTP) head; about 928M in the released weights, which drop the MTP head
Weights bfloat16 safetensors

Training

Tokens 99,999,547,392 (95,367 steps of 1,048,576 tokens: 256 sequences of 4,096 tokens)
Optimizer Muon (lr 0.02, momentum 0.95) for matrices; AdamW (lr 3e-4, betas 0.9/0.95) for the rest; weight decay 0.1; gradient clip 1.0
Schedule 0.5B-token warmup, constant learning rate to 85B tokens, then decay over the final 15%
Auxiliary losses Next-token prediction plus a multi-token-prediction head (loss on a sampled 25% of positions, weight 0.3 falling to 0.1); z-loss 1e-4
Hardware 8 Γ— NVIDIA H100 80GB on Modal, PyTorch 2.14.1 (CUDA 13.0), bf16, torch.compile
Throughput Median 380.5k tokens per second (2.76 s per step), median model FLOPs utilization 29.9%
Training time 75.3 hours of training steps (602 H100-hours), 2026-11-10 to 2026-11-14
Seed 1234 (training), 20261025 (data recipe)

Data

Recipe kda-foundation-100b-504010-v3: 50% general web text, 40% code and 10% math. Code from popular and curated repositories is sampled more often and repeated.

Source Share Max passes Dataset
General web 50.0% 1 karpathy/climbmix-400b-shuffle
Math 10.0% 1 nvidia/Nemotron-CC-Math-v1
Starred code (non-Python) 20.0% 3 codeparrot/github-code-clean, repositories with at least 10 watchers
Other code (non-Python) 8.3% 1 codeparrot/github-code-clean
Starred Python 4.0% 5 codeparrot/github-code-clean, repositories with at least 10 watchers
Other Python 3.5% 1 codeparrot/github-code-clean
Curated open-source projects 4.0% 5 jfrick/curated-oss-code-20261019
MIT-licensed collection 0.2% 5 jfrick/mit-claude-codex-code-20261019

Repository stars come from ibragim-bad/github-repos-metadata-40M (watchers count). Code is filtered by language, file size and path to drop markup, generated and vendored files. Training documents sharing a 13-gram with the test sets of GSM8K, HumanEval, MBPP, MMLU, ARC, HellaSwag, MATH and LiveCodeBench are removed. Each source holds out 2,000 documents from its last shard for evaluation.

Results

The training loss fell from 2.07 at the start of the learning-rate decay to 1.75 at the end (500-step mean).

Training loss

Held-out bits per byte on 128 documents per source, measured every 500 steps. Lower is better.

Held-out set 85B tokens (decay start) 100B tokens Change
General web (ClimbMix) 0.823 0.715 βˆ’13%
Math 0.757 0.634 βˆ’16%
Curated projects 0.650 0.514 βˆ’21%
MIT collection 0.614 0.474 βˆ’23%
Starred Python 0.564 0.407 βˆ’28%
Other code 0.447 0.352 βˆ’21%
Starred other code 0.420 0.336 βˆ’20%
Other Python 0.398 0.313 βˆ’21%

Held-out bits per byte

Most of the improvement after 50B tokens came in the decay phase. Between 50B and 85B tokens at constant learning rate, held-out scores improved by about 1% or less.

No benchmark scores yet. The planned evaluation is code infilling (SAFIM and HumanEval-Infilling), which needs fill-in-the-middle training first: this base model has not seen the GLM <|code_prefix|>, <|code_suffix|> and <|code_middle|> tokens.

Samples

samples/base-samples.json holds 12 prompts (Python, JavaScript, Rust, SQL, a README, prose and math), each continued greedily and with sampling (temperature 0.8, top-p 0.95, seed 0) for up to 256 tokens. The model writes fluent, idiomatic code and well-formed documents; the code is often wrong in substance (a broken Dijkstra, an LRU cache that misuses heapq), greedy decoding repeats itself on open-ended text, and the math is unreliable.

Run history

The run was trained in sessions of up to six hours on Modal. Each launch has a 24-hour limit, so the run took seven launches (attempts a1 to a7), each resuming from a checkpoint of the lineage.

Training throughput

  • Superseded branches. Three stretches of training were discarded and retrained from an earlier checkpoint: steps 9–8,276 of the first launch (sessions pretrain-s01, pretrain-s02), after a preemption of the workflow controller opened a second attempt; steps 27,909–30,527 of a3-s02, after a GPU preemption; and steps 85,676–85,972 of a6-s04, cut off by a launch deadline before its final checkpoint. Their logs are kept under logs/sessions/ and are excluded from the lineage files.
  • Code versions. Attempts a1 to a4 ran commit 0609859 of koogle/lm-exploration; a5 to a7 ran a5c0d47, which adds continuation from in-session checkpoints after a preemption. The trainer source is identical in both.
  • Throughput steps. Most level shifts in the throughput chart fall at session boundaries, where a new session can land on a different 8 Γ— H100 host.

Files

Path Contents
tokens-099999547392/ Final weights: model.safetensors, config.json, progress.json
tokens-005000658944/ to tokens-095000985600/ Intermediate exports every 5B tokens
logs/lineage-metrics.jsonl One row per training step for the final lineage (loss, MTP loss, learning-rate factor, gradient norm, throughput, MFU, per-source token counts), with the session it came from
logs/lineage-validation.jsonl Held-out evaluations every 500 steps for the final lineage
logs/sessions/<session>/ Every session's raw logs, including superseded ones: metrics.jsonl, validation.jsonl, torchrun.log, report.json, session.json, config.json, run-config.json, provenance.json
logs/workflow/ The workflow's records: stages.json, per-attempt workflow.json and per-session segment-NN.json reports
metrics.jsonl, validation.jsonl Logs of the last session, uploaded by the trainer
charts/ The charts in this card
samples/base-samples.json Generations from the final weights

Usage

The weights load with the model code in koogle/lm-exploration (training/hybrid/), not with transformers. Decoding needs a CUDA GPU.

import torch
from model_hybrid import generate
from tokenizer import glm

lm = generate.load_export("tokens-099999547392", dtype=torch.bfloat16, device="cuda")
tokenizer = glm.load()
prompt = tokenizer.encode("def fibonacci(n: int) -> int:\n", add_special_tokens=False).ids
[ids] = generate.generate_batch(lm, [prompt], 128, (glm.EOS_ID,))
print(tokenizer.decode(ids))

Limitations

  • A base model: it continues text and does not follow instructions or hold a conversation.
  • Code it writes often looks right and is wrong; nothing here measures functional correctness.
  • No fill-in-the-middle training yet.
  • English-centric web data; other languages are barely represented.
  • Trained on public web text and code, which contain errors, insecure code and offensive content.

License

Malama Base is released under CC BY-NC 4.0: free to use, share and adapt for non-commercial purposes, with attribution.

The license follows the most restrictive terms among the training data:

Source Share Terms
General web: ClimbMix, read from karpathy/climbmix-400b-shuffle, a reshuffle of NVIDIA's ClimbMix 50.0% CC BY-NC 4.0 (NVIDIA's license; the reshuffle's MIT label does not change it). This is the binding term.
Math: nvidia/Nemotron-CC-Math-v1 10.0% NVIDIA Data Agreement for Model Training: training permitted, trained models belong to the trainer. NVIDIA notes that models trained on it may be subject to the Phi-4 license (MIT), since the data was produced with phi-4.
GitHub code: codeparrot/github-code-clean 35.8% Files kept only under MIT, Apache-2.0, BSD-2/3-Clause, ISC, CC0-1.0 or Unlicense
MIT collection 0.2% MIT
Curated open-source projects 4.0% Each project's own license, including GPL projects (Linux, git, Blender)
GLM tokenizer (zai-org/GLM-4.7-Flash) MIT

Whether model weights are a derivative work of copyleft training code is unsettled; this license does not resolve that question for the GPL projects in the curated set. This section records how the license was chosen and is not legal advice.

Provenance

Workflow run kda-960m-foundation-20261110-01, plan SHA-256 b2095d578ec025f942caf8912aabb549422078226bf951ca0648b3b89ab9c80c
Model configuration training/hybrid/configs/kda-960m-100b.json
Data recipe training/hybrid/recipes/kda-foundation-100b-504010-v3.json
Tokenizer GLM tokenizer, SHA-256 19e773648cb4e65de8660ea6365e10acca112d42a854923df93db4a6f333a82d
Full training checkpoints With optimizer and data state: 100B tokens, jfrick/malama-base-checkpoint-100b (step-000095367.pt); 84.8B tokens, the last before the learning-rate decay, jfrick/malama-base-checkpoint-85b (step-000080895.pt)
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support