Clef-0.2B

A ~0.2B-parameter decoder-only Transformer for symbolic music generation (MusicXML / MIDI). Given a short conditioning header (genre / ensemble / tempo / key / meter / instrument plan) it generates a multi-track score, and it also supports CONTINUE (extend a given prefix) and INFILL (fill masked bars).

The model was pretrained from scratch on symbolic scores and then instruction-tuned on those three tasks. This is the instruction-tuned checkpoint.

Model summary

Architecture	decoder-only Transformer, RoPE, RMSNorm, SwiGLU, GQA
Params	201,398,272 (~0.2B)
Layers / d_model	17 / 1024
Attention	16 query heads : 4 KV heads (GQA), FFN 2816
Context length	4096 tokens max (trained on 2048-token windows)
Tokenizer	tokenizer_v2 — bar-major REMI+, vocab 9507, 24 TPQN
Precision	trained in fp8 (torchao) + torch.compile; weights ship bf16

Weights format

Inference weights are provided as model.safetensors in Llama (LlamaForCausalLM) layout with a matching config.json, so they load with transformers / vLLM directly. This layout was numerically verified against the native training checkpoint (greedy decode matches token-for-token). Note the tokenizer is custom (bar-major REMI+, in tokenizer/) — it is not an AutoTokenizer, so encoding/decoding to MusicXML/MIDI uses this project's pipeline, not AutoTokenizer.from_pretrained.

Tokenizer / codec

The tokenizer is bar-major REMI+: tracks are interleaved bar by bar so all instruments advance together, which gives the model an explicit notion of simultaneity and helps parts stop together instead of one voice trailing off.

A generation begins with a native conditioning header:

BOS [GENRE] [ENSEMBLE] TEMPO KEY METER <PLAN> {TRACK|DRUM}* <PLAN_END> (BAR ...)* EOS

Drums are encoded by their true General MIDI percussion number (resolved from <part-list>/midi-instrument/midi-unpitched), and note velocity / dynamics are preserved. Because drums are stored as real GM voices, decoded output renders correct kick / snare / hi-hat directly with no display→GM remap.

Prompt example

A generation is conditioned by a header of special tokens; the model then produces the bars after <PLAN_END>. These are the exact prompts used for the sample files below.

Rock band — distortion + overdrive guitar, electric bass, drums; A minor, 130 BPM:

<BOS> <GENRE_popular> <ENSEMBLE_band> <TEMPO_130> <KEY_0:minor> <METER_4/4>
<PLAN> <TRACK_30> <TRACK_29> <TRACK_33> <DRUM_TRACK> <PLAN_END>

Piano concerto — piano + string ensemble + cello; E♭ major, 96 BPM:

<BOS> <GENRE_classical> <ENSEMBLE_orchestral> <TEMPO_96> <KEY_-3:major> <METER_4/4>
<PLAN> <TRACK_0> <TRACK_48> <TRACK_42> <PLAN_END>

• <TRACK_n> picks a General MIDI program (0 = piano, 30 = distortion guitar, 33 = electric bass, 48 = string ensemble, 42 = cello, …).
• <DRUM_TRACK> adds a percussion lane.
• <KEY_k:mode> uses k = signed sharps(+) / flats(−); <TEMPO_n> = BPM; <METER_a/b> = time signature.

Conditioning vocabulary

• GENRE <GENRE_x> (9): classical · popular · jazz · folk · electronic · soundtrack · world · religious · unknown
• ENSEMBLE <ENSEMBLE_x> (7): solo · duo · band · chamber · orchestral · choir · other
• TRACK <TRACK_n>: any General MIDI program, n = 0–127; plus <DRUM_TRACK> for a percussion lane.
• KEY <KEY_k:mode>: k = −7…+7 (sharps/flats), mode ∈ major · minor · dorian · phrygian · lydian · mixolydian · none.
• TEMPO <TEMPO_n>: integer BPM (5–999).
• METER <METER_a/b>: time signature (e.g. 4/4, 3/4, 6/8).

(The genre/ensemble labels are coarse; unknown / other are valid and common.)

Samples

samples/promo_rock.mid and samples/promo_piano_concerto.mid are ~3-minute pieces generated from the two prompts above (best-of-N seeds, CONTINUE-stitched past the context window). Render with any General MIDI SoundFont.

Training

• Pretraining: from scratch, 3 epochs (13,821 steps), global batch 256, fp8 + torch.compile. Data: ~1.18M bar-major windows built from the public sources listed below, with ×12 transpose augmentation.
• Instruction tuning (this checkpoint): GENERATE / CONTINUE / INFILL tasks mixed with in-key curation (in-key ≥ 0.88) and a base-replay set to limit forgetting. 760 steps (2 epochs), best val loss 0.3088 @ step 299 (adopted). Hardware: 1× RTX PRO 6000 Blackwell (96 GB), ~80k tok/s, ~45 min.

Evaluation

Automatic metrics on held-out generation prompts (2 seeds). Values are absolute for this model; real human-composed scores are given only as a rough anchor.

metric	Clef-0.2B	real scores (ref)
in-key ratio ↑	0.81	~0.96
dissonance ↓	0.29	~0.15
co-termination spread ↓	0.14	—
early part death ↓	0.10	—

Task success (model emits ≥ 1 new valid bar for the task):

task	success
CONTINUE	12/12
INFILL	12/12

• in-key ratio — fraction of notes inside the conditioned key.
• dissonance — ratio of dissonant simultaneous intervals.
• co-termination spread / early part death — how tightly instruments end together vs. voices dropping out early (lower is better).

How to generate

Generation runs in this project's pipeline (bar-major decode → MusicXML/MIDI), then any General MIDI SoundFont (e.g. MuseScore) renders audio.

Recommended defaults: temperature 0.7, top_k 24, repetition_penalty ~1.12. This is the setting used for the sample pieces and is a good starting point.

Optional (supported when decoding with vLLM / transformers): top_p 0.95 and presence_penalty 0.15 for extra variety. ⚠️ Do not use frequency_penalty — it destabilizes the structural tokens (BAR / TRK / PLAN). For pieces longer than the context window, continue past it with a slightly lower continuation temperature (~0.76) to soften seams. Optionally post-process drums with a sanitize_drums step that snaps rare out-of-kit hits (e.g. GM 71/72) back to a standard kit.

Limitations

• Context is 4096 tokens (~35 bars). Longer pieces are produced by stitching CONTINUE windows, which can cause a slight mood shift at the seam. A long-context retrain (or RoPE scaling at inference) would fix this properly.
• Dissonance stays somewhat above real-score levels.
• Research artifact; quality is not guaranteed for production use.

Training data

Built from the following public symbolic-music sources:

• PDMX — https://github.com/pnlong/PDMX
• KernScores / Humdrum — https://github.com/humdrum-tools/humdrum-data · https://kern.humdrum.org/
• OpenScore Lieder — https://github.com/OpenScore/Lieder
• The Session — https://github.com/adactio/TheSession-data

License

Released under Apache-2.0.