cstr/mt3-GGUF

MT3 — GGUF (ggml)

GGUF conversion of Google Magenta's MT3 multi-instrument music transcription model (Apache-2.0) for CrispStrobe/CrispASR (§250).

MT3 is a T5-style encoder/decoder that reads a log-mel spectrogram and emits MIDI-like event tokens, transcribing multiple instruments at once — every note carries a General MIDI program, and drums are a separate class. It is the only per-program note-event model in CrispASR (piano-transcription is 88-key piano, basic-pitch is single-track polyphonic).

Files

File	Size	Notes
mt3-f16.gguf	96 MB	190 tensors; encoder+decoder, sinusoidal absolute positions, event codec

Converted with models/convert-mt3-to-gguf.py, which decodes the T5X/zarr checkpoint with stdlib + numpy only (no JAX, t5x or TensorStore). Weights verified bit-exact against the reference PyTorch export (189/189 tensors, max abs diff 0.0).

Usage

crispasr --piano --backend mt3 -m auto --auto-download -f song.wav
crispasr --piano -m mt3-f16.gguf -f song.wav --piano-format json   # + program/instrument/is_drum
crispasr --piano -m mt3-f16.gguf -f song.wav --piano-format midi   # multi-track SMF

--piano-format midi writes a format-1 Standard MIDI File with one track per program and drums on GM channel 10.

Parity

Validated stage-by-stage against a numpy reference implementation of the upstream graph (tools/reference_backends/mt3.py):

stage	result
mel front end	cos 1.000000000
encoder output	cos 0.999999879
first-step decoder logits	cos 0.999999999, argmax identical
greedy tokens	identical, 22/22 steps
end-to-end note events	exact positional match (88/88, 5/5, 8/8 across three clips)

The GGUF records mt3.pos_embed=sinusoidal and mt3.use_relative_attention_bias=0; the runtime hard-fails rather than silently falling back to T5's relative-bucket attention, which is a known failure mode of other ports.