---
library_name: mlx
pipeline_tag: text-generation
tags:
- music
- piano-roll
- causal-lm
- qwen3
- safetensors
---

# Musicroll-50M

Musicroll-50M is a small Qwen3-style causal language model over a 259-token
byte-level music vocabulary. It was used as the MusicRoll negative-control model
in the replication artifact for "Many Next-Token Predictors are In-Context
Learners".

The checkpoint is provided as `model.safetensors` with a compact tokenizer:

- vocabulary size: 259
- hidden size: 512
- layers: 16
- attention heads: 4
- key/value heads: 2
- context length: 4096
- BOS/EOS/PAD token ids: 256/257/258

The bitstring replication harness encodes tasks as piano-roll-like text and
loads this checkpoint through the local MLX model loader.

## Intended Use

This upload is primarily for reproducing the MusicRoll negative-control runs in
the ICLManyReplication artifact. General music-generation behavior has not been
characterized here.