Duplicate from cocktailpeanut/muscriptor-large

Co-authored-by: cocktail peanut <cocktailpeanut@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,171 @@
+---
+license: cc-by-nc-4.0
+library_name: muscriptor
+tags:
+- music
+- music-transcription
+- automatic-music-transcription
+- amt
+- audio-to-midi
+- midi
+- music-information-retrieval
+- transformer
+- pytorch
+---
+
+# MuScriptor — large (≈1.3B)
+
+**MuScriptor** is an open-weight model for **general-purpose, multi-instrument automatic music transcription (AMT)**: it converts a music recording (any genre, multiple simultaneous instruments) into a stream of notes played. This repository hosts the **large** variant (≈1.3B parameters) — the **flagship, best-quality checkpoint**.
+
+For a smaller footprint use [`muscriptor-medium`](https://huggingface.co/MuScriptor/muscriptor-medium) (≈300M, good trade-off) or [`muscriptor-small`](https://huggingface.co/MuScriptor/muscriptor-small) (≈100M, fastest).
+
+- Developed by [Mirelo](https://www.mirelo.ai/) x [kyutai](https://kyutai.org/)
+- 📄 Paper: *MuScriptor: An Open Model for Multi-Instrument Music Transcription* — Rouard, Krause, Roebel, Simon-Gabriel, Défossez (2026). _<!-- TODO: add arXiv link once public; it will auto-cross-link on the Hub -->_
+- 💻 Code: <https://github.com/muscriptor/muscriptor>
+- 🔊 Audio samples: <https://muscriptor.github.io>
+
+## Table of contents
+
+- [Quickstart](#quickstart)
+- [Model description](#model-description)
+- [Model variants](#model-variants)
+- [Intended uses & limitations](#intended-uses--limitations)
+- [Instrument conditioning](#instrument-conditioning)
+- [Training](#training)
+- [Evaluation](#evaluation)
+- [Citation](#citation)
+- [License](#license)
+
+## Quickstart
+
+Install the `muscriptor` package (it uses `huggingface_hub` to fetch weights automatically):
+
+```bash
+pip install git+https://github.com/muscriptor/muscriptor.git
+# TODO (PyPI release forthcoming: pip install muscriptor)
+```
+
+### Python
+
+```python
+from pathlib import Path
+from muscriptor import TranscriptionModel
+
+# "large" resolves to hf://MuScriptor/muscriptor-large and downloads on first use.
+model = TranscriptionModel.load_model("large")
+
+# Get a MIDI file directly:
+Path("out.mid").write_bytes(model.transcribe_to_midi("audio.wav"))
+
+# Or stream note events as they are transcribed:
+for event in model.transcribe("audio.wav"):
+    print(event)  # NoteStartEvent / NoteEndEvent / ProgressEvent
+```
+
+`load_model` accepts a size keyword (`"small"`/`"medium"`/`"large"`), a local `.safetensors` path, or an `hf://` / `https://` URL. Weights loaded by size keyword (or any `hf://` URL) are cached in the standard Hugging Face cache (`~/.cache/huggingface/hub`, configurable via `HF_HOME`); weights fetched from a plain `http(s)://` URL are cached under `~/.cache/muscriptor/`. Input audio can be WAV or any format `libsndfile` reads (mp3, flac, ogg, m4a, …); it is resampled to 16 kHz mono internally.
+
+### CLI
+
+```bash
+muscriptor transcribe --model large audio.wav -o out.mid
+```
+
+## Model description
+
+MuScriptor performs transcription by **autoregressively predicting a MIDI-like token sequence** given the mel-spectrogram of a short audio segment, following the sequence-to-sequence AMT paradigm (cf. MT3). It deliberately avoids complex architectural tweaks in favor of a simple, decoder-only Transformer.
+
+- **Architecture:** decoder-only Transformer (this variant: `dim=1536`, `num_heads=24`, `num_layers=48`).
+- **Input:** raw waveform (16 kHz, mono) of a 5-second segment → mel-spectrogram (STFT `n_fft=2048`, hop 160 → 100 Hz frame rate, 512 mel bins). The spectrogram is projected to the model dimension and used as a prefix condition.
+- **Output tokenization:** MT3-like note events; the 128 MIDI programs are mapped to **36 instrument subgroups** using the `MT3_FULL_PLUS` taxonomy. Decoding is greedy (argmax) by default, with optional classifier-free guidance (CFG).
+- **Inference:** audio is processed in 5-second chunks; note events are emitted in temporal order. Optional **instrument conditioning** stabilizes predictions across chunk boundaries and lets you restrict/customize the transcription (see below).
+
+**Note on the representation:** the tokenizer recovers onset/offset timing, pitch, and instrument, but **not velocity**. It also cannot represent two notes of the same pitch and instrument sounding at the same time. Drums are onset-only.
+
+## Model variants
+
+| Repo | Params | `dim` | heads | layers | Notes |
+|---|---|---|---|---|---|
+| [`muscriptor-small`](https://huggingface.co/MuScriptor/muscriptor-small) | ≈100M | 768 | 12 | 14 | smallest / fastest |
+| [`muscriptor-medium`](https://huggingface.co/MuScriptor/muscriptor-medium) | ≈300M | 1024 | 16 | 24 | good trade-off |
+| [`muscriptor-large`](https://huggingface.co/MuScriptor/muscriptor-large) | ≈1.3B | 1536 | 24 | 48 | **this model** · best quality |
+
+All variants share the same input pipeline, tokenizer, and training recipe; they differ only in latent dimension, attention heads, and depth.
+
+## Intended uses & limitations
+
+**Intended uses**
+- General-purpose transcription of real, multi-instrument music across genres (classical → heavy metal) into MIDI.
+- A building block for music information retrieval (chord/key recognition), musicological analysis, generative-modeling data pipelines, and tools for musicians.
+
+**Out of scope / use with care**
+- Not a substitute for a hand-annotated score; expect errors, especially on dense mixes, unusual timbres, and heavily processed audio.
+- Velocity/dynamics are **not** produced (see note above).
+- Onset/offset precision is lower for some styles (e.g. choral music), and exact offsets are inherently harder than onsets.
+
+**Limitations & biases**
+- Training data skews toward pop and Western classical music, and the instrument distribution is long-tailed (piano/guitar/bass/drums are most frequent). Rare instruments and underrepresented genres may be transcribed less reliably.
+- The fixed `MT3_FULL_PLUS` 36-group instrument taxonomy limits instrument granularity.
+- Simultaneous same-pitch/same-instrument notes cannot be represented by the tokenizer.
+
+## Instrument conditioning
+
+The model can be told which instrument groups are present in the track. Supplying the correct set improves quantitative scores and produces more coherent instrument assignments across segments.
+
+```python
+from muscriptor.tokenizer.mt3 import MT3_FULL_PLUS_GROUP_NAMES
+
+# `instrument_group` is a space-separated string of MT3_FULL_PLUS group IDs.
+# Convert readable group names to IDs:
+names = ["acoustic_piano", "acoustic_guitar", "acoustic_bass"]
+instrument_group = " ".join(str(MT3_FULL_PLUS_GROUP_NAMES[n]) for n in names)  # -> "0 4 7"
+
+# Only expect piano, acoustic guitar and bass in this track:
+model.transcribe_to_midi("audio.wav", instrument_group=instrument_group)
+```
+
+```bash
+muscriptor transcribe --model large --instruments "acoustic_piano,acoustic_guitar,acoustic_bass" audio.wav -o out.mid
+muscriptor list-instruments   # show all available group names
+```
+
+## Evaluation
+
+Metrics are instrument-agnostic F1 scores computed with [`mir_eval`](https://github.com/craffel/mir_eval).
+
+### Headline results on `D_Test`
+
+`D_Test` is the authors' held-out test set of 372 multi-instrument tracks. Results below are for this 1.3B model with the full training pipeline (`D_Synth` + `D_Real` + `D_RL`), CFG = 2:
+
+| Model | Onset F1 | Frame F1 | Offset F1 | Drums F1 | Multi F1 |
+|---|---|---|---|---|---|
+| YourMT3+ (baseline) | 32.5 | 45.5 | 17.8 | 41.4 | 21.9 |
+| **MuScriptor 1.3B** | **60.4** | **72.4** | **48.6** | **49.6** | **47.8** |
+
+### Model-size comparison
+
+F1 ↑ on `D_Test` from the paper's scaling study (models trained on `D_Real` only, CFG = 2). Note these ablation numbers omit synthetic pre-training and RL, so they are **lower** than the full-pipeline results above:
+
+| Variant | Params | Onset | Frame | Offset | Drums | Multi |
+|---|---|---|---|---|---|---|
+| `muscriptor-small` | 100M | 51.2 | 67.2 | 38.7 | 41.5 | 38.2 |
+| `muscriptor-medium` | 300M | 52.4 | 68.0 | 40.3 | 42.0 | 39.7 |
+| **`muscriptor-large`** | **1.3B** | **53.2** | **68.7** | **41.0** | **42.5** | **40.5** |
+
+
+## Citation
+
+```bibtex
+@inproceedings{muscriptor2026,
+  title     = {MuScriptor: An Open Model for Multi-Instrument Music Transcription},
+  author    = {Rouard, Simon and Krause, Michael and Roebel, Axel and
+               Simon-Gabriel, Carl-Johann and D{\'e}fossez, Alexandre},
+  year      = {2026},
+  note      = {Kyutai, Mirelo AI, IRCAM}
+}
+```
+
+<!-- TODO: replace with the final published citation (venue / arXiv id) once available. -->
+
+## License
+
+Code released under the [MIT License](https://github.com/muscriptor/muscriptor/blob/main/LICENSE). Weights released under CC-BY-NC.

config.json

@@ -0,0 +1,8 @@
+{
+  "model_type": "muscriptor",
+  "variant": "large",
+  "dim": 1536,
+  "num_heads": 24,
+  "num_layers": 48,
+  "card": 1395
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac4eb6ea87dfc26b6ca6b954c6b967ab87ad4c7d08e078b25214f13ed051f397
+size 5465642136