Update README.md

README.md

@@ -1,3 +1,193 @@
 ---
 license: cc-by-nc-4.0
+library_name: muscriptor
+extra_gated_prompt: >-
+  MuScriptor is the result of a research collaboration between Mirelo and Kyutai
+  whose purpose is to transcribe audio to MIDI/music sheet. It is provided
+  primarily for research purposes under the CC BY-NC 4.0 licence supplemented by
+  the below specific conditions of use.
+
+  Specific conditions of use: MuScriptor and any generated content by MuScriptor
+  are provided as is without any warranty of any kind, including but not limited
+  to any warranty of non-infringement. Use of MuScriptor and its output must
+  comply with all applicable laws and must not result in, involve, or facilitate
+  any illegal or unauthorized activity. Prohibited uses include, without
+  limitation, inputting music files and transcribing them to MIDI/music sheet
+  without having all the necessary rights, including intellectual property
+  rights, under applicable laws. Accordingly, users of MuScriptor undertake and
+  warrant to have all the necessary rights, including intellectual property
+  rights, in connection with their use of MuScriptor and its output. We disclaim
+  all liability for any non-compliant use and users of MuScriptor shall
+  indemnify, defend, and hold harmless Mirelo and Kyutai from and against any
+  and all claims, damages, losses, liabilities, and expenses (including
+  reasonable attorneys' fees) incurred by Mirelo and/or Kyutai arising out of or
+  resulting from their failure to comply with the terms of the CC BY-NC 4.0
+  licence and/or these specific conditions of use.
+extra_gated_fields:
+  Company or university if applicable: text
+  I am a:
+    type: select
+    options:
+    - Musician
+    - AI Researcher
+    - Other
+tags:
+- music
+- music-transcription
+- automatic-music-transcription
+- amt
+- audio-to-midi
+- midi
+- music-information-retrieval
+- transformer
+- pytorch
+base_model:
+- MuScriptor/muscriptor-medium
 ---
+
+# MuScriptor — medium (≈300M)
+
+**MuScriptor** is an open-weight model for **general-purpose, multi-instrument automatic music transcription (AMT)**: it converts a music recording (any genre, multiple simultaneous instruments) into a stream of notes played. This repository hosts the **medium** variant (≈300M parameters), the default checkpoint downloaded by the `muscriptor` library.
+
+`muscriptor-medium` balances quality and footprint. For the best transcription quality use [`muscriptor-large`](https://huggingface.co/MuScriptor/muscriptor-large) (≈1.3B); for the smallest/fastest option use [`muscriptor-small`](https://huggingface.co/MuScriptor/muscriptor-small) (≈100M).
+
+- Developed by [Mirelo](https://www.mirelo.ai/) x [kyutai](https://kyutai.org/)
+- 📄 Paper: *MuScriptor: An Open Model for Multi-Instrument Music Transcription* — Rouard, Krause, Roebel, Simon-Gabriel, Défossez (2026). _<!-- TODO: add arXiv link once public; it will auto-cross-link on the Hub -->_
+- 💻 Code: <https://github.com/muscriptor/muscriptor>
+- 🔊 Audio samples: <https://muscriptor.github.io>
+
+## Table of contents
+
+- [Quickstart](#quickstart)
+- [Model description](#model-description)
+- [Model variants](#model-variants)
+- [Intended uses & limitations](#intended-uses--limitations)
+- [Instrument conditioning](#instrument-conditioning)
+- [Training](#training)
+- [Evaluation](#evaluation)
+- [Citation](#citation)
+- [License](#license)
+
+## Quickstart
+
+Install the `muscriptor` package (it uses `huggingface_hub` to fetch weights automatically):
+
+```bash
+pip install git+https://github.com/muscriptor/muscriptor.git
+# TODO (PyPI release forthcoming: pip install muscriptor)
+```
+
+### Python
+
+```python
+from pathlib import Path
+from muscriptor import TranscriptionModel
+
+# "medium" resolves to hf://MuScriptor/muscriptor-medium and downloads on first use.
+model = TranscriptionModel.load_model("medium")
+
+# Get a MIDI file directly:
+Path("out.mid").write_bytes(model.transcribe_to_midi("audio.wav"))
+
+# Or stream note events as they are transcribed:
+for event in model.transcribe("audio.wav"):
+    print(event)  # NoteStartEvent / NoteEndEvent / ProgressEvent
+```
+
+`load_model` accepts a size keyword (`"small"`/`"medium"`/`"large"`), a local `.safetensors` path, or an `hf://` / `https://` URL. Weights loaded by size keyword (or any `hf://` URL) are cached in the standard Hugging Face cache (`~/.cache/huggingface/hub`, configurable via `HF_HOME`); weights fetched from a plain `http(s)://` URL are cached under `~/.cache/muscriptor/`. Input audio can be WAV or any format `libsndfile` reads (mp3, flac, ogg, m4a, …); it is resampled to 16 kHz mono internally.
+
+### CLI
+
+```bash
+muscriptor transcribe --model medium audio.wav -o out.mid
+```
+
+## Model description
+
+MuScriptor performs transcription by **autoregressively predicting a MIDI-like token sequence** given the mel-spectrogram of a short audio segment, following the sequence-to-sequence AMT paradigm (cf. MT3). It deliberately avoids complex architectural tweaks in favor of a simple, decoder-only Transformer.
+
+- **Architecture:** decoder-only Transformer (this variant: `dim=1024`, `num_heads=16`, `num_layers=24`).
+- **Input:** raw waveform (16 kHz, mono) of a 5-second segment → mel-spectrogram (STFT `n_fft=2048`, hop 160 → 100 Hz frame rate, 512 mel bins). The spectrogram is projected to the model dimension and used as a prefix condition.
+- **Output tokenization:** MT3-like note events; the 128 MIDI programs are mapped to **36 instrument subgroups** using the `MT3_FULL_PLUS` taxonomy. Decoding is greedy (argmax) by default, with optional classifier-free guidance (CFG).
+- **Inference:** audio is processed in 5-second chunks; note events are emitted in temporal order. Optional **instrument conditioning** stabilizes predictions across chunk boundaries and lets you restrict/customize the transcription (see below).
+
+**Note on the representation:** the tokenizer recovers onset/offset timing, pitch, and instrument, but **not velocity**. It also cannot represent two notes of the same pitch and instrument sounding at the same time. Drums are onset-only.
+
+## Model variants
+
+| Repo | Params | `dim` | heads | layers | Notes |
+|---|---|---|---|---|---|
+| [`muscriptor-small`](https://huggingface.co/MuScriptor/muscriptor-small) | ≈100M | 768 | 12 | 14 | smallest / fastest |
+| [`muscriptor-medium`](https://huggingface.co/MuScriptor/muscriptor-medium) | ≈300M | 1024 | 16 | 24 | **this model** · good trade-off |
+| [`muscriptor-large`](https://huggingface.co/MuScriptor/muscriptor-large) | ≈1.3B | 1536 | 24 | 48 | best quality |
+
+All variants share the same input pipeline, tokenizer, and training recipe; they differ only in latent dimension, attention heads, and depth.
+
+## Intended uses & limitations
+
+**Intended uses**
+- General-purpose transcription of real, multi-instrument music across genres (classical → heavy metal) into MIDI.
+- A building block for music information retrieval (chord/key recognition), musicological analysis, generative-modeling data pipelines, and tools for musicians.
+
+**Out of scope / use with care**
+- Not a substitute for a hand-annotated score; expect errors, especially on dense mixes, unusual timbres, and heavily processed audio.
+- Velocity/dynamics are **not** produced (see note above).
+- Onset/offset precision is lower for some styles (e.g. choral music), and exact offsets are inherently harder than onsets.
+
+**Limitations & biases**
+- Training data skews toward pop and Western classical music, and the instrument distribution is long-tailed (piano/guitar/bass/drums are most frequent). Rare instruments and underrepresented genres may be transcribed less reliably.
+- The fixed `MT3_FULL_PLUS` 36-group instrument taxonomy limits instrument granularity.
+- Simultaneous same-pitch/same-instrument notes cannot be represented by the tokenizer.
+
+## Instrument conditioning
+
+The model can be told which instrument groups are present in the track. Supplying the correct set improves quantitative scores and produces more coherent instrument assignments across segments.
+
+```python
+from muscriptor.tokenizer.mt3 import MT3_FULL_PLUS_GROUP_NAMES
+
+# `instrument_group` is a space-separated string of MT3_FULL_PLUS group IDs.
+# Convert readable group names to IDs:
+names = ["acoustic_piano", "acoustic_guitar", "acoustic_bass"]
+instrument_group = " ".join(str(MT3_FULL_PLUS_GROUP_NAMES[n]) for n in names)  # -> "0 4 7"
+
+# Only expect piano, acoustic guitar and bass in this track:
+model.transcribe_to_midi("audio.wav", instrument_group=instrument_group)
+```
+
+```bash
+muscriptor transcribe --model medium --instruments "acoustic_piano,acoustic_guitar,acoustic_bass" audio.wav -o out.mid
+muscriptor list-instruments   # show all available group names
+```
+
+## Evaluation
+
+Metrics are instrument-agnostic F1 scores computed with [`mir_eval`](https://github.com/craffel/mir_eval) on `D_Test`, the authors' held-out test set of 372 multi-instrument tracks.
+
+**Model-size comparison** (F1 ↑; from the paper's scaling study, models trained on `D_Real` only, CFG = 2):
+
+| Variant | Params | Onset | Frame | Offset | Drums | Multi |
+|---|---|---|---|---|---|---|
+| `muscriptor-small` | 100M | 51.2 | 67.2 | 38.7 | 41.5 | 38.2 |
+| **`muscriptor-medium`** | **300M** | **52.4** | **68.0** | **40.3** | **42.0** | **39.7** |
+| `muscriptor-large` | 1.3B | 53.2 | 68.7 | 41.0 | 42.5 | 40.5 |
+
+These numbers come from the model-size ablation, which trains on real audio **only**. The **released checkpoints additionally use synthetic pre-training and RL post-training**, which improve real-world quality substantially beyond these figures. See [`muscriptor-large`](https://huggingface.co/MuScriptor/muscriptor-large) and the paper for per-dataset results.
+
+## Citation
+
+```bibtex
+@inproceedings{muscriptor2026,
+  title     = {MuScriptor: An Open Model for Multi-Instrument Music Transcription},
+  author    = {Rouard, Simon and Krause, Michael and Roebel, Axel and
+               Simon-Gabriel, Carl-Johann and D{\'e}fossez, Alexandre},
+  year      = {2026},
+  note      = {Kyutai, Mirelo AI, IRCAM}
+}
+```
+
+<!-- TODO: replace with the final published citation (venue / arXiv id) once available. -->
+
+## License
+
+Code released under the [MIT License](https://github.com/muscriptor/muscriptor/blob/main/LICENSE). Weights released under CC-BY-NC.