MuScriptor: an open model that turns real songs into sheet music
Hand a machine a full band recording, and get back the actual notes. Every instrument, written out as a score. That's MuScriptor, from a Kyutai, Mirelo, and IRCAM team, and the weights are open.
The errors vanish as it trains
You've seen apps turn speech into text. This does it for a whole band. Here's one guitar part from a real song. Up top, the last open model. It misses notes, shown in green, and gets others wrong, in red. Watch them turn blue, correct, as MuScriptor adds real data, then reinforcement learning. The errors basically vanish.
It won on data and training, not a new architecture
Here's the surprise. It's not a cleverer network, it's a plain transformer. They pretrained on a million and a half synthetic scores, then fine tuned on a hundred and seventy thousand real recordings. Eleven thousand hours, classical to heavy metal. That real audio pass adds about twenty points. A final round of reinforcement learning adds six more. Data, not architecture.
More than double the best open model on the full-mix metric
So how good is it. The benchmark is the authors' own held-out set, scored against the previous open model, YourMT3 plus. The headline metric asks one thing. Did it get the right note, and the right instrument. MuScriptor scores forty eight point two. YourMT3 plus, twenty one point nine. More than double. Frame accuracy, seventy three versus forty six.
The honest read: it wins on notes, not yet on timing
Now the honest part. That crown is on the team's own test set. On outside datasets it still wins on pitch content, but it often loses on exact timing. On Bach ten, its onset score is forty three, below YourMT3 plus at sixty. And forty eight point two Multi means under half the notes land, at a strict fifty millisecond window. A step change from unusable, not perfect. And for builders, the weights are non-commercial, and gated.
Try it now: paste a song, get the notes
You can try it yourself. There's a live web demo, inference code you run with one command, and three model sizes on Hugging Face. This is the demo transcribing a real rock track, each instrument drawn in its own color.
The open bar for real-world transcription just moved
The open bar for turning real recordings into scores just moved. And it moved on data and training, not a new architecture. The thing to watch now is whether the timing gap closes.
