Heard a Song and Want the Sheet Music? This AI Can Write It for You
You know that feeling when you hear a great bassline or a chord progression, and you wish you could instantly get the notes? Maybe you’re a music teacher trying to create practice materials, a video editor looking for a sample, or a producer wanting to rework a mix. In the past, that meant hours of ear training or expensive software. Today, a new AI tool from a French lab called Kyutai might just make that process nearly effortless.
What Happened
Kyutai just released MuScriptor, an open-weight model that can transcribe multi-instrument audio into MIDI data. That means you feed it a recording – say, a full band mix – and it outputs a MIDI file with separate tracks for each instrument: bass, drums, piano, and more (source).
The model is a decoder-only Transformer, similar in family to language models like GPT, but trained on music. It processes a mel-spectrogram of the audio and predicts MIDI tokens step by step – effectively turning transcription into a language modeling task. The team released three size variants: Small (103M parameters), Medium (307M, the default), and Large (1.4B) (source).
What makes MuScriptor stand out is its training recipe. The model went through three stages: pre‑training on 1.45 million synthetic MIDI files, fine‑tuning on 170,000 real recordings (over 11,000 hours of audio), and then a reinforcement‑learning post‑training step using 300 manually verified tracks. The result is a massive jump in accuracy. On the held-out test set, the Multi F1 score – which measures correct note, timing and instrument – rose from 21.9 with the baseline to 48.2 with the full training pipeline (source). That’s more than double the performance.
Why This Matters for Your Business
If your SME is involved in music production, education, content creation, or even sound design, MuScriptor gives you a taste of what’s becoming possible with AI – and it’s available today.
- For a recording studio or home producer: You can take a rough demo and immediately extract the piano part, then re‑voice it with different sounds. No more manual transcription.
- For a music school or tutor: Turn popular songs into editable scores for your students. MuScriptor can output MIDI, which easily translates to sheet music in any notation software.
- For a video or podcast creator: Need to lift a music loop from an existing track? The model can transcribe it and let you isolate exactly the instrument you want.
- For an app or tool developer: The inference code is MIT‑licensed, so you can integrate it into your own products (provided you handle the non‑commercial weight license separately).
The tool even offers instrument conditioning – you can tell it to focus only on drums or acoustic piano, stabilising the output across longer recordings (source). That kind of control is what turns a clever demo into something you can actually use in a real workflow.
The Bigger Picture
MuScriptor is a window into a larger trend: AI models are getting scarily good at understanding and generating music. A few years ago, transcribing a full band mix required a PhD in signal processing or a paid subscription to an online service. Now, an open‑weight model that can run on your own laptop does it with surprising fidelity.
For you as a business owner, this means two things. First, the cost of high‑quality music transcription is dropping fast. Second, these tools are becoming standard building blocks – much like how speech‑to‑text is now everywhere. The businesses that figure out how to slot these capabilities into their services early are likely to have a real edge.
“You can now take any audio recording and convert it into editable MIDI in seconds. What might that unlock for your workflow?”
Of course, there are limitations. The model’s weights are licensed CC BY‑NC 4.0, so commercial deployment is not currently allowed without Kyutai’s permission (source). It also can’t handle overlapping same‑pitch same‑instrument notes, and the 5‑second segment limit means long pieces need to be stitched together. But for a first release, the performance is impressive – and it feels like a sign of where the industry is heading.
If you run a business that touches audio, this is worth paying attention to. The ability to extract musical information from recordings with a simple Python library could open up new revenue streams or save countless hours of manual work.
Curious how AI transcription could fit into your specific setup? Let’s talk about it.
Book a free 15-min call to see how AI transcription can apply to your business →