YuE2: Open Music Generation with Editable Scores and AI Agents
YuE2 turns lyrics and style prompts into editable musical scores, frontier-quality songs, zero-shot covers, and agent-driven revisions.
YuE2: Open Music Generation with Editable Scores and AI Agents
Most AI music generators hide the composition process behind a single prompt-to-audio operation. That makes them convenient, but difficult to control: if the harmony is wrong, the melody drifts, or the arrangement needs to change, the usual solution is to regenerate everything and hope for a better result.
YuE2 takes a different approach. It first creates an explicit, editable musical plan—containing melody and chord information—and then renders that plan into a complete stereo recording. The result is an open music-generation system designed not only for creation, but also for zero-shot covers, score-based editing, and agentic workflows.
What makes YuE2 different?
YuE2 supports three related workflows:
- Creation: provide lyrics and a style prompt, then generate a score and full song.
- Cover generation: transcribe a source recording, preserve its melody, and reinterpret it in a new style.
- Agentic editing: let a person or an AI agent revise harmony, melody, lyrics, tempo, instrumentation, or song form before rendering another version.
The central idea is a white-box music pipeline. Instead of treating audio as an opaque final artifact, YuE2 exposes the intermediate composition so it can be inspected, played, changed, and regenerated.
The repository reports that YuE2 is competitive with Suno v5 and v6 on WildSongBench. Its best-of-8 configuration reaches a 6.9632 SongBench Avg, the highest observed mean among the evaluated settings. The benchmark includes 192 prompts, although the authors note that the small gap between top systems does not establish statistical significance.
From lyrics to an editable score
The architecture combines an autoregressive/non-autoregressive mixture-of-transformers backbone with flow matching and a variational autoencoder:
- The model predicts a symbolic plan and semantic music tokens autoregressively.
- Acoustic latents are generated with flow matching.
- A VAE decodes those latents into 48 kHz stereo audio.
The staged API makes those steps visible:
plan() → generate_semantic() → synthesize() → decode()
This separation is useful for developers. You can reuse an exact plan, select a decoder, retain intermediate artifacts, and compare multiple audio realizations without discarding the composition itself.
Creation, covering, and editing differ mainly in where the score comes from:
- YuE2 generates the score from lyrics and style.
- SheetSage2 transcribes a source recording into a melody score.
- A user or agent edits an existing ABC composition.
Install and run YuE2 locally
The current quick-start environment targets Linux, Python 3.12, and an NVIDIA GPU with BF16 support and 24 GB of VRAM. The model files are downloaded from Hugging Face on first use, and the default pipeline produces unquantized 48 kHz stereo audio.
git clone https://github.com/multimodal-art-projection/YuE.git
cd YuE
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install .
python examples/generate.py \
--output outputs/first-song
The generated directory is more than an audio export. It retains the score, semantic tokens, acoustic latents, generation settings, and model identities. Keeping these artifacts makes experiments reproducible and gives you material for later editing.
The Python interface is similarly compact:
import json
from pathlib import Path
from yue2 import YuE2Pipeline
request = json.loads(
Path("examples/song.json").read_text(encoding="utf-8")
)
with YuE2Pipeline.from_pretrained(
"m-a-p/YuE2-3B",
device="cuda",
) as pipe:
song = pipe(**request)
song.save_artifacts("outputs/my-song")
print(song.truncated)
Choose the right generation mode
YuE2 exposes a cot setting that controls how much symbolic planning is used:
| Setting | Behavior |
|---|---|
cot="full" |
Generate an editable melody-and-chord plan; this is the default for new songs. |
cot="melody" |
Use a melody plan with free accompaniment; recommended for covers. |
cot="off" |
Generate directly from lyrics and style. |
You can also provide an ABC score through abc=.... Full-score generation gives you more control over harmony, while melody-only conditioning leaves the accompaniment free to adapt to a target style.
Generate a zero-shot cover
A cover workflow begins with transcription. Use SheetSage2 to extract the source melody as ABC notation, review the transcription, and then supply new lyrics or a different target style to YuE2.
For covers, the repository recommends cot="melody" and a score without chord symbols. This preserves the essential melody while allowing YuE2 to create new accompaniment that matches the requested genre.
from pathlib import Path
from yue2 import YuE2Pipeline
with YuE2Pipeline.from_pretrained(
"m-a-p/YuE2-3B",
device="cuda",
) as pipe:
cover = pipe(
style=(
"English, jazz-funk, warm lead vocal, Rhodes, "
"bass and drums"
),
lyrics=Path("cover-lyrics.txt").read_text(encoding="utf-8"),
abc=Path("cover-score/score.abc").read_text(encoding="utf-8"),
cot="melody",
seed=42,
)
cover.save_artifacts("outputs/cover")
The reported zero-shot cover evaluation is notable: across 948 works, full-score YuE2 reaches 0.647 CLEWS mAP, compared with 0.006 without a score. This is achieved with the general generator rather than a cover-specific fine-tuned model. Melody-only covers trade some source fidelity for more freedom to change the arrangement.
SheetSage2 runs in a separate environment and automatically loads its MERT2 encoder. The repository includes a complete transcription and generation guide, as well as an original melody example for score-conditioned generation without first processing a recording.
Edit the composition instead of regenerating blindly
The symbolic score is YuE2's most important developer-facing interface. First export a plan:
import json
from pathlib import Path
from yue2 import YuE2Pipeline
request = json.loads(
Path("examples/song.json").read_text(encoding="utf-8")
)
with YuE2Pipeline.from_pretrained(
"m-a-p/YuE2-3B",
device="cuda",
) as pipe:
plan = pipe.plan(**request)
plan.save("outputs/plan")
Copy outputs/plan/score.abc to an editable file, such as edited.abc. You can then revise harmony, melody, tempo, instrumentation, or form manually or through an agent. Render the modified score with:
python examples/generate.py \
--request examples/song.json \
--abc-file edited.abc \
--cot full \
--output outputs/edited
This is not waveform-preserving editing. YuE2 generates a new complete recording from the edited composition; it does not retain the original audio outside the symbolic changes. That distinction matters when designing production workflows or comparing versions.
Agentic music editing
The repository includes a skills/yue2-music/ package for agents that support SKILL.md-style skill directories. It teaches an agent to:
- Generate songs from lyrics and style prompts.
- Transcribe and cover recordings.
- Edit ABC scores.
- Check musical invariants.
- Organize listening comparisons.
The Python runtime is installed separately with pip install .; installing the skill package does not replace the model environment.
A useful agent request might be:
Use the yue2-music skill to create an English piano-pop song. Keep the original audio and score. Make a second version with jazz harmony, preserve the vocal melody and lyric order, and give me both versions to compare.
The project demonstrates this style of interaction with The Last Train: nine editing steps and fourteen versions move from Mandarin pop to English jazz, including new harmony and a saxophone solo. Each version can be listened to alongside its conversation, score, prompt, and lyrics.
WildSongBench results
The reported benchmark uses 192 WildSongBench prompts and the benchmark decoder, YuE2-Vae-legacy. Standard YuE2 selects from two candidates, while best-of-8 selects from eight.
| System / setting | SongBench Avg | AudioBox PQ | MuLan | PER |
|---|---|---|---|---|
| YuE2 best-of-8 | 6.9632 | 8.2714 | 0.5051 | 9.79% |
| Mureka 9 | 6.9377 | 8.0226 | 0.4394 | 11.69% |
| Suno v5 | 6.8721 | 8.1698 | 0.5428 | 8.10% |
| YuE2 | 6.7316 | 8.2598 | 0.5068 | 8.44% |
| Suno v5.5 | 6.7150 | 8.1955 | 0.5089 | 5.96% |
| Suno v4.5 | 6.6995 | 8.2541 | 0.5022 | 5.80% |
| Suno v6 | 6.5562 | 8.1296 | 0.4916 | 7.58% |
| Suno v6 Wild | 6.4195 | 8.1785 | 0.4999 | 7.45% |
| LeVo 2 | 6.3247 | 8.3966 | 0.3542 | 26.12% |
| MiniMax Music 2.6 | 6.3222 | 8.1711 | 0.4251 | 24.55% |
| MiniMax Music 3 | 6.2830 | 8.2825 | 0.3928 | 6.27% |
| HeartMuLa | 6.2483 | 8.2933 | 0.3823 | 10.71% |
| Muse | 6.0349 | 8.0517 | 0.3937 | 33.42% |
| ACE-Step 1.5 | 6.0118 | 8.0518 | 0.4372 | 7.46% |
| DiffRhythm 2 | 5.2428 | 7.9782 | 0.3782 | 18.41% |
| YuE 1 | 4.9165 | 7.8683 | 0.2623 | 36.38% |
| SongBloom | 4.2350 | 8.1539 | 0.2697 | 19.19% |
The table should be read as a multi-metric comparison rather than a single leaderboard. SongBench Avg favors YuE2 best-of-8, while other systems lead individual metrics. The authors explicitly caution that rankings and small differences should not be treated as proof of statistical significance.
Related models and resources
YuE2 is part of a broader music AI stack:
- YuE2-3B: generation, symbolic planning, covers, and editing.
- YuE2-Vae: default generation and listening decoder.
- YuE2-Vae-legacy: decoder used for the reported benchmark protocol.
- SheetSage2: audio-to-score transcription.
- MERT-v2-FullSong: full-song representations used by SheetSage2.
- MERT-v2-30s: representations for short recordings.
- WildSongBench: evaluation prompts and benchmark resources.
MERT2 feature extraction is optional for ordinary generation; YuE2 does not require a separate MERT2 model download.
Licensing and practical considerations
The current repository's first-party code, documentation, and agent skill are licensed under Apache 2.0. Model weights are separately licensed under CC BY-NC 4.0, and third-party components retain their original licenses. The archived YuE-v1 branch preserves its original license, while older yue2-v0.1.6 archives retain their bundled licenses.
That separation is important for commercial deployments. Apache 2.0 covers the source code, but it does not automatically grant commercial rights to the model weights or third-party assets. Review all applicable licenses before shipping a product or distributing generated content.
Why YuE2 matters
YuE2's main contribution is not simply another text-to-song checkpoint. It proposes a more controllable abstraction for generative music: compose in symbols, then create in sound. Scores make the system easier to inspect, allow deterministic or semi-deterministic experiments, and provide a shared interface for people and agents.
For developers, this opens several practical workflows: batch-rendering arrangement variants, evaluating edits against musical invariants, building conversational songwriting tools, and preserving the reasoning artifacts behind each audio version. With a 24 GB BF16-capable GPU and the released models, YuE2 offers an unusually open path from a lyric prompt to editable, reproducible music production.