SymphonyGen

3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton

Xuzheng He1, Nan Nan2,†, Zhilin Wang3, Ziyue Kang2, Zhuoru Mo4, Ao Li2, Yu Pan1, Xiaobing Li1, Feng Yu1, Xiaohong Guan1,2,†

1Central Conservatory of Music | 2Xi'an Jiaotong University | 3University of Science and Technology of China | 4Shenzhen University

† Corresponding authors

arXiv Code 🤗 Models

Abstract

Generating symphonic music requires simultaneously managing high-level structural form and dense, multi-track orchestration, yet existing symbolic models often struggle with a "complexity-control imbalance" between scalability and steerability. We present SymphonyGen, a 3D hierarchical framework for contemporary orchestral generation, whose cascading decoders decompose the bar, track, and event axes, keeping decoding memory far below flat token streams and enabling conditioning at every structural level. A beat-quantized multi-pitch harmony skeleton, which may be user-written, analyzed, or model-generated, provides "short-score" conditioning, enabling outline control while producing orchestral textures. The model is refined with reinforcement learning against a cross-modal acoustic reward from CLaMP 3 audio embeddings, and a dissonance-averse sampling algorithm suppresses unintended tonal clashes during inference. Objective evaluations show that both post-training mechanisms reduce dissonance while maintaining independent melodic metrics, and in subjective tests SymphonyGen is rated above baseline systems in quality and preference, significantly so among general listeners.

Best Examples

📢 Disclaimer: You are listening to cherry-picked examples of our model.

Average Examples

Orchestral Composition Task

SymphonyGen first generates a harmony skeleton and then produces the full orchestration based on that skeleton. Reinforced with CLaMP 3 score only

Starting with a Major Chord

Starting with a Minor Chord

Orchestral Arrangement Task

SymphonyGen uses harmony skeletons analyzed from excerpts in the SymphonyNet Dataset (validation split), and re-orchestrates the skeleton. Reinforced with CLaMP 3 score and track density

Baseline Systems

The baseline excerpts rated in the subjective listening tests, shown here under their original (de-anonymized) names: excerpts from the SymphonyNet Dataset, official demo samples of SymphonyNet and METEOR, and samples from NotaGen-X (label: Romantic, Brahms/Dvořák, Orchestral).

Dataset Excerpts

SymphonyNet

NotaGen

METEOR