CMA-OT: Hierarchical Expert Supervision for Dance-t... | AI Research

CMA-OT: Hierarchical Expert Supervision for Dance-to-Music Generation
Dance-to-music (D2M) generation aims to create music that matches the rhythm and style of a dance video. A major hurdle in this field is the "semantic mismatch": dance videos provide sparse cues like basic rhythm, while music composition requires dense, complex information such as instrumentation, melody, and structural organization. Existing methods often struggle to bridge this gap, leading to music that lacks coherence and musicality. This paper introduces CMA-OT, a new framework that uses an external "music expert" to provide hierarchical guidance to the music generator, ensuring the final output is both rhythmically aligned and musically sophisticated. The same computer vision question is explored in Geospatial AI, Dataverse Metadata, and the..., which adds a research perspective.

Bridging the Semantic Gap

To solve the mismatch between sparse dance inputs and dense musical requirements, the authors propose a hierarchical supervision paradigm. Instead of relying only on the final audio output, the model uses a pre-trained multi-scale music expert. This expert acts like a composer consultant, providing guidance at three distinct levels: top-level (global style and genre), middle-level (melodic and harmonic phrases), and bottom-level (rhythmic textures and timbre). By aligning the generator’s internal features with these expert representations, the model learns to produce music with better structural integrity and expressive richness.

Curriculum-Guided Learning

Directly applying multi-scale supervision can be unstable because different musical scales vary significantly in structure. To manage this, the researchers developed a curriculum-guided learning strategy. The model follows a three-phase process: it first learns to align with the top-level expert signals, then gradually incorporates middle-level structures, and finally refines the output with bottom-level details. This progressive approach prevents conflicts between different scales and ensures that the model learns global musical organization before focusing on fine-grained temporal dynamics. The same ai systems question is explored in Efficient Test-Time Adaptation through Human-AI Interaction, which adds a research perspective.

Scale-Aware Alignment

Because the expert’s representations and the generator’s internal features often have different temporal resolutions, the authors introduced a "scale-aware" alignment mechanism based on Fused Gromov-Wasserstein (FGW) distance. Unlike standard methods that only look at global feature similarity, this mechanism models soft correspondences between the two sequences. It dynamically balances semantic consistency and structural preservation at each scale, allowing the model to adapt to the varying nature of musical information and achieve precise, fine-grained alignment.

Performance and Results

Extensive experiments on two datasets (AIST++ and TikTok) demonstrate that CMA-OT achieves state-of-the-art performance. The framework shows significant improvements in rhythmic synchronization, aesthetic quality, and overall musicality compared to existing methods. By moving beyond simple reconstruction losses and incorporating structured, hierarchical knowledge, the model successfully generates music that is more coherent, expressive, and better aligned with the underlying dance movements. The same ai systems question is explored in DRACO, which adds a research perspective. as detailed in the full paper on Arxiv