Multi-Modal Semantic Expansion with Constrained LLM... | AI Research

Multi-Modal Semantic Expansion with Constrained LLM Reranking for Conversational Music Recommendation presents a system designed for the ACM RecSys 2026 TalkPlayData Challenge. The authors, Naman Garg, Sarika Jain, and George Fazekas, developed a conversational recommender system that interprets user preferences through multi-turn dialogue to retrieve music and generate natural-language responses. The system aims to balance retrieval accuracy, response quality, and catalog diversity within a unified evaluation framework.

The Three-Stage Pipeline

The system processes requests through a multi-stage architecture:

1. Multi-Modal Retrieval: The system uses seven dense embedding spaces—including track- and user-level collaborative filtering, Qwen3 metadata, lyrics, attributes, audio (CLAP), and visual (SigLIP)—alongside BM25 lexical retrieval and artist substring matching. These signals are fused using weighted Reciprocal Rank Fusion (RRF), with weights optimized via differential evolution on a 500-session development set. 2. Lightweight Reranking: The system filters out previously played tracks, applies popularity smoothing, and uses a penalty to maintain catalog diversity. 3. Response Generation: The system uses GPT-4o-mini to generate persona-driven responses. To ensure high lexical diversity, the system assigns 10 distinct personas to the model, such as a "passionate music blogger" or "veteran vinyl record store owner," while enforcing constraints like sentence length and the exclusion of technical jargon.

Performance and Observations

The submitted system achieved a composite score of 0.3213 on the Blind B evaluation set. During development, the researchers observed that optimizing individual metrics can lead to negative outcomes in other areas, such as when aggressive LLM-guided reranking caused a significant drop in retrieval accuracy.
A notable finding from the Blind A evaluation was the sensitivity of LLM-guided artist injection. The authors observed that applying this technique to all sessions resulted in a 18.9% regression in nDCG. However, applying it conservatively—only to sessions where the target artist was already present in the top 20 results—yielded the best observed nDCG for that set.

Development-Time Trade-offs

The authors identified several components that were excluded from the final Blind B submission due to cost and complexity constraints. These included an album continuation signal, which improved nDCG by 40% on the development set, and an XGBoost LambdaMART model. Additionally, the team opted for GPT-4o-mini over a more expensive GPT-4.1 "Conversational Mirroring" prompt, which had achieved higher response quality scores during development.

Franklin Analysis

The authors’ analysis suggests that the performance gap between their development results and the final Blind B score is largely due to these deliberate deployment simplifications. The evidence indicates that the system is highly sensitive to the "operating range" of LLM interventions; specifically, the LLM adds value when the correct artist is already surfaced but ranked poorly, but it introduces "position bias" that degrades performance when applied to sessions where the retrieval is already accurate. This highlights the difficulty of maintaining a stable composite score when balancing retrieval precision with generative response quality.