Back to AI Research

AI Research

Multi-Modal Semantic Expansion with Constrained LLM... | AI Research

Key Takeaways

  • Multi-Modal Semantic Expansion with Constrained LLM Reranking for Conversational Music Recommendation presents a system designed for the ACM RecSys 2026 Talk...
  • We present Team Semiintelligencn's solution for the ACM RecSys 2026 TalkPlayData Challenge, addressing conversational music recommendation through a multi-modal and personalized conversational recommender system.
  • We optimize RRF weights on a 500-session development split via differential evolution, improving MRR by +19.5%.
  • The submitted system achieves a Blind B composite score of 0.3213.
  • Multi-Modal Semantic Expansion with Constrained LLM Reranking for Conversational Music Recommendation presents a system designed for the ACM RecSys 2026 TalkPlayData Challenge.
Paper AbstractExpand

We present Team Semiintelligencn's solution for the ACM RecSys 2026 TalkPlayData Challenge, addressing conversational music recommendation through a multi-modal and personalized conversational recommender system. Our submitted system employs a three-stage pipeline: (1) multi-modal retrieval constructing decay-weighted centroids across seven dense embedding spaces - track- and user-level CF-BPR, Qwen3 (metadata, lyrics, attributes), CLAP audio, and SigLIP visual - supplemented by BM25 lexical retrieval and an artist substring-match signal, all fused via weighted Reciprocal Rank Fusion (RRF) with optimized signal weights; (2) lightweight reranking (history filtering, popularity smoothing, and catalog diversity penalization); and (3) persona-diversified response generation using GPT-4o-mini. Beyond this submitted configuration, we report development-time experiments with additional components - constrained LLM-guided artist injection, album continuation signals, XGBoost LambdaMART, and a superior GPT-4.1 response prompt - that were not deployed to Blind B due to cost and complexity constraints. We optimize RRF weights on a 500-session development split via differential evolution, improving MRR by +19.5%. On Blind A, we observe that unconstrained LLM-guided injection across 54 sessions causes catastrophic nDCG regression (-18.9%), while conservative injection on only 9 sessions yields the best observed Blind A nDCG - a finding we present as a Blind A observation warranting further validation. The submitted system achieves a Blind B composite score of 0.3213.

Multi-Modal Semantic Expansion with Constrained LLM Reranking for Conversational Music Recommendation presents a system designed for the ACM RecSys 2026 TalkPlayData Challenge. The authors, Naman Garg, Sarika Jain, and George Fazekas, developed a conversational recommender system that interprets user preferences through multi-turn dialogue to retrieve music and generate natural-language responses. The system aims to balance retrieval accuracy, response quality, and catalog diversity within a unified evaluation framework. Methods and results are detailed in the full paper on arxiv.org.

The Three-Stage Pipeline

The system processes requests through a multi-stage architecture: 1. Multi-Modal Retrieval: The system uses seven dense embedding spaces—including track- and user-level collaborative filtering, Qwen3 metadata, lyrics, attributes, audio (CLAP), and visual (SigLIP)—alongside BM25 lexical retrieval and artist substring matching. These signals are fused using weighted Reciprocal Rank Fusion (RRF), with weights optimized via differential evolution on a 500-session development set. 2. Lightweight Reranking: The system filters out previously played tracks, applies popularity smoothing, and uses a penalty to maintain catalog diversity. 3. Response Generation: The system uses GPT-4o-mini to generate persona-driven responses. To ensure high lexical diversity, the system assigns 10 distinct personas to the model, such as a "passionate music blogger" or "veteran vinyl record store owner," while enforcing constraints like sentence length and the exclusion of technical jargon. The same Large Language Models question is explored in pro-team at LLMs4OL 2026 Tasks Flagship..., which adds a research perspective.

Performance and Observations

The submitted system achieved a composite score of 0.3213 on the Blind B evaluation set. During development, the researchers observed that optimizing individual metrics can lead to negative outcomes in other areas, such as when aggressive LLM-guided reranking caused a significant drop in retrieval accuracy. A notable finding from the Blind A evaluation was the sensitivity of LLM-guided artist injection. The authors observed that applying this technique to all sessions resulted in a 18.9% regression in nDCG. However, applying it conservatively—only to sessions where the target artist was already present in the top 20 results—yielded the best observed nDCG for that set. The same AI Evaluation question is explored in Verify Smarter, Evolve Further, which adds a research perspective.

Development-Time Trade-offs

The authors identified several components that were excluded from the final Blind B submission due to cost and complexity constraints. These included an album continuation signal, which improved nDCG by 40% on the development set, and an XGBoost LambdaMART model. Additionally, the team opted for GPT-4o-mini over a more expensive GPT-4.1 "Conversational Mirroring" prompt, which had achieved higher response quality scores during development. To see music in practice, How To Make Your Own AI... walks through a concrete example.

Franklin Analysis The authors’ analysis suggests that the performance gap between their development results and the final Blind B score is largely due to these deliberate deployment simplifications. The evidence indicates that the system is highly sensitive to the "operating range" of LLM interventions; specifically, the LLM adds value when the correct artist is already surfaced but ranked poorly, but it introduces "position bias" that degrades performance when applied to sessions where the retrieval is already accurate. This highlights the difficulty of maintaining a stable composite score when balancing retrieval precision with generative response quality.

More from Franklin

pro-team at LLMs4OL 2026 Tasks Flagship...researchVerify Smarter, Evolve FurtherresearchCounterfactual Bias Testing for Application Tracking...researchHow To Make Your Own AI...videoYour Phone Just Became a Mini...videoCreate A Full Music Video In...video

Comments