Omni-Interactive Universal Embedder

Omni-Interactive Universal Embedder | AI Research

Key Takeaways

Omni-Interactive Universal Embedder (OmniUE) is a framework designed to create a unified embedding space for text, video, and audio.
Multimodal representation learning has been shifting from traditional two-tower architectures to large language model (LLM)-based embedders due to their strong instruction-following capabilities.
Despite this progress, existing approaches primarily focus on language and image modalities, which also remain the dominant modalities for user-conditioned interactions in current embedders.
Within OmniUE, visual and audio segmenters process diverse user interactions and integrate them with an omni-LLM to produce user-conditioned any-to-any embeddings via context aggregation.
We believe that jointly advancing omni-modal representation learning and omni-interactive querying paves the way toward universal embedders.

Paper AbstractExpand

Multimodal representation learning has been shifting from traditional two-tower architectures to large language model (LLM)-based embedders due to their strong instruction-following capabilities. Despite this progress, existing approaches primarily focus on language and image modalities, which also remain the dominant modalities for user-conditioned interactions in current embedders. In this paper, we propose the first Omni-Interactive Universal Embedder (OmniUE), which not only learns a unified embedding space across text, video, and audio by leveraging intermediate-layer representations from dedicated learnable tokens, but also supports omni-interactive querying, enabling users to provide inputs in the form of text, visual regions of interest, and audio spans. Within OmniUE, visual and audio segmenters process diverse user interactions and integrate them with an omni-LLM to produce user-conditioned any-to-any embeddings via context aggregation. To evaluate OmniUE's omni-interactive capabilities, we introduce OmniCHOIR, benchmarking models for omni-interactive compositional audio retrieval based on the given text, video, and audio as well as unimodal or multimodal interaction prompts. OmniUE consistently surpasses state-of-the-art baselines across diverse modalities, with average improvements of 10.5% on textual-interactive video benchmarks (MMEB-v2-video), 1.1% on audio tasks (MAEB), 83.7% on visual-interactive benchmarks (SCaR), and 24.1% on our omni-interactive OmniCHOIR benchmark. We believe that jointly advancing omni-modal representation learning and omni-interactive querying paves the way toward universal embedders.

Omni-Interactive Universal Embedder (OmniUE) is a framework designed to create a unified embedding space for text, video, and audio. By moving beyond traditional text-only instructions, this model allows users to interact with and query embedding systems using a combination of text, visual regions of interest, and specific audio time spans.

How OmniUE Works

The model processes inputs through two primary streams: a query stream and an interactive prompt stream. The query stream encodes multimodal inputs into holistic representations, while the interactive prompt stream captures specific user signals.
To handle these inputs, the researchers integrated dedicated visual and audio segmenters—specifically SAM-3 for visual regions and SAM-Audio for temporal audio spans. These segmenters process user interactions and pass them through connectors into an omni-LLM. To capture complex, compositional information, the model appends learnable tokens to the input sequence. These tokens are extracted from intermediate layers of the LLM and processed through a context aggregation module, which uses adaptive weights to emphasize the most relevant information for a given task.

The OmniCHOIR Benchmark

To evaluate the model's ability to handle complex, multi-modal interactions, the authors introduced the OmniCHOIR benchmark. This testbed focuses on "text-video-audio-to-audio" (TVA2A) retrieval. In this task, a model must identify a specific sound within a mixed audio track, conditioned on a video and various interaction prompts. The benchmark is designed to be difficult by including "hard negative" distractors, such as background sounds that share categories with the target or contain similar audio characteristics, forcing the model to rely on precise multimodal cues rather than simple keyword matching.

Performance Results

According to the authors, OmniUE consistently outperformed existing state-of-the-art baselines across several established benchmarks:

MMEB-v2-video (text-interactive video): 10.5% average improvement.
MAEB (audio tasks): 1.1% average improvement.
SCaR (visual-interactive benchmarks): 83.7% average improvement.
OmniCHOIR (omni-interactive benchmark): 24.1% average improvement.

Franklin Analysis

The research suggests that current multimodal embedding models are limited by their reliance on text as the primary interface for user interaction. By enabling "omni-interactions"—where users can point to a specific visual region or select a precise audio segment—the authors provide a method to resolve ambiguities that text alone cannot address. The use of intermediate-layer token aggregation is a notable technical choice, as it attempts to solve the problem of information loss that often occurs when relying solely on the final hidden state of an LLM. The authors note that this approach maintains computational efficiency compared to other layer-aggregation methods that require significant parameter overhead.