MTEB for comparing embedding models across tasks

How it works

MTEB organizes datasets and metrics behind a common benchmark runner. It makes comparisons easier while keeping task type, language and dataset visible.

What you need

Choose tasks and languages that reflect how the model will be used. Document any dataset access requirements so another reviewer can reproduce the selection.

What to try first

Run several tasks that resemble your workload and inspect individual scores alongside any aggregate. Include one small private relevance set for application-specific validation.

Read the task's metric before comparing scores. Strong classification performance says little about whether an embedding works for a multilingual help center.

Where to look for failures

Leaderboard averages can hide language and domain gaps. Compare the same task revisions, subsets and metrics before drawing conclusions.

Related Franklin pages

Separate encoder capability, retrieval quality and privacy assumptions. Each needs a different test, even when all three experiments start from the same set of embeddings.

• Explore methods that reconstruct text from embeddings and test privacy assumptions. vec2text for studying embedding inversion
• Study an encoder architecture for tasks such as classification, retrieval and token labeling. ModernBERT for encoder-based NLP

For the role of the encoder in a search stack, read Franklin’s model guide. Embedding models

Project documentation

• Current project README
• README at the reviewed revision