Contrastive image-text model
Maps images and text into the same space, so a sentence can find matching pictures and vice versa.
What it is
Contrastive image-text models, best known through CLIP, learn a shared embedding space for pictures and captions. They do not generate anything; instead they score how well an image and a piece of text match, which powers image search, zero-shot classification and the text guidance inside many image generators.
How it works
An image encoder and a text encoder are trained together on millions of image–caption pairs. For each batch, the model pulls matching pairs close together and pushes mismatched pairs apart. Afterwards, comparing embeddings with cosine similarity tells you how related an image and a sentence are.
What it is good for
- Searching photos with natural language
- Zero-shot image classification
- Filtering or tagging large image sets
- Guiding text-to-image models
Limits to know
- Weak at counting, spatial relations and fine detail
- Inherits biases from web captions
- Gives a similarity score, not an explanation
