Vision-language model
Takes images and text together, so it can describe pictures, read documents and answer questions about what it sees.
What it is
A vision-language model (VLM) combines an image encoder with a language model. You can give it a photo, chart, screenshot or scanned page plus a question, and it answers in text. Most modern chat assistants are VLMs under the hood.
How it works
An image encoder, often a vision transformer, turns the picture into a sequence of embeddings. A small projection layer maps those into the language model’s token space, so the LLM reads image patches like words. Training on image–caption pairs and visual question answering teaches the connection.
What it is good for
- Describing images and alt text
- Reading charts, receipts and documents
- Answering questions about screenshots
- Agents that operate a computer screen
Limits to know
- Can misread small text or count objects wrongly
- May describe things that are not in the image
- High-resolution images use many tokens
