Encoder model (BERT-style)
Reads a whole text at once to understand it, ideal for classification, tagging and search rather than writing.
What it is
Encoder models such as BERT read text in both directions and produce a representation of its meaning instead of generating new text. They remain the workhorse for classification, named-entity recognition, spam and sentiment detection, and as the base of embedding models, because they are small, fast and cheap.
How it works
Training hides random words and asks the model to fill them in using context on both sides (masked language modeling). A small task-specific head is then fine-tuned on top, for example to output a label for each sentence or each word.
What it is good for
- Text classification and moderation
- Named-entity recognition
- The backbone of embedding models
- High-volume, low-latency text processing
Limits to know
- Does not generate fluent text
- Usually needs labeled data to fine-tune
- Shorter context than modern LLMs
