Text Embeddings Inference for serving embedding models

How it works

The server loads supported models and batches requests for inference. Deployment configuration controls model selection and supported hardware paths.

What you need

Choose an image or build for the supported device and model. Provide model access explicitly if artifacts need authorization, and check how the service reports rejected inputs.

What to try first

Deploy one checkpoint and compare returned embeddings with a reference implementation. Measure latency on a mix of short and long inputs.

Send mixed-length requests concurrently and inspect tail latency. A convenient per-item timing can hide the cost of batching inputs of very different lengths.

Where to look for failures

Tokenization and truncation need to match the indexing pipeline. A silent change to a model or preprocessing can make existing vectors incompatible.

Related Franklin pages

Check whether the task needs a library inside the app or a separate model endpoint. Preserve query formatting, checkpoint and vector dimensions when comparing either deployment.

• Generate embeddings in an application using supported ONNX-backed models. FastEmbed for lightweight local embeddings
• Evaluate and use embedding and reranking models from the BGE family. FlagEmbedding for BGE retrieval models

For the role of the encoder in a search stack, read Franklin’s model guide. Embedding models

Project documentation

• Current project README
• README at the reviewed revision