AI GitHub
LanceDB for retrieval over columnar data
Query embeddings and associated records in tables based on the Lance format.
Key Takeaways
- LanceDB exposes vector and other search modes over a columnar table.
- Start with a local table and specify the embedding column and search inputs.
- Load a small table containing text, embeddings and document metadata.
- Distance metrics must match the embedding model.
How it works
LanceDB exposes vector and other search modes over a columnar table. Its data and index layout are useful when the same dataset also feeds analysis or ingestion jobs.
What you need
Start with a local table and specify the embedding column and search inputs. Keep the table's source IDs available so results can be inspected outside the search API.
What to try first
Load a small table containing text, embeddings and document metadata. Compare vector and hybrid queries and confirm that updates appear in results as expected.
Add a row update and a deletion to your benchmark. A static corpus can hide costs that are important for live catalogs.
Where to look for failures
Distance metrics must match the embedding model. Test index coverage and update behavior before treating a prototype as a continuously updated search service.
Related Franklin pages
Separate the search interface from where the data lives. A small import, query, update and delete test reveals important differences between client, embedded and server deployments.
- Search embeddings alongside structured metadata in a dedicated database. Qdrant for filtered vector retrieval
- Connect Python applications to turbopuffer's hosted search API. turbopuffer Python client for hosted retrieval
For the role of the index in nearest-neighbor search, see Franklin’s glossary. Vector indexes
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗