AI GitHub
Haystack for component-based retrieval pipelines
Wire document stores, retrievers, rankers and generators into explicit pipelines.
Key Takeaways
- Haystack components declare inputs and outputs and can be connected into a graph.
- Start with a document store and a retriever whose inputs you can supply independently.
- Build a retrieval-only pipeline before adding generation.
- External integrations can have different document formats and settings.
How it works
Haystack components declare inputs and outputs and can be connected into a graph. Retrieval and generation can be exercised independently using the same pipeline interfaces.
What you need
Start with a document store and a retriever whose inputs you can supply independently. A generator can be added after a test shows that the relevant passages are retrieved.
What to try first
Build a retrieval-only pipeline before adding generation. Feed it a question with two close matches and inspect the candidates and scores at each step.
Return stage outputs from a test pipeline. A fixed document fixture makes changes in retrieval or ranking visible even if the final answer happens to stay the same.
Where to look for failures
External integrations can have different document formats and settings. Keep track of embedding dimensions and source metadata across component boundaries.
Related Franklin pages
Test conversion, indexing and retrieval independently before connecting a generator. Keep an original page reference on each result so you can debug a bad answer.
- Extract document structure, reading order and tables for search or analysis. Docling for structured document conversion
- Build ingestion, indexing and query workflows around external data. LlamaIndex for connecting documents to model queries
For the full path from source documents to generated answers, start with Franklin’s glossary. Retrieval-augmented generation
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗