AI GitHub
Unstructured for partitioning source documents
Turn documents into typed elements that a retrieval pipeline can filter and chunk.
Key Takeaways
- Partitioning identifies elements such as headings and narrative text.
- Decide whether the application needs raw partitioned elements or already assembled chunks.
- Partition a multi-page document, inspect element types and compare two chunk sizes on known queries.
- Strategies for text extraction and image-heavy pages have different dependencies and costs.
How it works
Partitioning identifies elements such as headings and narrative text. Chunking can use those boundaries instead of treating the whole file as one block of text.
What you need
Decide whether the application needs raw partitioned elements or already assembled chunks. Keep metadata on elements so downstream consumers can recover the original location.
What to try first
Partition a multi-page document, inspect element types and compare two chunk sizes on known queries. Keep the page metadata with the test output.
Keep page boundaries on a retrieval test that uses repeated headers. It helps reveal whether headers dominate results or source references survive chunking.
Where to look for failures
Strategies for text extraction and image-heavy pages have different dependencies and costs. Check what your chosen strategy returns for scans and tables.
Related Franklin pages
Compare converted output on a few documents with different structures. Choose according to the information downstream consumers need to recover, such as tables, source locations or heading boundaries.
- Convert supported office and media files into text-oriented Markdown for downstream AI tools. MarkItDown for converting files into Markdown
- Extract text and metadata from supported files through a native library and language bindings. Xberg for document extraction across languages
For how recognizing characters fits into the larger document processing workflow, see Franklin’s glossary. Optical character recognition
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗