Sycamore
Sycamore is an open-source document processing engine that partitions, enriches and chunks unstructured documents such as PDFs and presentations for search, retrieval-augmented generation and analytics, using an AI model trained specifically for document layout.
Sycamore partitions and enriches a wide range of document types, including reports, presentations, transcripts and manuals, and can analyze and chunk complex PDFs and images containing tables, figures, graphs and other infographics. Document processing relies on Aryn DocParse, a GPU-powered API that segments and labels documents, performs OCR, and extracts tables and images using an open-source DETR model trained on more than 80,000 enterprise documents; the partitioner can also run locally instead of through the hosted service.
Its DocSet abstraction represents a scalable, reliable pipeline of document transforms, written with a functional-programming style in Python, that a user can customize to experiment with chunking quality for retrieval-augmented generation. Sycamore loads the resulting data into vector databases and hybrid search engines including OpenSearch, Elasticsearch, Pinecone, DuckDB, Qdrant and Weaviate, and includes automatic data crawlers for Amazon S3 and HTTP sources.
It is written in Python and licensed under Apache 2.0.