Pathway Live Data Framework AI Pipelines
A set of ready-to-run templates built on the Pathway framework for retrieval-augmented generation, enterprise document search and other live-data AI pipelines, deployable to the cloud or on premises.
Templates stay synced with sources such as a file system, Google Drive, SharePoint, S3, Kafka and PostgreSQL, reflecting new, changed and deleted documents in real time, with built-in indexing for vector, hybrid and full-text search performed in memory with caching, and no separate infrastructure to stand up. Each template scales to millions of pages and can be swapped between a simpler and a more accuracy-optimized pipeline, or have its data source or index type changed with a small edit.
Included templates cover a basic question-answering RAG app, a live vector-store or retriever service usable as a LangChain or LlamaIndex backend, a multimodal RAG pipeline that uses GPT-4o to parse PDFs including charts and tables, an unstructured-to-SQL pipeline for financial documents, and an Adaptive RAG technique aimed at cutting token cost while keeping accuracy.
Jupyter Notebook, MIT licensed.