Pathway Live Data Framework
Pathway is a Python ETL framework for stream processing, real-time analytics and LLM or RAG pipelines, where the same code runs for local development, batch jobs and live data streams.
It is powered by a Rust engine based on differential dataflow that performs incremental computation, so Python code gets multithreaded, multiprocessing and distributed execution underneath, and the whole pipeline can be kept in memory and deployed with Docker and Kubernetes. Connectors reach sources such as Kafka, Google Drive, PostgreSQL and SharePoint, plus, through an Airbyte connector, several hundred more, with a documented interface for writing a custom one.
Built-in stateful transformations cover joins, windowing and sorting alongside ordinary Python functions, and dedicated LLM tooling supports runnable templates for real-time RAG, an unstructured-to-SQL pipeline for financial documents, and an adaptive-RAG technique aimed at lowering token cost while keeping accuracy. Persistence lets a running computation restart from where it left off.
Rust engine, Python API.