Back to directory

Flyte

A Kubernetes-native orchestration platform for reliably running ML pipelines, models and AI agents at scale, defined entirely in Python.

Flyte

Flyte is an open-source platform, graduated by the LF AI & Data Foundation, for reliably orchestrating machine-learning pipelines, models and agents at scale, defined in pure Python. Flyte 2, its current major version, installs with a single pip command and includes a CLI for running and serving workflows, plus a TUI for a richer local development experience.

A workflow can be run locally as a plain Python script or dispatched through the Flyte CLI, and a model can similarly be served locally or through the CLI's serve command. The open-source, Kubernetes-native backend infrastructure for deploying Flyte 2 as a distributed, multi-node service is still in development; an enterprise-ready, production-grade backend is available today through Union.ai.

The project provides a DevBox for a ten-minute introduction, runnable locally or in GitHub Codespaces, alongside SDK and CLI reference documentation. It is released under the Apache 2.0 license.

More in Data engineering

See all 98
  • ppgrid

    mrzk.io

    An open-source tool that turns tens of millions of point-data rows into a smooth, adaptive-resolution raster in seconds, using a push-pull mipmap approximation of inverse distance weighting that stays fast as the dataset grows.

  • Scope Lineage

    github.com

    An offline static analyzer that turns Spark and Hive SQL into structured, versioned column-lineage and diagnostics files, so AI agents and knowledge bases can reason from addressable, checkable facts instead of guessing from raw SQL.

  • A pair of Colab notebooks demonstrating a retrieval-augmented generation workflow for extracting structured datasets from scientific paper abstracts, applied here to hydrogen-storage properties of metal alloys and hydrides.