Back to directory

convtools

A Python library for declaring data transformations, such as pipes, filters, joins, group-bys and window functions, as plain Python expressions that compile at runtime into small, optimized native functions.

Visit website (opens in a new tab)convtools.readthedocs.io
convtools

convtools lets a developer express a data transformation as a composition of expressions and then compile it, with .gen_converter(), into a real Python function, so data stays in native iterables such as lists, dicts, generators or CSV streams rather than a dataframe container. Its Table helper processes CSV-like data without loading it all into memory.

Aggregations are handled by a set of built-in reducers, exposed as c.ReduceFuncs.*, covering value reducers such as Sum, Average, CountDistinct and TopK and dictionary reducers that aggregate into per-key dictionaries; nested aggregations and custom two-argument reducers are also supported.

The generated code can be printed for inspection with a debug flag, and global options are set through an options context, which works with standard Python debuggers. It is intended as a lean, composable layer for custom transforms, used alongside pandas or Polars rather than as a replacement for them.

More in Data engineering

See all 98
  • ppgrid

    mrzk.io

    An open-source tool that turns tens of millions of point-data rows into a smooth, adaptive-resolution raster in seconds, using a push-pull mipmap approximation of inverse distance weighting that stays fast as the dataset grows.

  • Scope Lineage

    github.com

    An offline static analyzer that turns Spark and Hive SQL into structured, versioned column-lineage and diagnostics files, so AI agents and knowledge bases can reason from addressable, checkable facts instead of guessing from raw SQL.

  • A pair of Colab notebooks demonstrating a retrieval-augmented generation workflow for extracting structured datasets from scientific paper abstracts, applied here to hydrogen-storage properties of metal alloys and hydrides.