Back to directory

Generic Metadata Schema for Data Solution Automation

A JSON Schema and C# object model for an exchange format that describes the metadata needed to generate data warehouse transformation logic, so different data-automation tools can share the same source-to-target and physical-model metadata.

Generic Metadata Schema for Data Solution Automation

This project defines an exchange format for data logistics generation metadata that supports Data Solution Automation: generating the transformation logic for a data warehouse from a shared description rather than a tool-specific one. The repository holds the JSON Schema definition for that format, a C# class library for deserializing and validating files against it, C# examples using the interface, and a regression test project.

The metadata a conforming file carries includes source-to-target mappings, physical model metadata such as columns, tables and data types, and connectivity information or a proxy for it, kept text-based so it works under version control.

The intent is to separate the metadata format from the user interface and management layer built on top of it, so different vendors' tools, including a dedicated UI with templates or an adapter exposing the same data as JSON or a database view, can all read and write the same underlying metadata.

More in Data engineering

See all 98
  • ppgrid

    mrzk.io

    An open-source tool that turns tens of millions of point-data rows into a smooth, adaptive-resolution raster in seconds, using a push-pull mipmap approximation of inverse distance weighting that stays fast as the dataset grows.

  • Scope Lineage

    github.com

    An offline static analyzer that turns Spark and Hive SQL into structured, versioned column-lineage and diagnostics files, so AI agents and knowledge bases can reason from addressable, checkable facts instead of guessing from raw SQL.

  • A pair of Colab notebooks demonstrating a retrieval-augmented generation workflow for extracting structured datasets from scientific paper abstracts, applied here to hydrogen-storage properties of metal alloys and hydrides.