Data engineering

An open-source tool that turns tens of millions of point-data rows into a smooth, adaptive-resolution raster in seconds, using a push-pull mipmap approximation of inverse distance weighting that stays fast as the dataset grows.

• Data engineering

An offline static analyzer that turns Spark and Hive SQL into structured, versioned column-lineage and diagnostics files, so AI agents and knowledge bases can reason from addressable, checkable facts instead of guessing from raw SQL.

• Data engineering

A pair of Colab notebooks demonstrating a retrieval-augmented generation workflow for extracting structured datasets from scientific paper abstracts, applied here to hydrogen-storage properties of metal alloys and hydrides.

• RAG
• Data engineering

A multimodal semantic search pipeline that ingests documents and images through Apache Airflow DAGs, extracts text and generates image descriptions, embeds the content, and serves cosine-similarity search over a Qdrant vector index with PostgreSQL for metadata.

• RAG
• Data engineering

An Apache Flink connector for the Lance columnar data format that provides source and sink support, Flink SQL and Table API integration, KNN vector search, and index building for Lance vector datasets.

• Databases
• Data engineering

A self-hosted platform where an AI operator builds, deploys, runs, debugs and reruns data pipelines against a team's existing databases, warehouses and APIs.

• Data engineering
• Workflow automation

A fast Rust command-line tool that pulls in data from local files, git repositories, websites, Hugging Face datasets, and file hosts, cleans and classifies it, and builds deduplicated, leakage-safe chat-format JSONL datasets for training and retrieval.

• RAG
• Data engineering

An example Apache Airflow project that orchestrates a daily ETL pipeline fetching weather data from a free API, cleaning and transforming it, and storing it in PostgreSQL, ready for downstream machine learning tasks.

• Data engineering
• Tutorials

A set of Databricks scripts that build a corpus from GitHub repository content, train and serve a semantic search model on CPU, and summarize repository code with the Azure OpenAI API.

• RAG
• Data engineering

An open-source semantic layer and MCP server that sits between the AI tools a team uses and its data warehouse, answering questions in plain language with the SQL that produced the answer, a confidence grade, and a receipt.

• MCP servers
• Data engineering

A context provider for data agents that keeps metric definitions, verified example queries and interpretation notes in one knowledge base, served over MCP, REST and CLI.

• MCP servers
• Data engineering

A practical knowledge base and Python SDK for the OpenDataProducts.org standards family (ODPS, ODPC, ODPV and ODPG), with FAQs, reusable YAML examples, and SDK guides for validating, generating and automating data product specifications.

• Data engineering
• Documentation tools

A chat-driven LLMOps agent that turns a folder of documents into a fine-tuned, domain-specific language model, with human approval at each pipeline step.

• Models
• Data engineering

An open-source MCP server that lets AI agents query Databricks SQL, browse Unity Catalog, run jobs and use vector search, under per-tool policy controls.

• MCP servers
• Data engineering

End-to-end tutorials, how-to Jupyter notebooks and guides for Siemens' AI Software Development Kit, which packages, tests and creates inference pipelines for deployment on the AI Inference Server.

• Data engineering
• Tutorials

An open-source Apache Spark extension for large-scale document processing that combines OCR, LLMs, and vision-language models to extract structured data from PDFs and images at scale.

• RAG
• Data engineering

The reference Java and Spring implementation of the Open Ingestion Standard, a vendor-neutral enterprise data-integration platform that orchestrates connectors, embedding, and vector search output using modern Java concurrency features.

• RAG
• Data engineering

An open-source Python framework for building ETL pipelines that feed Retrieval-Augmented Generation systems, with multi-tenancy, cloud-native deployment and job scheduling.

• RAG
• Data engineering

Dug applies semantic web and knowledge graph techniques to biomedical research metadata, letting a researcher search underused datasets such as dbGaP by concept rather than exact keyword, based on curated ontologies.

• RAG
• Data engineering

An LLM-powered tool that turns a natural-language causal question and a dataset into a full causal-inference analysis, selecting a suitable method, running it, checking diagnostics and explaining the result in plain language.

• Agent frameworks
• Data engineering

A local-first application for curating, tagging, downloading and managing datasets and models, with built-in AI assistants for tag editing, code and dataset-pipeline work, plus 3D generation and media metadata handling.

• Generative media
• Data engineering

A Python library for declaring data transformations, such as pipes, filters, joins, group-bys and window functions, as plain Python expressions that compile at runtime into small, optimized native functions.

• Data engineering

A JSON Schema and C# object model for an exchange format that describes the metadata needed to generate data warehouse transformation logic, so different data-automation tools can share the same source-to-target and physical-model metadata.

• Data engineering

A vector-native data lakehouse file format, written in Rust and built on the Apache Iceberg specification, that stores tabular data, embeddings and an HNSW vector index together in one self-contained Parquet-extended file.

• Databases
• Data engineering