Data engineering

Sycamore is an open-source document processing engine that partitions, enriches and chunks unstructured documents such as PDFs and presentations for search, retrieval-augmented generation and analytics, using an AI model trained specifically for document layout.

• RAG
• Data engineering

dbt MCP Server is a Model Context Protocol server that gives AI agents tools to run and explain SQL, query the dbt Semantic Layer, and browse a dbt project's models, sources, macros, exposures and lineage across dbt Core, dbt Fusion and dbt Platform.

• MCP servers
• Data engineering

A Julia framework that automatically parallelizes code across CPU cores, GPUs, and multiple machines by scheduling tasks as a directed-acyclic graph, without manual coordination.

• Libraries
• Data engineering

A curated collection of Agent Skills for working with dbt, helping an AI coding agent build and modify models, manage a semantic layer, work across multi-project setups and troubleshoot job failures without user-invoked commands.

• Agent skills
• Data engineering

An LLM-driven data analyst that runs in a persistent Python kernel, writing and executing its own code one cell at a time and re-running its cited results in a fresh kernel before presenting a report.

• Data engineering

An open-source, AI-powered harness for data engineering that gives a coding agent more than 100 deterministic tools for SQL analysis, column-level lineage, dbt and warehouse connectivity across ten cloud warehouses.

• Databases
• Data engineering

A plugin marketplace of Power BI and Microsoft Fabric skills, subagents and hooks that teach Claude Code and GitHub Copilot to work with semantic models, DAX, TMDL and reports.

• Agent skills
• Data engineering

A local-first document redaction workbench that finds and anonymizes sensitive information in PDFs, scans, Office files, images and text using local LLMs and vision-language models.

• Security
• Data engineering

An MCP server that lets an AI agent connect to and manage Jupyter notebooks in real time, running against a local Jupyter install, JupyterHub, or a scalable hosted code sandbox from local to the cloud.

• MCP servers
• Data engineering

A self-improving context layer that teaches Claude Code, Codex and similar agents to query a company's data warehouse accurately, using approved metric definitions instead of invented SQL.

• Agent memory
• Data engineering

An extensible, parallel Python implementation of t-SNE, a dimensionality-reduction algorithm, with support for adding new points to an existing embedding and scaling to millions of data points.

• Libraries
• Data engineering

A Python library that combines a multimodal database, computed-column orchestration and HTTP serving into a single file, so a coding agent can write and review a whole data application at once.

• Databases
• Data engineering

A Python library from Stanford and Berkeley that adds LLM-based map, reduce and filter operators to bulk-process large datasets with agents, optimizing for accuracy and cost.

• Data engineering

A Python library for bulk inference and scalable synthetic-data curation, for preparing post-training datasets and extracting structured data from LLM outputs, with a viewer for monitoring generation.

• Models
• Data engineering

A GPU-accelerated data curation toolkit that builds repeatable pipelines to load, filter, deduplicate and transform large text, image, video and audio datasets for AI training, running the same pipeline on a laptop or a multi-node Ray cluster.

• Models
• Data engineering

A multi-agent AI platform for data analysis and research that automates hypothesis generation, data cleaning, visualization and report writing using LangChain, OpenAI models and LangGraph.

• Agent frameworks
• Data engineering

A conversational data-analysis system that turns natural-language questions into validated SQL, runs them against a database, and renders the results as charts, using retrieval-augmented generation and a multi-agent pipeline.

• RAG
• Data engineering

An AutoML library in Scala that runs on Apache Spark, automating feature engineering and model selection to build strongly typed, production machine learning pipelines with close to hand-tuned accuracy.

• Models
• Data engineering

An open-source, cloud-native, one-stop AI platform covering traditional machine learning, deep learning and large-model workflows, from multi-tenant compute scheduling through pipeline orchestration to model deployment and an LLMOps agent layer.

• Data engineering
• Hosting

A conversational business data-analysis agent that lets a non-technical user upload a spreadsheet or connect a database, ask a question in plain language, and get back generated SQL, a chart and a written insight instead of writing queries by hand.

• Data engineering

A language and runtime that improves the performance of data-analytics workflows by optimizing computation across multiple libraries and frameworks together.

• Data engineering

A portable data and AI engine, written in Rust, that adds a real-time analytics node to an operational database and serves SQL query, search and LLM inference to applications and agents from one binary.

• Databases
• Data engineering

A system for processing large collections of structured and unstructured data with LLMs, letting users describe operations in natural language while it orchestrates, optimizes and parallelizes the pipeline.

• Data engineering

An open-source natural language processing library built on Apache Spark, offering pretrained pipelines and models for tasks like tokenization, named entity recognition, sentiment analysis, translation and question answering.

• Libraries
• Data engineering