# Scope Lineage

An offline static analyzer that turns Spark and Hive SQL into structured, versioned column-lineage and diagnostics files, so AI agents and knowledge bases can reason from addressable, checkable facts instead of guessing from raw SQL.

Scope Lineage is an offline static analyzer for Spark and Hive SQL that produces versioned lineage and diagnostics artifacts, preserving common table expressions, subqueries, field expressions, joins, filters, aggregates, windows and uncertainty rather than collapsing a query down to a flat table-to-table edge. It needs no Spark cluster, database credentials or large-language model to run, and this repository is the open-source core: SQL and task ingestion, scope parsing, column-level lineage and diagnostics.

The goal is to be explicit about which parts of a lineage claim the analyzer can prove and which parts it cannot, for example correctly identifying a literal value used as a column alias rather than mistaking it for a real column, which the project shows other lineage tools getting wrong on the same example query. Embeddings, knowledge-graph storage and generating business-level semantics from the extracted facts are treated as separate, downstream concerns outside this core.

Website: https://github.com/realyin/scope-lineage