# Mechanistic Interpretability Study: Induction Circuit Stability Under Fine-Tuning

A mechanistic interpretability study tracking whether the induction-head circuit in a small attention-only transformer survives fine-tuning from prose to structured Python code, using per-head induction scores and activation patching across training checkpoints.

This repository holds a mechanistic interpretability study asking whether the induction head circuit in a 2-layer, attention-only transformer survives fine-tuning on a narrow distribution or changes, and in which direction. It measures per-head induction scores and activation-patching attribution at every training checkpoint, across three random seeds, for both a Python code fine-tuning run and a TinyStories prose control.

The headline result is that the circuit does not dissolve under fine-tuning; it strengthens under both conditions, and significantly more so under code than under prose, though the authors note the finding comes from a minimal model and may not generalize to larger or frontier-scale transformers. An interactive dashboard on Hugging Face Spaces lets a reader toggle pre- and post-fine-tuning states and inspect the attention pattern on their own input text.

Experiment results and paper figures are committed to the repository rather than gitignored, since the output is small enough that committing it lets anyone inspect the exact numbers behind each figure without re-running anything.

Website: https://github.com/Mattral/Mechanistic-Interpretability-Study-Induction-Circuit-Stability-Under-Fine-Tuning