# Ouroboros

A research technique that attaches a small hypernetwork to a recursive transformer block, generating a different LoRA modulation at each recurrence step instead of repeating the same transformation, improving depth-recycled models.

Ouroboros attaches a compact controller hypernetwork to a recursive transformer block, so that the controller observes the current hidden state and generates a per-step diagonal modulation vector applied to frozen, SVD-initialized LoRA bases derived from the residuals of removed layers, meaning each recurrence step performs a different, input-conditioned transformation rather than repeating the same operation. It combines this with gated recurrence, initialized to retain 88 percent of the previous hidden state for stable deep iteration, and per-step layer normalization giving each recurrence step its own normalization statistics.

Applied to a 36-layer Qwen2.5-3B model reduced to 17 layers, eight prelude, one recurrent and eight coda, plus 9.2 million trainable parameters, the controller outperforms a static-LoRA baseline at every tested recurrence depth, with the largest advantage at depth one, and eight tested configurations all converge to a similar loss, recovering just over half of the gap to the full 36-layer model. The gated recurrence is shown to be essential: without it, recursive layer reuse makes the model measurably worse, and the project notes that these gains are measured on the training distribution and do not yet transfer to held-out text, attributed to frozen downstream layers that cannot adapt to the modified hidden-state distribution.

Website: https://github.com/RightNow-AI/ouroboros