Rose
A stateless PyTorch optimizer that normalizes gradients by their per-slice dynamic range instead of maintaining running moment statistics like Adam or RMSprop.
Rose, short for Range-Of-Slice Equilibration, is a PyTorch optimizer that estimates gradient scale directly from the current gradient, rather than accumulating historical first- and second-moment statistics the way Adam and RMSprop do. At each step it normalizes every gradient tensor by a per-slice range, yielding one adaptive scale factor per output unit.
An optional coefficient-of-variation trust gate blends per-slice ranges with their global mean when the ranges are noisy, and an optional gradient centralization step removes shared directional bias before scaling. Because it keeps no momentum, variance estimate or step counter, its memory footprint is just the parameters, gradients and working memory.
It installs with pip, requires Python 3.10 or newer and PyTorch 2.0 or newer, and is released under the Apache-2.0 license.