RWKV
An RNN architecture that reaches transformer-level language-model performance while training in parallel like a transformer, with linear time, constant memory, no key-value cache and unlimited context length.
RWKV is a recurrent neural network that reaches the performance of transformer-based large language models while remaining parallelizable during training, combining properties of RNNs and transformers: strong performance, linear-time computation, constant memory use with no key-value cache, fast training, effectively unlimited context length, and embeddings that fall out of the model for free.
The current generation, RWKV-7 ("Goose"), is a fully attention-free, 100 percent RNN architecture suited to language and multimodal models; it is also described as a meta-in-context learner, since it trains its own internal state on the current context through in-context gradient descent at every token, giving it a particularly small state size for efficient large-batch inference.
RWKV is maintained as a Linux Foundation AI project and is fully free to use. Pretrained weights, GGUF conversions and efficient inference implementations are published separately, along with a runtime already integrated into Windows and Office.