x-transformers
A concise but fully-featured PyTorch transformer implementation bundling many experimental architecture variants and training techniques from published papers.
x-transformers provides encoder, decoder, and encoder-decoder transformer wrappers usable for a decoder-only GPT-like model, a BERT-like encoder, or a vision transformer for image classification through the same underlying TransformerWrapper and Encoder/Decoder building blocks, configured by arguments such as depth, number of heads, and attention head dimension. It implements Flash Attention, the fused attention technique that processes the attention matrix in tiles and recomputes on the backward pass to keep memory linear in sequence length, letting models reach longer context lengths without hitting a memory ceiling, and it composes with image-captioning setups such as a PaLI-style vision-language model.
The library also exposes configurable dropout and a large set of other experimental features drawn from individual research papers, intended to be combined and tried rather than fixed into one architecture.
It installs with pip install x-transformers and is distributed under the MIT license.