MoonViT - Pytorch
A single-file PyTorch implementation of MoonViT, the native-resolution vision encoder from Kimi-VL, that packs images of different sizes into one sequence instead of resizing or padding them.
MoonViT tokenizes each image at its own native height and width and packs images of different shapes into one long sequence, so a large screenshot and a small portrait can share a batch without resizing, distortion or padding tokens. It combines an interpolated SigLIP absolute position embedding, which preserves the pretrained prior, with two-dimensional RoPE for fine-grained, resolution-robust position information, and uses variable-length attention so that each image only attends to itself within the packed sequence.
An included MLP projector composes with the encoder to feed a language model: it applies a two-by-two pixel-shuffle that turns four tokens into one with four times the channels, then a two-layer MLP that projects into the LLM's hidden size. FlashAttention is used automatically when available on CUDA, and a block-diagonal scaled-dot-product-attention fallback runs on CPU, MPS or CUDA with no extra dependencies. It installs from PyPI or from source and is Apache-2.0 licensed.