# Pali3

A Python implementation of the PALI-3 vision-language model, which encodes images with a contrastively pretrained SigLIP vision transformer and generates text through a UL2 encoder-decoder transformer.

Pali3 is an implementation of PALI-3, a vision-language model described in its accompanying paper as smaller, faster and stronger than earlier versions. Images are encoded into visual tokens by a contrastively pretrained SigLIP vision transformer, and those tokens are passed along with a text prompt into a 3B-parameter UL2 encoder-decoder transformer that produces the answer text.

Training follows several stages: unimodal pretraining of the vision and text encoder-decoder separately, multimodal training with the image encoder frozen, a resolution-increase fine-tuning stage, and finally task-specific fine-tuning with the vision encoder frozen.

It installs with pip and exposes a small Python API that takes an image tensor, a prompt and an output-text tensor and returns a result.

Website: https://github.com/kyegomez/PALI3