Gigatoken
A fast tokenizer for language models, written in Rust, that aims to match HuggingFace Tokenizers output while running at multi-gigabyte-per-second throughput.
Gigatoken is a tokenizer for language modeling that supports a wide range of CPU hardware and most commonly used tokenizer formats, including compatibility modes for HuggingFace Tokenizers and Tiktoken. It can be used through its own API for maximum throughput or through a compatibility mode whose output closely matches HuggingFace Tokenizers, at some cost to performance.
Its own API lets the Rust implementation read text data directly from files, skipping much of the overhead of passing data through Python, and supports loading tokenizers by HuggingFace model name. Benchmarks in the README report encoding throughput across models such as GPT-2, Phi-4, GPT-OSS, Llama and Qwen, compared against HuggingFace tokenizers and Tiktoken on a multi-socket AMD EPYC server.
It installs as a Python package and is licensed under MIT.