Submit a resource

Navigation

Back to directory

tokenizers

A fast tokenization library implemented in Rust, with Python and Node.js bindings, aimed at becoming a standard tokenization engine across languages, models and hardware.

Visit website (opens in a new tab)github.com
tokenizers

tokenizers is a tokenization library for natural-language and language-model pipelines, aimed at being a standard tokenization engine across languages, models and hardware. It is currently moving from its 0.23 line to a 1.0.0 release candidate, which does not yet ship every feature of the earlier version.

The Rust core is the reference implementation, with full Python and Node.js bindings; C, C++, Java and Go bindings are planned. Work toward 1.0 includes bringing back training support, a batch-encoding API with zero-copy views, a dlpack API for interop with PyTorch, JAX and NumPy, and editable pipeline components.

It installs as a Python package and is licensed under Apache-2.0.

More in Libraries

See all 292

  • OpenDecision

    deepanwadhwa.github.io

    An open-source Python package that answers typed questions about application state or documents using a local natural-language-inference model, returning structured choice, score, yes/no or evidence-relation values instead of free text.

  • TradeCanvas

    bonguynvan.github.io

    A zero-dependency canvas charting engine for building TradingView-style trading charts in JavaScript, with 66 technical indicators, 24 drawing tools, 17 chart types, and real-time streaming.

  • 3d retro graphics

    3d-retro.com

    A free, CC0-licensed library of copy-paste retro 3D and demoscene graphics effects, such as plasma, raytracing, and voxel terrain, runnable as WebGL 1 or plain 2D canvas.

Cookies

We use necessary cookies to run the site. Measurement and marketing are optional; neither is used today. Privacy policy