tokenizers v1: encode, decode and scaling, measured

The new tokenizers v1 introduces a refactored architecture centered on bitstream processing instead of regular expressions, significantly boosting performance for encoding and decoding tasks. The update replaces regex-based pre-tokenization with hand-written SIMD-optimized functions tailored to specific byte-level patterns, reducing CPU overhead and enabling parallel byte processing. Benchmarks show single-threaded and multi-threaded gains, with performance improvements varying by model and language due to differences in grammar coverage.

A thread-local cache stores previously processed pre-tokens, allowing repeated words to skip the merge loop entirely and reducing redundant computations. This optimization particularly benefits inputs with high repetition, such as long documents or datasets with recurring phrases, while offering minimal gains for inputs with low repetition. The cache’s effectiveness grows as input size increases, as repeated words constitute a larger share of the total text.

The merge loop, a core component of byte pair encoding (BPE) tokenizers, has been optimized to eliminate repeated memory allocations and priority queue rebuilds. The new implementation reuses a caller-owned scratch buffer, stores symbols in a flat array, and packs merge candidates into 64-bit integers for faster comparisons. Batch processing of pre-tokens further reduces overhead, with the loop finding the next merge without branching.

Hugging Face’s tokenizers v1 maintains full backward compatibility, producing identical token IDs to v0.23 while supporting all tokenizer families (BPE, WordPiece, Unigram) and their vocabularies. The library remains general-purpose, loading all models previously supported by v0.23. Benchmarks, available via the tokbench repository, compare latency, throughput, memory usage, and crate size across hardware, with results reproducible using provided commands.