py-tokenizers-0.23.1

tokenizers with a focus on performance

Index | textproc

Description

An implementation of today's most used tokenizers, with a focus on
performance.

Build Dependencies

• textproc/oniguruma