py-tiktoken-0.13.0

fast BPE tokenizer for OpenAI models

Index | textproc

Description

tiktoken is a fast BPE (byte pair encoding) tokenizer for use with
OpenAI's models.

Run Dependencies

• devel/py-regex
• www/py-requests