openai/tiktoken

This is stars and forks stats for /openai/tiktoken repository. As of 03 May, 2024 this repository has 6929 stars and 481 forks.

⏳ tiktoken tiktoken is a fast BPE tokeniser for use with OpenAI's models. import tiktoken enc = tiktoken.get_encoding("cl100k_base") assert enc.decode(enc.encode("hello world")) == "hello world" # To get the tokeniser corresponding to a specific model in the OpenAI API: enc = tiktoken.encoding_for_model("gpt-4") The open source version of tiktoken can be installed from PyPI: pip install tiktoken The tokeniser API is documented in tiktoken/core.py. Example code using tiktoken can be found in the OpenAI...