This is stars and forks stats for /openai/tiktoken repository. As of 07 Dec, 2023 this repository has 6929 stars and 481 forks.
⏳ tiktoken tiktoken is a fast BPE tokeniser for use with OpenAI's models. import tiktoken enc = tiktoken.get_encoding("cl100k_base") assert enc.decode(enc.encode("hello world")) == "hello world" # To get the tokeniser corresponding to a specific model in the OpenAI API: enc = tiktoken.encoding_for_model("gpt-4") The open source version of tiktoken can be installed from PyPI: pip install tiktoken The tokeniser API is documented in tiktoken/core.py. Example code using tiktoken can be found in the OpenAI...
⏳ tiktoken tiktoken is a fast BPE tokeniser for use with OpenAI's models. import tiktoken enc = tiktoken.get_encoding("cl100k_base") assert enc.decode(enc.encode("hello world")) == "hello world" # To get the tokeniser corresponding to a specific model in the OpenAI API: enc = tiktoken.encoding_for_model("gpt-4") The open source version of tiktoken can be installed from PyPI: pip install tiktoken The tokeniser API is documented in tiktoken/core.py. Example code using tiktoken can be found in the OpenAI...
repo | techs | stars | weekly | forks | weekly |
---|---|---|---|---|---|
cloudfoundry/bosh-vsphere-cpi-release | RubyPythonMustache | 30 | 0 | 36 | 0 |
google/forma | RustHTML | 2.5k | 0 | 49 | 0 |
linebender/vello | RustWGSL | 1.5k | 0 | 75 | 0 |
kanashter/bothermione-ao3wrapped | SveltePythonJavaScript | 18 | 0 | 8 | 0 |
damus-io/damus | CSwiftPython | 1.9k | 0 | 467 | 0 |
AllenDowney/ThinkBayes | TeXPythonOther | 1.6k | 0 | 1.9k | 0 |
Cjkkkk/CUDA_gemm | CudaPythonC++ | 187 | +2 | 23 | 0 |
FLAMEGPU/FLAMEGPU2 | CudaC++Python | 72 | 0 | 14 | 0 |
resemble-ai/monotonic_align | CythonPython | 53 | 0 | 5 | 0 |
elixir-nx/tokenizers | ElixirRustNix | 80 | 0 | 13 | 0 |