elixir-nx/tokenizers

This is stars and forks stats for /elixir-nx/tokenizers repository. As of 25 Apr, 2024 this repository has 80 stars and 13 forks.

Tokenizers Elixir bindings for Hugging Face Tokenizers. Installation You can add :tokenizers as dependency in your mix.exs: def deps do [ {:tokenizers, "~> 0.3.0"}, ] end If you are using Livebook or IEx, you can instead run: Mix.install([ {:tokenizers, "~> 0.3.0"}, ]) Example You can use any pre-trained tokenizer from any model repo on Hugging Face Hub, such as bert-base-cased. {:ok, tokenizer} = Tokenizers.Tokenizer.from_pretrained("bert-base-cased") {:ok, encoding} = Tokenizers.Tokenizer.encode(tokenizer,...