This is polish fast tokenizer.

Number of documents used to train tokenizer:

25 088 398

Sample usge with transformers:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('radlab/polish-fast-tokenizer')
tokenizer.decode(tokenizer("Ala ma kota i psa").input_ids)

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference API

Unable to determine this model’s pipeline type. Check the docs .

radlab
/

polish-fast-tokenizer

Datasets used to train radlab/polish-fast-tokenizer