How to use erhanalsr/langusta-kpss-bpe-tokenizer with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("erhanalsr/langusta-kpss-bpe-tokenizer", device_map="auto")
Byte-level BPE tokenizer trained on the filtered Turkish KPSS reasoning corpus with <unk>, <pad>, <bos>, and <eos> special tokens.
<unk>
<pad>
<bos>
<eos>
-
