File size: 387 Bytes
470a654 08838e9 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
---
license: apache-2.0
---
### How to use
###
```
from transformers import LlamaTokenizerFast
tokenizer = LlamaTokenizerFast.from_pretrained("mimir-project/tokenizer", token=True)
```
or
```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("mimir-project/tokenizer", token=True)
```
Copied from https://github.com/SmartmediaAI/MIMIR-project/tree/main
|