| license: apache-2.0 | |
| ### How to use | |
| ### | |
| ``` | |
| from transformers import LlamaTokenizerFast | |
| tokenizer = LlamaTokenizerFast.from_pretrained("mimir-project/tokenizer", token=True) | |
| ``` | |
| or | |
| ``` | |
| from transformers import AutoTokenizer | |
| tokenizer = AutoTokenizer.from_pretrained("mimir-project/tokenizer", token=True) | |
| ``` | |
| Copied from https://github.com/SmartmediaAI/MIMIR-project/tree/main | |