danielhanchen commited on
Commit
ed571f9
1 Parent(s): a0181ca

Enable LlamaTokenizerFast and AutoTokenizer to load in seconds rather than 5 minutes.

Browse files

Same procedure as last time converting Tokenizer to support HF's AutoTokenizer. See https://huggingface.co/danielhanchen/open_llama_3b_600bt_preview for details.
Ie:
```
model_name = "openlm-research/open_llama_7b"
tokenizer = AutoTokenizer.from_pretrained(model_name, pad_token = "</s>")
tokenizer.push_to_hub("danielhanchen/open_llama_7b")
```

Files changed (1) hide show
  1. tokenizer.json +0 -0
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff