openlm-research

/

open_llama_3b_v2

like 129

Adding tokenizer.json for easier consumption. The issue with fast tokenizers referenced in the README is fixed now so we should be good to include this file.

If this change sounds good, I can also generate the tokenizer.json for the 7B model as well.

Generated with:

#!/usr/bin/env python3

import os
from transformers import AutoTokenizer

path = os.path.join(os.getcwd(),"open_llama_3b_v2")

tokenizer = AutoTokenizer.from_pretrained(path)
tokenizer.save_pretrained(path)