quantumaikr

/

llama-2-ko-7b-chat-hf-4bit

like 5

Hi! I was wondering how did you obtain tokenizer.model file? In other available korean base llama models like beomi/llama-2-ko-7b, they used FastTokenizer provided by HF tokenizers not the sentencepiece package, so the output doesn't contain tokenizer.model. Can you share how you obtained tokenizer.model file from tokenizer.json?
Thank you for your help!