Custom tokenizer
#1
by
BoburAmirov
- opened
Is a custom tokenizer used for the issai/LLama-3.1-KazLLM-1.0-8B model or is Llama's native tokenizer used?
https://www.youtube.com/watch?v=TJV1SoH4KkA
this source talked about creating a custom tokenizer
Hello, thanks for your interest!
While we’ve been working on a model with the custom tokenizer, the currently released models still come with the native Llama tokenizer, which has provided better overall model performance so far.