Where is the mixtral-8x7b's tokenizer encoder? Is there a specific repository or node module?
#180
by
RamanSB
- opened
I could only find this for Mistral: https://github.com/imoneoi/mistral-tokenizer - Doesn't have many starts, currently i'm using openai's gpt-tokenizer module, but not sure if this accurate or appropriate to use for Mixtral, I doubt it.
It's the same as for Mistral, afaik. At least I'm using the Mistral's one and it works well.