Tokenizer
#23
by
vschindepalli
- opened
Can we perform RAG with this model? I was not able to find code for tokenization, please let me know
The tokenizer is included in the files depicted in this repository. It is a Llama-base vocabulary with 10 additional tokens.
gugarosa
changed discussion status to
closed