What tokenizer is used?

#2
by nickypro - opened

Hello!

I am trying to test how this performs on some datasets as a reference, but I can't seem to find what tokenizer is being used? I tried "lmsys/vicuna-7b-v1.3" but it didn't seem to work very well so was wondering which you used for this project?

Thank you!

Sign up or log in to comment