Please add default tokenizer to config.json

#6
by kmcgrath - opened

I would like to automatically reconcile the correct tokenizer for this model without having to implement custom logic in my code. Could you please add a tokenizer field to your config.json? Thanks!

Sign up or log in to comment