Update tokenizer_config.json
#7
by
abhipn
- opened
Configuring bos, eos, pad and unk tokens helps load the tokenizer successfully, otherwise getting a recursion error "RecursionError: maximum recursion depth exceeded while getting the str of an object."
Thanks!
kbressem
changed pull request status to
merged