tokenizer adds default eos_token_id while build_inputs function adds eos_token too!

#9
by suzhu001 - opened
suzhu001 changed discussion title from tokenizer add default eos_token_id while build_inputs function add eos_token too! to tokenizer adds default eos_token_id while build_inputs function adds eos_token too!

Sign up or log in to comment