tokenizer adds default eos_token_id while build_inputs function adds eos_token too!

by suzhu001 - opened Feb 23

Feb 23

https://huggingface.co/internlm/internlm2-chat-20b/blob/3f710f76f56f8c40dc5dd800dbe66f3341cb2c87/modeling_internlm2.py#L1146

suzhu001 changed discussion title from tokenizer add default eos_token_id while build_inputs function add eos_token too! to tokenizer adds default eos_token_id while build_inputs function adds eos_token too! Feb 23

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment