Why we do not set the <EOT> token as a special token?
#28
by
YalunHu
- opened
Hi, I'm new to DeepSeek Coder. I am curious about the setting in the tokenizer.json. Since we set the "<EOT> " token as eos token in the tokenizer_config.json file. But why we set the "is_special" attribute of the <EOT> token as False?
If my understanding is correct, the token won't be skipped even if we set the "skip_special_token" as True when doing inference right?