deepseek-ai

/

deepseek-coder-33b-instruct

like 431

Hi, I'm new to DeepSeek Coder. I am curious about the setting in the tokenizer.json. Since we set the "<EOT> " token as eos token in the tokenizer_config.json file. But why we set the "is_special" attribute of the <EOT> token as False?
If my understanding is correct, the token won't be skipped even if we set the "skip_special_token" as True when doing inference right?