Why legacy tokenizer?
#1
by
yuyijiong
- opened
为什么tokenizer的legacy=True？
这样会导致警告 You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at https://github.com/huggingface/transformers/pull/24565