tokenizer.eos_token_id is not defined. So, this example cannot run.
I recommend that we remove the line temporally.

可能ならば、OpenCALMあたりからトークンの設定ファイルを拾ってくると良いらしいです。

https://huggingface.co/cyberagent/open-calm-3b/blob/main/special_tokens_map.json

Cannot merge
This branch has merge conflicts in the following files:
  • README.md

Sign up or log in to comment