Explicitly set model_max_length in the tokenizer config. 98c2170 Milan Straka commited on Sep 30, 2024
Add a link to a post describing the non-standard RobeCzech tokenizer. 8a75d1f foxik commited on Jul 20, 2023