flaubert lysandre HF staff commited on
Commit
5b5cf0a
1 Parent(s): 56ea0bf

Updates incorrect tokenizer configuration file (#2)

Browse files

- Adds tokenizer_config.json file (37008fbc353e275d28329aee43f82f0914dc176f)


Co-authored-by: Lysandre <lysandre@users.noreply.huggingface.co>

Files changed (1) hide show
  1. tokenizer_config.json +1 -4
tokenizer_config.json CHANGED
@@ -1,4 +1 @@
1
- {
2
- "do_lowercase": true,
3
- "do_lower_case": true
4
- }
 
1
+ {"do_lowercase": true, "do_lower_case": true, "model_max_length": 512}