Francois Vieille commited on
Commit
44384d6
1 Parent(s): 6a436a2

add tokenizer.json with sep and hl toks

Browse files
Files changed (2) hide show
  1. added_tokens.json +1 -1
  2. tokenizer.json +0 -0
added_tokens.json CHANGED
@@ -1 +1 @@
1
- {"<sep>": 50002, "<hl>": 50003}
 
1
+ {"<hl>": 50003, "<sep>": 50002}
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff