alvinwatner commited on
Commit
07fc755
1 Parent(s): eb84e5f

change endoftext value to len of tokenizer vocab size

Browse files
Files changed (1) hide show
  1. added_tokens.json +1 -1
added_tokens.json CHANGED
@@ -1 +1 @@
1
- {"<|endoftext|>": 50265}
 
1
+ {"<|endoftext|>": 50257}