ocisd4
/

openllama_tokenizer_ext_zh

Model card Files Files and versions Community

samleeasus commited on Jun 2, 2023

Commit

627d437

•

1 Parent(s): e119f9b

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -5,7 +5,7 @@ tokenizer = LlamaTokenizer.from_pretrained(
         'ocisd4/openllama_tokenizer_ext_zh',
         pad_token="<pad>",
         add_bos_token=True,
-        add_eos_token=True,
         use_auth_token='True',
 )

         'ocisd4/openllama_tokenizer_ext_zh',
         pad_token="<pad>",
         add_bos_token=True,
+        add_eos_token=False,
         use_auth_token='True',
 )