turing-usp commited on
Commit
d1309b4
1 Parent(s): efdc39a

add tokenizer

Browse files
pytorch_model.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:024a9cd21f73c8a91e9471e00166c3abe50c434015bc145eb057ed3db0951eb7
3
- size 435771121
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e98e485ad386d7a463965f5dff38e7c8f3efb5da7ca3eea64e2b7c054c016289
3
+ size 200261120
special_tokens_map.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {}
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"tokenizer_class": "PreTrainedTokenizerFast"}