mdizak
commited on
Commit
•
786494d
1
Parent(s):
944d302
Initial
Browse files- README.md +150 -1
- bert-large-NER-rust/.gitattributes +35 -0
- bert-large-NER-rust/README.md +3 -0
- config.json +67 -0
- flax_model.msgpack +3 -0
- model.npz +3 -0
- model.safetensors +3 -0
- onnx/added_tokens.json +7 -0
- onnx/config.json +54 -0
- onnx/model.onnx +3 -0
- onnx/special_tokens_map.json +7 -0
- onnx/tokenizer.json +0 -0
- onnx/tokenizer_config.json +59 -0
- onnx/vocab.txt +0 -0
- pytorch_model.bin +3 -0
- rust_model.ot +3 -0
- special_tokens_map.json +1 -0
- tf_model.h5 +3 -0
- tokenizer_config.json +1 -0
- vocab.txt +0 -0
README.md
CHANGED
@@ -1,3 +1,152 @@
|
|
1 |
---
|
2 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
---
|
2 |
+
language: en
|
3 |
+
datasets:
|
4 |
+
- conll2003
|
5 |
+
license: mit
|
6 |
+
model-index:
|
7 |
+
- name: dslim/bert-large-NER
|
8 |
+
results:
|
9 |
+
- task:
|
10 |
+
type: token-classification
|
11 |
+
name: Token Classification
|
12 |
+
dataset:
|
13 |
+
name: conll2003
|
14 |
+
type: conll2003
|
15 |
+
config: conll2003
|
16 |
+
split: test
|
17 |
+
metrics:
|
18 |
+
- name: Accuracy
|
19 |
+
type: accuracy
|
20 |
+
value: 0.9031688753722759
|
21 |
+
verified: true
|
22 |
+
- name: Precision
|
23 |
+
type: precision
|
24 |
+
value: 0.920025068328604
|
25 |
+
verified: true
|
26 |
+
- name: Recall
|
27 |
+
type: recall
|
28 |
+
value: 0.9193688678588825
|
29 |
+
verified: true
|
30 |
+
- name: F1
|
31 |
+
type: f1
|
32 |
+
value: 0.9196968510445761
|
33 |
+
verified: true
|
34 |
+
- name: loss
|
35 |
+
type: loss
|
36 |
+
value: 0.5085050463676453
|
37 |
+
verified: true
|
38 |
---
|
39 |
+
# bert-large-NER
|
40 |
+
|
41 |
+
## Model description
|
42 |
+
|
43 |
+
**bert-large-NER** is a fine-tuned BERT model that is ready to use for **Named Entity Recognition** and achieves **state-of-the-art performance** for the NER task. It has been trained to recognize four types of entities: location (LOC), organizations (ORG), person (PER) and Miscellaneous (MISC).
|
44 |
+
|
45 |
+
Specifically, this model is a *bert-large-cased* model that was fine-tuned on the English version of the standard [CoNLL-2003 Named Entity Recognition](https://www.aclweb.org/anthology/W03-0419.pdf) dataset.
|
46 |
+
|
47 |
+
If you'd like to use a smaller BERT model fine-tuned on the same dataset, a [**bert-base-NER**](https://huggingface.co/dslim/bert-base-NER/) version is also available.
|
48 |
+
|
49 |
+
|
50 |
+
## Intended uses & limitations
|
51 |
+
|
52 |
+
#### How to use
|
53 |
+
|
54 |
+
You can use this model with Transformers *pipeline* for NER.
|
55 |
+
|
56 |
+
```python
|
57 |
+
from transformers import AutoTokenizer, AutoModelForTokenClassification
|
58 |
+
from transformers import pipeline
|
59 |
+
|
60 |
+
tokenizer = AutoTokenizer.from_pretrained("dslim/bert-large-NER")
|
61 |
+
model = AutoModelForTokenClassification.from_pretrained("dslim/bert-large-NER")
|
62 |
+
|
63 |
+
nlp = pipeline("ner", model=model, tokenizer=tokenizer)
|
64 |
+
example = "My name is Wolfgang and I live in Berlin"
|
65 |
+
|
66 |
+
ner_results = nlp(example)
|
67 |
+
print(ner_results)
|
68 |
+
```
|
69 |
+
|
70 |
+
#### Limitations and bias
|
71 |
+
|
72 |
+
This model is limited by its training dataset of entity-annotated news articles from a specific span of time. This may not generalize well for all use cases in different domains. Furthermore, the model occassionally tags subword tokens as entities and post-processing of results may be necessary to handle those cases.
|
73 |
+
|
74 |
+
## Training data
|
75 |
+
|
76 |
+
This model was fine-tuned on English version of the standard [CoNLL-2003 Named Entity Recognition](https://www.aclweb.org/anthology/W03-0419.pdf) dataset.
|
77 |
+
|
78 |
+
The training dataset distinguishes between the beginning and continuation of an entity so that if there are back-to-back entities of the same type, the model can output where the second entity begins. As in the dataset, each token will be classified as one of the following classes:
|
79 |
+
|
80 |
+
Abbreviation|Description
|
81 |
+
-|-
|
82 |
+
O|Outside of a named entity
|
83 |
+
B-MIS |Beginning of a miscellaneous entity right after another miscellaneous entity
|
84 |
+
I-MIS | Miscellaneous entity
|
85 |
+
B-PER |Beginning of a person’s name right after another person’s name
|
86 |
+
I-PER |Person’s name
|
87 |
+
B-ORG |Beginning of an organization right after another organization
|
88 |
+
I-ORG |organization
|
89 |
+
B-LOC |Beginning of a location right after another location
|
90 |
+
I-LOC |Location
|
91 |
+
|
92 |
+
|
93 |
+
### CoNLL-2003 English Dataset Statistics
|
94 |
+
This dataset was derived from the Reuters corpus which consists of Reuters news stories. You can read more about how this dataset was created in the CoNLL-2003 paper.
|
95 |
+
#### # of training examples per entity type
|
96 |
+
Dataset|LOC|MISC|ORG|PER
|
97 |
+
-|-|-|-|-
|
98 |
+
Train|7140|3438|6321|6600
|
99 |
+
Dev|1837|922|1341|1842
|
100 |
+
Test|1668|702|1661|1617
|
101 |
+
#### # of articles/sentences/tokens per dataset
|
102 |
+
Dataset |Articles |Sentences |Tokens
|
103 |
+
-|-|-|-
|
104 |
+
Train |946 |14,987 |203,621
|
105 |
+
Dev |216 |3,466 |51,362
|
106 |
+
Test |231 |3,684 |46,435
|
107 |
+
|
108 |
+
## Training procedure
|
109 |
+
|
110 |
+
This model was trained on a single NVIDIA V100 GPU with recommended hyperparameters from the [original BERT paper](https://arxiv.org/pdf/1810.04805) which trained & evaluated the model on CoNLL-2003 NER task.
|
111 |
+
|
112 |
+
## Eval results
|
113 |
+
metric|dev|test
|
114 |
+
-|-|-
|
115 |
+
f1 |95.7 |91.7
|
116 |
+
precision |95.3 |91.2
|
117 |
+
recall |96.1 |92.3
|
118 |
+
|
119 |
+
The test metrics are a little lower than the official Google BERT results which encoded document context & experimented with CRF. More on replicating the original results [here](https://github.com/google-research/bert/issues/223).
|
120 |
+
|
121 |
+
### BibTeX entry and citation info
|
122 |
+
|
123 |
+
```
|
124 |
+
@article{DBLP:journals/corr/abs-1810-04805,
|
125 |
+
author = {Jacob Devlin and
|
126 |
+
Ming{-}Wei Chang and
|
127 |
+
Kenton Lee and
|
128 |
+
Kristina Toutanova},
|
129 |
+
title = {{BERT:} Pre-training of Deep Bidirectional Transformers for Language
|
130 |
+
Understanding},
|
131 |
+
journal = {CoRR},
|
132 |
+
volume = {abs/1810.04805},
|
133 |
+
year = {2018},
|
134 |
+
url = {http://arxiv.org/abs/1810.04805},
|
135 |
+
archivePrefix = {arXiv},
|
136 |
+
eprint = {1810.04805},
|
137 |
+
timestamp = {Tue, 30 Oct 2018 20:39:56 +0100},
|
138 |
+
biburl = {https://dblp.org/rec/journals/corr/abs-1810-04805.bib},
|
139 |
+
bibsource = {dblp computer science bibliography, https://dblp.org}
|
140 |
+
}
|
141 |
+
```
|
142 |
+
```
|
143 |
+
@inproceedings{tjong-kim-sang-de-meulder-2003-introduction,
|
144 |
+
title = "Introduction to the {C}o{NLL}-2003 Shared Task: Language-Independent Named Entity Recognition",
|
145 |
+
author = "Tjong Kim Sang, Erik F. and
|
146 |
+
De Meulder, Fien",
|
147 |
+
booktitle = "Proceedings of the Seventh Conference on Natural Language Learning at {HLT}-{NAACL} 2003",
|
148 |
+
year = "2003",
|
149 |
+
url = "https://www.aclweb.org/anthology/W03-0419",
|
150 |
+
pages = "142--147",
|
151 |
+
}
|
152 |
+
```
|
bert-large-NER-rust/.gitattributes
ADDED
@@ -0,0 +1,35 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
*.7z filter=lfs diff=lfs merge=lfs -text
|
2 |
+
*.arrow filter=lfs diff=lfs merge=lfs -text
|
3 |
+
*.bin filter=lfs diff=lfs merge=lfs -text
|
4 |
+
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
5 |
+
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
6 |
+
*.ftz filter=lfs diff=lfs merge=lfs -text
|
7 |
+
*.gz filter=lfs diff=lfs merge=lfs -text
|
8 |
+
*.h5 filter=lfs diff=lfs merge=lfs -text
|
9 |
+
*.joblib filter=lfs diff=lfs merge=lfs -text
|
10 |
+
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
11 |
+
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
12 |
+
*.model filter=lfs diff=lfs merge=lfs -text
|
13 |
+
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
14 |
+
*.npy filter=lfs diff=lfs merge=lfs -text
|
15 |
+
*.npz filter=lfs diff=lfs merge=lfs -text
|
16 |
+
*.onnx filter=lfs diff=lfs merge=lfs -text
|
17 |
+
*.ot filter=lfs diff=lfs merge=lfs -text
|
18 |
+
*.parquet filter=lfs diff=lfs merge=lfs -text
|
19 |
+
*.pb filter=lfs diff=lfs merge=lfs -text
|
20 |
+
*.pickle filter=lfs diff=lfs merge=lfs -text
|
21 |
+
*.pkl filter=lfs diff=lfs merge=lfs -text
|
22 |
+
*.pt filter=lfs diff=lfs merge=lfs -text
|
23 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
24 |
+
*.rar filter=lfs diff=lfs merge=lfs -text
|
25 |
+
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
26 |
+
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
27 |
+
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
28 |
+
*.tar filter=lfs diff=lfs merge=lfs -text
|
29 |
+
*.tflite filter=lfs diff=lfs merge=lfs -text
|
30 |
+
*.tgz filter=lfs diff=lfs merge=lfs -text
|
31 |
+
*.wasm filter=lfs diff=lfs merge=lfs -text
|
32 |
+
*.xz filter=lfs diff=lfs merge=lfs -text
|
33 |
+
*.zip filter=lfs diff=lfs merge=lfs -text
|
34 |
+
*.zst filter=lfs diff=lfs merge=lfs -text
|
35 |
+
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
bert-large-NER-rust/README.md
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
license: apache-2.0
|
3 |
+
---
|
config.json
ADDED
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
{
|
2 |
+
"architectures": [
|
3 |
+
"BertForTokenClassification"
|
4 |
+
],
|
5 |
+
"attention_probs_dropout_prob": 0.1,
|
6 |
+
"bos_token_id": null,
|
7 |
+
"directionality": "bidi",
|
8 |
+
"do_sample": false,
|
9 |
+
"eos_token_ids": null,
|
10 |
+
"finetuning_task": null,
|
11 |
+
"hidden_act": "gelu",
|
12 |
+
"hidden_dropout_prob": 0.1,
|
13 |
+
"hidden_size": 1024,
|
14 |
+
"id2label": {
|
15 |
+
"0": "O",
|
16 |
+
"1": "B-MISC",
|
17 |
+
"2": "I-MISC",
|
18 |
+
"3": "B-PER",
|
19 |
+
"4": "I-PER",
|
20 |
+
"5": "B-ORG",
|
21 |
+
"6": "I-ORG",
|
22 |
+
"7": "B-LOC",
|
23 |
+
"8": "I-LOC"
|
24 |
+
},
|
25 |
+
"initializer_range": 0.02,
|
26 |
+
"intermediate_size": 4096,
|
27 |
+
"is_decoder": false,
|
28 |
+
"label2id": {
|
29 |
+
"B-LOC": 7,
|
30 |
+
"B-MISC": 1,
|
31 |
+
"B-ORG": 5,
|
32 |
+
"B-PER": 3,
|
33 |
+
"I-LOC": 8,
|
34 |
+
"I-MISC": 2,
|
35 |
+
"I-ORG": 6,
|
36 |
+
"I-PER": 4,
|
37 |
+
"O": 0
|
38 |
+
},
|
39 |
+
"layer_norm_eps": 1e-12,
|
40 |
+
"length_penalty": 1.0,
|
41 |
+
"max_length": 20,
|
42 |
+
"max_position_embeddings": 512,
|
43 |
+
"model_type": "bert",
|
44 |
+
"num_attention_heads": 16,
|
45 |
+
"num_beams": 1,
|
46 |
+
"num_hidden_layers": 24,
|
47 |
+
"num_labels": 9,
|
48 |
+
"num_return_sequences": 1,
|
49 |
+
"output_attentions": false,
|
50 |
+
"output_hidden_states": false,
|
51 |
+
"output_past": true,
|
52 |
+
"pad_token_id": 0,
|
53 |
+
"pooler_fc_size": 768,
|
54 |
+
"pooler_num_attention_heads": 12,
|
55 |
+
"pooler_num_fc_layers": 3,
|
56 |
+
"pooler_size_per_head": 128,
|
57 |
+
"pooler_type": "first_token_transform",
|
58 |
+
"pruned_heads": {},
|
59 |
+
"repetition_penalty": 1.0,
|
60 |
+
"temperature": 1.0,
|
61 |
+
"top_k": 50,
|
62 |
+
"top_p": 1.0,
|
63 |
+
"torchscript": false,
|
64 |
+
"type_vocab_size": 2,
|
65 |
+
"use_bfloat16": false,
|
66 |
+
"vocab_size": 28996
|
67 |
+
}
|
flax_model.msgpack
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:d63d39520929bbeab16344157582be46f09a3ab43095b14f1b1586636a50f171
|
3 |
+
size 1330169528
|
model.npz
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:d627d10676d2ee3e2a5f7a91605f1527b702e2c618608fce8aaf6a0310514d5d
|
3 |
+
size 1334481458
|
model.safetensors
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:81d6a7a3967f136e18bac045a2c8c0ab30f39ae70268129e60255afa21d4205b
|
3 |
+
size 1334400964
|
onnx/added_tokens.json
ADDED
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
{
|
2 |
+
"[CLS]": 101,
|
3 |
+
"[MASK]": 103,
|
4 |
+
"[PAD]": 0,
|
5 |
+
"[SEP]": 102,
|
6 |
+
"[UNK]": 100
|
7 |
+
}
|
onnx/config.json
ADDED
@@ -0,0 +1,54 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
{
|
2 |
+
"_name_or_path": "dslim/bert-large-NER",
|
3 |
+
"architectures": [
|
4 |
+
"BertForTokenClassification"
|
5 |
+
],
|
6 |
+
"attention_probs_dropout_prob": 0.1,
|
7 |
+
"classifier_dropout": null,
|
8 |
+
"directionality": "bidi",
|
9 |
+
"eos_token_ids": null,
|
10 |
+
"hidden_act": "gelu",
|
11 |
+
"hidden_dropout_prob": 0.1,
|
12 |
+
"hidden_size": 1024,
|
13 |
+
"id2label": {
|
14 |
+
"0": "O",
|
15 |
+
"1": "B-MISC",
|
16 |
+
"2": "I-MISC",
|
17 |
+
"3": "B-PER",
|
18 |
+
"4": "I-PER",
|
19 |
+
"5": "B-ORG",
|
20 |
+
"6": "I-ORG",
|
21 |
+
"7": "B-LOC",
|
22 |
+
"8": "I-LOC"
|
23 |
+
},
|
24 |
+
"initializer_range": 0.02,
|
25 |
+
"intermediate_size": 4096,
|
26 |
+
"label2id": {
|
27 |
+
"B-LOC": 7,
|
28 |
+
"B-MISC": 1,
|
29 |
+
"B-ORG": 5,
|
30 |
+
"B-PER": 3,
|
31 |
+
"I-LOC": 8,
|
32 |
+
"I-MISC": 2,
|
33 |
+
"I-ORG": 6,
|
34 |
+
"I-PER": 4,
|
35 |
+
"O": 0
|
36 |
+
},
|
37 |
+
"layer_norm_eps": 1e-12,
|
38 |
+
"max_position_embeddings": 512,
|
39 |
+
"model_type": "bert",
|
40 |
+
"num_attention_heads": 16,
|
41 |
+
"num_hidden_layers": 24,
|
42 |
+
"output_past": true,
|
43 |
+
"pad_token_id": 0,
|
44 |
+
"pooler_fc_size": 768,
|
45 |
+
"pooler_num_attention_heads": 12,
|
46 |
+
"pooler_num_fc_layers": 3,
|
47 |
+
"pooler_size_per_head": 128,
|
48 |
+
"pooler_type": "first_token_transform",
|
49 |
+
"position_embedding_type": "absolute",
|
50 |
+
"transformers_version": "4.34.0",
|
51 |
+
"type_vocab_size": 2,
|
52 |
+
"use_cache": true,
|
53 |
+
"vocab_size": 28996
|
54 |
+
}
|
onnx/model.onnx
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:46ba1e057362a10e8163f69a2c9ef1cbb609e9eca6ab265835d8f7fa4c44106b
|
3 |
+
size 1330682413
|
onnx/special_tokens_map.json
ADDED
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
{
|
2 |
+
"cls_token": "[CLS]",
|
3 |
+
"mask_token": "[MASK]",
|
4 |
+
"pad_token": "[PAD]",
|
5 |
+
"sep_token": "[SEP]",
|
6 |
+
"unk_token": "[UNK]"
|
7 |
+
}
|
onnx/tokenizer.json
ADDED
The diff for this file is too large to render.
See raw diff
|
|
onnx/tokenizer_config.json
ADDED
@@ -0,0 +1,59 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
{
|
2 |
+
"added_tokens_decoder": {
|
3 |
+
"0": {
|
4 |
+
"content": "[PAD]",
|
5 |
+
"lstrip": false,
|
6 |
+
"normalized": false,
|
7 |
+
"rstrip": false,
|
8 |
+
"single_word": false,
|
9 |
+
"special": true
|
10 |
+
},
|
11 |
+
"100": {
|
12 |
+
"content": "[UNK]",
|
13 |
+
"lstrip": false,
|
14 |
+
"normalized": false,
|
15 |
+
"rstrip": false,
|
16 |
+
"single_word": false,
|
17 |
+
"special": true
|
18 |
+
},
|
19 |
+
"101": {
|
20 |
+
"content": "[CLS]",
|
21 |
+
"lstrip": false,
|
22 |
+
"normalized": false,
|
23 |
+
"rstrip": false,
|
24 |
+
"single_word": false,
|
25 |
+
"special": true
|
26 |
+
},
|
27 |
+
"102": {
|
28 |
+
"content": "[SEP]",
|
29 |
+
"lstrip": false,
|
30 |
+
"normalized": false,
|
31 |
+
"rstrip": false,
|
32 |
+
"single_word": false,
|
33 |
+
"special": true
|
34 |
+
},
|
35 |
+
"103": {
|
36 |
+
"content": "[MASK]",
|
37 |
+
"lstrip": false,
|
38 |
+
"normalized": false,
|
39 |
+
"rstrip": false,
|
40 |
+
"single_word": false,
|
41 |
+
"special": true
|
42 |
+
}
|
43 |
+
},
|
44 |
+
"additional_special_tokens": [],
|
45 |
+
"clean_up_tokenization_spaces": true,
|
46 |
+
"cls_token": "[CLS]",
|
47 |
+
"do_basic_tokenize": true,
|
48 |
+
"do_lower_case": false,
|
49 |
+
"mask_token": "[MASK]",
|
50 |
+
"max_len": 512,
|
51 |
+
"model_max_length": 512,
|
52 |
+
"never_split": null,
|
53 |
+
"pad_token": "[PAD]",
|
54 |
+
"sep_token": "[SEP]",
|
55 |
+
"strip_accents": null,
|
56 |
+
"tokenize_chinese_chars": true,
|
57 |
+
"tokenizer_class": "BertTokenizer",
|
58 |
+
"unk_token": "[UNK]"
|
59 |
+
}
|
onnx/vocab.txt
ADDED
The diff for this file is too large to render.
See raw diff
|
|
pytorch_model.bin
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:db4d615a10a12cfe134388d43d8fba48e909e7f7afa3e4d3b1bd2c24ff7cea3d
|
3 |
+
size 1334453533
|
rust_model.ot
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:7e7064a7b5b931482c99d04fb99f394254f2d3fd5f5d420f5d145b7d22f3cd56
|
3 |
+
size 1334478187
|
special_tokens_map.json
ADDED
@@ -0,0 +1 @@
|
|
|
|
|
1 |
+
{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}
|
tf_model.h5
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:febce5a7915f2c894cdeb81cbe53bcc06ff3a8c22a96eaf9605aea0bbab172ba
|
3 |
+
size 1334871740
|
tokenizer_config.json
ADDED
@@ -0,0 +1 @@
|
|
|
|
|
1 |
+
{"do_lower_case": false, "max_len": 512}
|
vocab.txt
ADDED
The diff for this file is too large to render.
See raw diff
|
|