lysandre HF staff commited on
Commit
cdbfa29
1 Parent(s): 0ce9a5b

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +224 -0
README.md ADDED
@@ -0,0 +1,224 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ license: apache-2.0
4
+ datasets:
5
+ - bookcorpus
6
+ - wikipedia
7
+ ---
8
+
9
+ # BERT large model (uncased)
10
+
11
+ Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in
12
+ [this paper](https://arxiv.org/abs/1810.04805) and first released in
13
+ [this repository](https://github.com/google-research/bert). This model is uncased: it does not make a difference
14
+ between english and English.
15
+
16
+ Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by
17
+ the Hugging Face team.
18
+
19
+ ## Model description
20
+
21
+ BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it
22
+ was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
23
+ publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it
24
+ was pretrained with two objectives:
25
+
26
+ - Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run
27
+ the entire masked sentence through the model and has to predict the masked words. This is different from traditional
28
+ recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like
29
+ GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the
30
+ sentence.
31
+ - Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes
32
+ they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to
33
+ predict if the two sentences were following each other or not.
34
+
35
+ This way, the model learns an inner representation of the English language that can then be used to extract features
36
+ useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
37
+ classifier using the features produced by the BERT model as inputs.
38
+
39
+ ## Intended uses & limitations
40
+
41
+ You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to
42
+ be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=bert) to look for
43
+ fine-tuned versions on a task that interests you.
44
+
45
+ Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
46
+ to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
47
+ generation you should look at model like GPT2.
48
+
49
+ ### How to use
50
+
51
+ You can use this model directly with a pipeline for masked language modeling:
52
+
53
+ ```python
54
+ >>> from transformers import pipeline
55
+ >>> unmasker = pipeline('fill-mask', model='bert-large-uncased')
56
+ >>> unmasker("Hello I'm a [MASK] model.")
57
+ [{'sequence': "[CLS] hello i'm a fashion model. [SEP]",
58
+ 'score': 0.1886913776397705,
59
+ 'token': 4827,
60
+ 'token_str': 'fashion'},
61
+ {'sequence': "[CLS] hello i'm a professional model. [SEP]",
62
+ 'score': 0.07157472521066666,
63
+ 'token': 2658,
64
+ 'token_str': 'professional'},
65
+ {'sequence': "[CLS] hello i'm a male model. [SEP]",
66
+ 'score': 0.04053466394543648,
67
+ 'token': 3287,
68
+ 'token_str': 'male'},
69
+ {'sequence': "[CLS] hello i'm a role model. [SEP]",
70
+ 'score': 0.03891477733850479,
71
+ 'token': 2535,
72
+ 'token_str': 'role'},
73
+ {'sequence': "[CLS] hello i'm a fitness model. [SEP]",
74
+ 'score': 0.03038121573626995,
75
+ 'token': 10516,
76
+ 'token_str': 'fitness'}]
77
+ ```
78
+
79
+ Here is how to use this model to get the features of a given text in PyTorch:
80
+
81
+ ```python
82
+ from transformers import BertTokenizer, BertModel
83
+ tokenizer = BertTokenizer.from_pretrained('bert-large-uncased')
84
+ model = BertModel.from_pretrained("bert-large-uncased")
85
+ text = "Replace me by any text you'd like."
86
+ encoded_input = tokenizer(text, return_tensors='pt')
87
+ output = model(**encoded_input)
88
+ ```
89
+
90
+ and in TensorFlow:
91
+
92
+ ```python
93
+ from transformers import BertTokenizer, TFBertModel
94
+ tokenizer = BertTokenizer.from_pretrained('bert-large-uncased')
95
+ model = TFBertModel.from_pretrained("bert-large-uncased")
96
+ text = "Replace me by any text you'd like."
97
+ encoded_input = tokenizer(text, return_tensors='tf')
98
+ output = model(encoded_input)
99
+ ```
100
+
101
+ ### Limitations and bias
102
+
103
+ Even if the training data used for this model could be characterized as fairly neutral, this model can have biased
104
+ predictions:
105
+
106
+ ```python
107
+ >>> from transformers import pipeline
108
+ >>> unmasker = pipeline('fill-mask', model='bert-large-uncased')
109
+ >>> unmasker("The man worked as a [MASK].")
110
+
111
+ [{'sequence': '[CLS] the man worked as a bartender. [SEP]',
112
+ 'score': 0.10426565259695053,
113
+ 'token': 15812,
114
+ 'token_str': 'bartender'},
115
+ {'sequence': '[CLS] the man worked as a waiter. [SEP]',
116
+ 'score': 0.10232779383659363,
117
+ 'token': 15610,
118
+ 'token_str': 'waiter'},
119
+ {'sequence': '[CLS] the man worked as a mechanic. [SEP]',
120
+ 'score': 0.06281787157058716,
121
+ 'token': 15893,
122
+ 'token_str': 'mechanic'},
123
+ {'sequence': '[CLS] the man worked as a lawyer. [SEP]',
124
+ 'score': 0.050936125218868256,
125
+ 'token': 5160,
126
+ 'token_str': 'lawyer'},
127
+ {'sequence': '[CLS] the man worked as a carpenter. [SEP]',
128
+ 'score': 0.041034240275621414,
129
+ 'token': 10533,
130
+ 'token_str': 'carpenter'}]
131
+
132
+ >>> unmasker("The woman worked as a [MASK].")
133
+
134
+ [{'sequence': '[CLS] the woman worked as a waitress. [SEP]',
135
+ 'score': 0.28473711013793945,
136
+ 'token': 13877,
137
+ 'token_str': 'waitress'},
138
+ {'sequence': '[CLS] the woman worked as a nurse. [SEP]',
139
+ 'score': 0.11336520314216614,
140
+ 'token': 6821,
141
+ 'token_str': 'nurse'},
142
+ {'sequence': '[CLS] the woman worked as a bartender. [SEP]',
143
+ 'score': 0.09574324637651443,
144
+ 'token': 15812,
145
+ 'token_str': 'bartender'},
146
+ {'sequence': '[CLS] the woman worked as a maid. [SEP]',
147
+ 'score': 0.06351090222597122,
148
+ 'token': 10850,
149
+ 'token_str': 'maid'},
150
+ {'sequence': '[CLS] the woman worked as a secretary. [SEP]',
151
+ 'score': 0.048970773816108704,
152
+ 'token': 3187,
153
+ 'token_str': 'secretary'}]
154
+ ```
155
+
156
+ This bias will also affect all fine-tuned versions of this model.
157
+
158
+ ## Training data
159
+
160
+ The BERT model was pretrained on [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038
161
+ unpublished books and [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and
162
+ headers).
163
+
164
+ ## Training procedure
165
+
166
+ ### Preprocessing
167
+
168
+ The texts are lowercased and tokenized using WordPiece and a vocabulary size of 30,000. The inputs of the model are
169
+ then of the form:
170
+
171
+ ```
172
+ [CLS] Sentence A [SEP] Sentence B [SEP]
173
+ ```
174
+
175
+ With probability 0.5, sentence A and sentence B correspond to two consecutive sentences in the original corpus and in
176
+ the other cases, it's another random sentence in the corpus. Note that what is considered a sentence here is a
177
+ consecutive span of text usually longer than a single sentence. The only constrain is that the result with the two
178
+ "sentences" has a combined length of less than 512 tokens.
179
+
180
+ The details of the masking procedure for each sentence are the following:
181
+ - 15% of the tokens are masked.
182
+ - In 80% of the cases, the masked tokens are replaced by `[MASK]`.
183
+ - In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
184
+ - In the 10% remaining cases, the masked tokens are left as is.
185
+
186
+ ### Pretraining
187
+
188
+ The model was trained on 4 cloud TPUs in Pod configuration (16 TPU chips total) for one million steps with a batch size
189
+ of 256. The sequence length was limited to 128 tokens for 90% of the steps and 512 for the remaining 10%. The optimizer
190
+ used is Adam with a learning rate of 1e-4, \\(\beta_{1} = 0.9\\) and \\(\beta_{2} = 0.999\\), a weight decay of 0.01,
191
+ learning rate warmup for 10,000 steps and linear decay of the learning rate after.
192
+
193
+ ## Evaluation results
194
+
195
+ When fine-tuned on downstream tasks, this model achieves the following results:
196
+
197
+ Glue test results:
198
+
199
+ | Task | MNLI-(m/mm) | QQP | QNLI | SST-2 | CoLA | STS-B | MRPC | RTE | Average |
200
+ |:----:|:-----------:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|:-------:|
201
+ | | 84.6/83.4 | 71.2 | 90.5 | 93.5 | 52.1 | 85.8 | 88.9 | 66.4 | 79.6 |
202
+
203
+
204
+ ### BibTeX entry and citation info
205
+
206
+ ```bibtex
207
+ @article{DBLP:journals/corr/abs-1810-04805,
208
+ author = {Jacob Devlin and
209
+ Ming{-}Wei Chang and
210
+ Kenton Lee and
211
+ Kristina Toutanova},
212
+ title = {{BERT:} Pre-training of Deep Bidirectional Transformers for Language
213
+ Understanding},
214
+ journal = {CoRR},
215
+ volume = {abs/1810.04805},
216
+ year = {2018},
217
+ url = {http://arxiv.org/abs/1810.04805},
218
+ archivePrefix = {arXiv},
219
+ eprint = {1810.04805},
220
+ timestamp = {Tue, 30 Oct 2018 20:39:56 +0100},
221
+ biburl = {https://dblp.org/rec/journals/corr/abs-1810-04805.bib},
222
+ bibsource = {dblp computer science bibliography, https://dblp.org}
223
+ }
224
+ ```