lysandre HF staff commited on
Commit
c76159d
1 Parent(s): 8553342

Create ALBERT Large v2 readme

Browse files
Files changed (1) hide show
  1. README.md +271 -0
README.md ADDED
@@ -0,0 +1,271 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ license: apache-2.0
4
+ datasets:
5
+ - bookcorpus
6
+ - wikipedia
7
+ ---
8
+
9
+ # ALBERT Large v2
10
+
11
+ Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in
12
+ [this paper](https://arxiv.org/abs/1909.11942) and first released in
13
+ [this repository](https://github.com/google-research/albert). This model, as all ALBERT models, is uncased: it does not make a difference
14
+ between english and English.
15
+
16
+ Disclaimer: The team releasing ALBERT did not write a model card for this model so this model card has been written by
17
+ the Hugging Face team.
18
+
19
+ ## Model description
20
+
21
+ ALBERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it
22
+ was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
23
+ publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it
24
+ was pretrained with two objectives:
25
+
26
+ - Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run
27
+ the entire masked sentence through the model and has to predict the masked words. This is different from traditional
28
+ recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like
29
+ GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the
30
+ sentence.
31
+ - Sentence Ordering Prediction (SOP): ALBERT uses a pretraining loss based on predicting the ordering of two consecutive segments of text.
32
+
33
+ This way, the model learns an inner representation of the English language that can then be used to extract features
34
+ useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
35
+ classifier using the features produced by the ALBERT model as inputs.
36
+
37
+ ALBERT is particular in that it shares its layers across its Transformer. Therefore, all layers have the same weights. Using repeating layers results in a small memory footprint, however, the computational cost remains similar to a BERT-like architecture with the same number of hidden layers as it has to iterate through the same number of (repeating) layers.
38
+
39
+ This is the second version of the large model. Version 2 is different from version 1 due to different dropout rates, additional training data, and longer training. It has better results in nearly all downstream tasks.
40
+
41
+ This model has the following configuration:
42
+
43
+ - 24 repeating layers
44
+ - 128 embedding dimension
45
+ - 1024 hidden dimension
46
+ - 16 attention heads
47
+ - 17M parameters
48
+
49
+ ## Intended uses & limitations
50
+
51
+ You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to
52
+ be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=albert) to look for
53
+ fine-tuned versions on a task that interests you.
54
+
55
+ Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
56
+ to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
57
+ generation you should look at model like GPT2.
58
+
59
+ ### How to use
60
+
61
+ You can use this model directly with a pipeline for masked language modeling:
62
+
63
+ ```python
64
+ >>> from transformers import pipeline
65
+ >>> unmasker = pipeline('fill-mask', model='albert-large-v2')
66
+ >>> unmasker("Hello I'm a [MASK] model.")
67
+ [
68
+ {
69
+ "sequence":"[CLS] hello i'm a modeling model.[SEP]",
70
+ "score":0.05816134437918663,
71
+ "token":12807,
72
+ "token_str":"▁modeling"
73
+ },
74
+ {
75
+ "sequence":"[CLS] hello i'm a modelling model.[SEP]",
76
+ "score":0.03748830780386925,
77
+ "token":23089,
78
+ "token_str":"▁modelling"
79
+ },
80
+ {
81
+ "sequence":"[CLS] hello i'm a model model.[SEP]",
82
+ "score":0.033725276589393616,
83
+ "token":1061,
84
+ "token_str":"▁model"
85
+ },
86
+ {
87
+ "sequence":"[CLS] hello i'm a runway model.[SEP]",
88
+ "score":0.017313428223133087,
89
+ "token":8014,
90
+ "token_str":"▁runway"
91
+ },
92
+ {
93
+ "sequence":"[CLS] hello i'm a lingerie model.[SEP]",
94
+ "score":0.014405295252799988,
95
+ "token":29104,
96
+ "token_str":"▁lingerie"
97
+ }
98
+ ]
99
+ ```
100
+
101
+ Here is how to use this model to get the features of a given text in PyTorch:
102
+
103
+ ```python
104
+ from transformers import AlbertTokenizer, AlbertModel
105
+ tokenizer = AlbertTokenizer.from_pretrained('albert-large-v2')
106
+ model = AlbertModel.from_pretrained("albert-large-v2")
107
+ text = "Replace me by any text you'd like."
108
+ encoded_input = tokenizer(text, return_tensors='pt')
109
+ output = model(**encoded_input)
110
+ ```
111
+
112
+ and in TensorFlow:
113
+
114
+ ```python
115
+ from transformers import AlbertTokenizer, TFAlbertModel
116
+ tokenizer = AlbertTokenizer.from_pretrained('albert-large-v2')
117
+ model = TFAlbertModel.from_pretrained("albert-large-v2")
118
+ text = "Replace me by any text you'd like."
119
+ encoded_input = tokenizer(text, return_tensors='tf')
120
+ output = model(encoded_input)
121
+ ```
122
+
123
+ ### Limitations and bias
124
+
125
+ Even if the training data used for this model could be characterized as fairly neutral, this model can have biased
126
+ predictions:
127
+
128
+ ```python
129
+ >>> from transformers import pipeline
130
+ >>> unmasker = pipeline('fill-mask', model='albert-large-v2')
131
+ >>> unmasker("The man worked as a [MASK].")
132
+
133
+ [
134
+ {
135
+ "sequence":"[CLS] the man worked as a chauffeur.[SEP]",
136
+ "score":0.029577180743217468,
137
+ "token":28744,
138
+ "token_str":"▁chauffeur"
139
+ },
140
+ {
141
+ "sequence":"[CLS] the man worked as a janitor.[SEP]",
142
+ "score":0.028865724802017212,
143
+ "token":29477,
144
+ "token_str":"▁janitor"
145
+ },
146
+ {
147
+ "sequence":"[CLS] the man worked as a shoemaker.[SEP]",
148
+ "score":0.02581118606030941,
149
+ "token":29024,
150
+ "token_str":"▁shoemaker"
151
+ },
152
+ {
153
+ "sequence":"[CLS] the man worked as a blacksmith.[SEP]",
154
+ "score":0.01849772222340107,
155
+ "token":21238,
156
+ "token_str":"▁blacksmith"
157
+ },
158
+ {
159
+ "sequence":"[CLS] the man worked as a lawyer.[SEP]",
160
+ "score":0.01820771023631096,
161
+ "token":3672,
162
+ "token_str":"▁lawyer"
163
+ }
164
+ ]
165
+
166
+ >>> unmasker("The woman worked as a [MASK].")
167
+
168
+ [
169
+ {
170
+ "sequence":"[CLS] the woman worked as a receptionist.[SEP]",
171
+ "score":0.04604868218302727,
172
+ "token":25331,
173
+ "token_str":"▁receptionist"
174
+ },
175
+ {
176
+ "sequence":"[CLS] the woman worked as a janitor.[SEP]",
177
+ "score":0.028220869600772858,
178
+ "token":29477,
179
+ "token_str":"▁janitor"
180
+ },
181
+ {
182
+ "sequence":"[CLS] the woman worked as a paramedic.[SEP]",
183
+ "score":0.0261906236410141,
184
+ "token":23386,
185
+ "token_str":"▁paramedic"
186
+ },
187
+ {
188
+ "sequence":"[CLS] the woman worked as a chauffeur.[SEP]",
189
+ "score":0.024797942489385605,
190
+ "token":28744,
191
+ "token_str":"▁chauffeur"
192
+ },
193
+ {
194
+ "sequence":"[CLS] the woman worked as a waitress.[SEP]",
195
+ "score":0.024124596267938614,
196
+ "token":13678,
197
+ "token_str":"▁waitress"
198
+ }
199
+ ]
200
+ ```
201
+
202
+ This bias will also affect all fine-tuned versions of this model.
203
+
204
+ ## Training data
205
+
206
+ The ALBERT model was pretrained on [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038
207
+ unpublished books and [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and
208
+ headers).
209
+
210
+ ## Training procedure
211
+
212
+ ### Preprocessing
213
+
214
+ The texts are lowercased and tokenized using SentencePiece and a vocabulary size of 30,000. The inputs of the model are
215
+ then of the form:
216
+
217
+ ```
218
+ [CLS] Sentence A [SEP] Sentence B [SEP]
219
+ ```
220
+
221
+ ### Training
222
+
223
+ The ALBERT procedure follows the BERT setup.
224
+
225
+ The details of the masking procedure for each sentence are the following:
226
+ - 15% of the tokens are masked.
227
+ - In 80% of the cases, the masked tokens are replaced by `[MASK]`.
228
+ - In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
229
+ - In the 10% remaining cases, the masked tokens are left as is.
230
+
231
+ ## Evaluation results
232
+
233
+ When fine-tuned on downstream tasks, the ALBERT models achieve the following results:
234
+
235
+ | | Average | SQuAD1.1 | SQuAD2.0 | MNLI | SST-2 | RACE |
236
+ |----------------|----------|----------|----------|----------|----------|----------|
237
+ |V2 |
238
+ |ALBERT-base |82.3 |90.2/83.2 |82.1/79.3 |84.6 |92.9 |66.8 |
239
+ |ALBERT-large |85.7 |91.8/85.2 |84.9/81.8 |86.5 |94.9 |75.2 |
240
+ |ALBERT-xlarge |87.9 |92.9/86.4 |87.9/84.1 |87.9 |95.4 |80.7 |
241
+ |ALBERT-xxlarge |90.9 |94.6/89.1 |89.8/86.9 |90.6 |96.8 |86.8 |
242
+ |V1 |
243
+ |ALBERT-base |80.1 |89.3/82.3 | 80.0/77.1|81.6 |90.3 | 64.0 |
244
+ |ALBERT-large |82.4 |90.6/83.9 | 82.3/79.4|83.5 |91.7 | 68.5 |
245
+ |ALBERT-xlarge |85.5 |92.5/86.1 | 86.1/83.1|86.4 |92.4 | 74.8 |
246
+ |ALBERT-xxlarge |91.0 |94.8/89.3 | 90.2/87.4|90.8 |96.9 | 86.5 |
247
+
248
+
249
+ ### BibTeX entry and citation info
250
+
251
+ ```bibtex
252
+ @article{DBLP:journals/corr/abs-1909-11942,
253
+ author = {Zhenzhong Lan and
254
+ Mingda Chen and
255
+ Sebastian Goodman and
256
+ Kevin Gimpel and
257
+ Piyush Sharma and
258
+ Radu Soricut},
259
+ title = {{ALBERT:} {A} Lite {BERT} for Self-supervised Learning of Language
260
+ Representations},
261
+ journal = {CoRR},
262
+ volume = {abs/1909.11942},
263
+ year = {2019},
264
+ url = {http://arxiv.org/abs/1909.11942},
265
+ archivePrefix = {arXiv},
266
+ eprint = {1909.11942},
267
+ timestamp = {Fri, 27 Sep 2019 13:04:21 +0200},
268
+ biburl = {https://dblp.org/rec/journals/corr/abs-1909-11942.bib},
269
+ bibsource = {dblp computer science bibliography, https://dblp.org}
270
+ }
271
+ ```