Create README.md
Browse files
README.md
ADDED
@@ -0,0 +1,126 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
language: en
|
3 |
+
license: apache-2.0
|
4 |
+
datasets:
|
5 |
+
- bookcorpus
|
6 |
+
- wikipedia
|
7 |
+
---
|
8 |
+
|
9 |
+
# BERT large model (uncased) whole word masking finetuned on SQuAD
|
10 |
+
|
11 |
+
Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in
|
12 |
+
[this paper](https://arxiv.org/abs/1810.04805) and first released in
|
13 |
+
[this repository](https://github.com/google-research/bert). This model is uncased: it does not make a difference
|
14 |
+
between english and English.
|
15 |
+
|
16 |
+
Differently to other BERT models, this model was trained with a new technique: Whole Word Masking. In this case, all of the tokens corresponding to a word are masked at once. The overall masking rate remains the same.
|
17 |
+
|
18 |
+
The training is identical -- each masked WordPiece token is predicted independently.
|
19 |
+
|
20 |
+
After pre-training, this model was fine-tuned on the SQuAD dataset with one of our fine-tuning scripts. See below for more information regarding this fine-tuning.
|
21 |
+
|
22 |
+
Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by
|
23 |
+
the Hugging Face team.
|
24 |
+
|
25 |
+
## Model description
|
26 |
+
|
27 |
+
BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it
|
28 |
+
was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
|
29 |
+
publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it
|
30 |
+
was pretrained with two objectives:
|
31 |
+
|
32 |
+
- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run
|
33 |
+
the entire masked sentence through the model and has to predict the masked words. This is different from traditional
|
34 |
+
recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like
|
35 |
+
GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the
|
36 |
+
sentence.
|
37 |
+
- Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes
|
38 |
+
they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to
|
39 |
+
predict if the two sentences were following each other or not.
|
40 |
+
|
41 |
+
This way, the model learns an inner representation of the English language that can then be used to extract features
|
42 |
+
useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
|
43 |
+
classifier using the features produced by the BERT model as inputs.
|
44 |
+
|
45 |
+
## Intended uses & limitations
|
46 |
+
This model should be used as a question-answering model. You may use it in a question answering pipeline, or use it to output raw results given a query and a context. You may see other use cases in the [task summary](https://huggingface.co/transformers/task_summary.html#extractive-question-answering) of the transformers documentation.## Training data
|
47 |
+
|
48 |
+
The BERT model was pretrained on [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038
|
49 |
+
unpublished books and [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and
|
50 |
+
headers).
|
51 |
+
|
52 |
+
## Training procedure
|
53 |
+
|
54 |
+
### Preprocessing
|
55 |
+
|
56 |
+
The texts are lowercased and tokenized using WordPiece and a vocabulary size of 30,000. The inputs of the model are
|
57 |
+
then of the form:
|
58 |
+
|
59 |
+
```
|
60 |
+
[CLS] Sentence A [SEP] Sentence B [SEP]
|
61 |
+
```
|
62 |
+
|
63 |
+
With probability 0.5, sentence A and sentence B correspond to two consecutive sentences in the original corpus and in
|
64 |
+
the other cases, it's another random sentence in the corpus. Note that what is considered a sentence here is a
|
65 |
+
consecutive span of text usually longer than a single sentence. The only constrain is that the result with the two
|
66 |
+
"sentences" has a combined length of less than 512 tokens.
|
67 |
+
|
68 |
+
The details of the masking procedure for each sentence are the following:
|
69 |
+
- 15% of the tokens are masked.
|
70 |
+
- In 80% of the cases, the masked tokens are replaced by `[MASK]`.
|
71 |
+
- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
|
72 |
+
- In the 10% remaining cases, the masked tokens are left as is.
|
73 |
+
|
74 |
+
### Pretraining
|
75 |
+
|
76 |
+
The model was trained on 4 cloud TPUs in Pod configuration (16 TPU chips total) for one million steps with a batch size
|
77 |
+
of 256. The sequence length was limited to 128 tokens for 90% of the steps and 512 for the remaining 10%. The optimizer
|
78 |
+
used is Adam with a learning rate of 1e-4, \\(\beta_{1} = 0.9\\) and \\(\beta_{2} = 0.999\\), a weight decay of 0.01,
|
79 |
+
learning rate warmup for 10,000 steps and linear decay of the learning rate after.
|
80 |
+
### Fine-tuningAfter pre-training, this model was fine-tuned on the SQuAD dataset with one of our fine-tuning scripts. In order to reproduce the training, you may use the following command:
|
81 |
+
```
|
82 |
+
python -m torch.distributed.launch --nproc_per_node=8 ./examples/question-answering/run_qa.py \
|
83 |
+
--model_name_or_path bert-large-uncased-whole-word-masking \
|
84 |
+
--dataset_name squad \
|
85 |
+
--do_train \
|
86 |
+
--do_eval \
|
87 |
+
--learning_rate 3e-5 \
|
88 |
+
--num_train_epochs 2 \
|
89 |
+
--max_seq_length 384 \
|
90 |
+
--doc_stride 128 \
|
91 |
+
--output_dir ./examples/models/wwm_uncased_finetuned_squad/ \
|
92 |
+
--per_device_eval_batch_size=3 \
|
93 |
+
--per_device_train_batch_size=3 \
|
94 |
+
```
|
95 |
+
|
96 |
+
## Evaluation results
|
97 |
+
|
98 |
+
The results obtained are the following:
|
99 |
+
|
100 |
+
```
|
101 |
+
f1 = 93.15
|
102 |
+
exact_match = 86.91
|
103 |
+
```
|
104 |
+
|
105 |
+
|
106 |
+
### BibTeX entry and citation info
|
107 |
+
|
108 |
+
```bibtex
|
109 |
+
@article{DBLP:journals/corr/abs-1810-04805,
|
110 |
+
author = {Jacob Devlin and
|
111 |
+
Ming{-}Wei Chang and
|
112 |
+
Kenton Lee and
|
113 |
+
Kristina Toutanova},
|
114 |
+
title = {{BERT:} Pre-training of Deep Bidirectional Transformers for Language
|
115 |
+
Understanding},
|
116 |
+
journal = {CoRR},
|
117 |
+
volume = {abs/1810.04805},
|
118 |
+
year = {2018},
|
119 |
+
url = {http://arxiv.org/abs/1810.04805},
|
120 |
+
archivePrefix = {arXiv},
|
121 |
+
eprint = {1810.04805},
|
122 |
+
timestamp = {Tue, 30 Oct 2018 20:39:56 +0100},
|
123 |
+
biburl = {https://dblp.org/rec/journals/corr/abs-1810-04805.bib},
|
124 |
+
bibsource = {dblp computer science bibliography, https://dblp.org}
|
125 |
+
}
|
126 |
+
```
|