T HE M ULTI B ERTS : B ERT R EPRODUCTIONS FOR R OBUSTNESS A NALYSIS
Google Research
A BSTRACT
Experiments with pre-trained models such as B ERT are often based on a single checkpoint.
While the conclusions drawn apply to the artifact tested in the experiment (i.e., the particular instance of the model), it is not always clear whether they hold for the more general procedure which includes the architecture, training data, initialization scheme, and loss function.
Recent work has shown that repeating the pre-training process can lead to substantially different performance, suggesting that an alternate strategy is needed to make principled statements about procedures.
To enable researchers to draw more robust conclusions, we introduce the MultiB ERT s, a set of 25 B ERT -Base checkpoints, trained with similar hyper-parameters as the original B ERT model but differing in random weight initialization and shufﬂing of training data.
We also deﬁne the Multi-Bootstrap, a non-parametric bootstrap method for statistical inference designed for settings where there are multiple pre-trained models and limited test data.
To illustrate our approach, we present a case study of gender bias in coreference resolution, in which the Multi-Bootstrap lets us measure effects that may not be detected with a single checkpoint.
We release our models and statistical library, 1 along with an additional set of 140 intermediate checkpoints captured during pre-training to facilitate research on learning dynamics.
I NTRODUCTION
Contemporary natural language processing (NLP) relies heavily on pretrained language models, which are trained using large-scale unlabeled data (Bommasani et al., 2021).
B ERT (Devlin et al., 2019) is a particularly popular choice: it has been widely adopted in academia and industry, and aspects of its performance have been reported on in thousands of research papers (see, e.g., Rogers et al., 2020, for an overview).
Because pre-training large language models is computationally expensive (Strubell et al., 2019), researchers often rely on the release of model checkpoints through libraries such as HuggingFace Transformers (Wolf et al., 2020), which enable them to use large-scale language models without repeating the pre-training work.
Consequently, most published results are based on a small number of publicly released model checkpoints.
While this reuse of model checkpoints has lowered the cost of research and facilitated head-to-head comparisons, it limits our ability to draw general scientiﬁc conclusions about the performance of a particular class of models (Dror et al., 2019; D’Amour et al., 2020; Zhong et al., 2021).
The key issue is that reusing model checkpoints makes it hard to generalize observations about the behavior of a single model artifact to statements about the underlying pre-training procedure which created it.
Pre-training such models is an inherently stochastic process which depends on the initialization of the model’s parameters and the ordering of training examples; for example, D’Amour et al.
(2020) report substantial quantitative differences across multiple checkpoints of the same model architecture on several “stress tests” (Naik et al., 2018; McCoy et al., 2019).
It is therefore difﬁcult to know how much of the success of a model based on the original B ERT checkpoint is due to B ERT ’s design, and how much is due to idiosyncracies of a particular artifact.
Understanding this difference is critical if we are to generate reusable insights about deep learning for NLP, and improve the state-of-the-art going forward (Zhou et al., 2020; Dodge et al., 2020;
Aribandi et al., 2021).
This paper describes the MultiB ERT s, an effort to facilitate more robust research on the B ERT model.
Our primary contributions are:
• We release the MultiB ERT s, a set of 25 B ERT -Base, Uncased checkpoints to facilitate studies of robustness to parameter initialization and order of training examples ( § 2).
Releasing these models preserves the beneﬁts to the community of a single checkpoint release (i e., low cost of experiments, apples-to-apples comparisons between studies based on these checkpoints), while enabling researchers to draw more general conclusions about the B ERT pre-training procedure.
• We present the Multi-Bootstrap, a non-parametric method to quantify the uncertainty of experimental results based on multiple pre-training seeds ( § 3), and provide recommendations for how to use the Multi-Bootstrap and MultiB ERT s in typical experimental scenarios.
We implement these recommendations in a software library.
• We illustrate the approach with a practical use case: we investigate the impact of counterfactual data augmentation on gender bias, in a B ERT -based coreference resolution systems (Webster et al., 2020) ( § 4).
Additional examples are provided in Appendix E, where we document challenges with reproducing the widely-used original B ERT checkpoint.
The release also includes an additional 140 intermediate checkpoints, captured during training forof the runs (28 checkpoints per run), to facilitate studies of learning dynamics.
Our checkpoints and statistical libraries are available at: http://goo.gle/multiberts .
Additional Related Work.
The MultiB ERT s release builds on top of a large body of work that seeks to analyze the behavior of B ERT (Rogers et al., 2020).
In addition to the studies of robustness cited above, several authors have introduced methods to reduce B ERT ’s variability during ﬁnetuning (Zhang et al., 2021;
Mosbach et al., 2021; Dodge et al., 2020; Lee et al., 2020; Phang et al., 2018).
Other authors have also studied the time dimension, which motivates our release of intermediate checkpoints (Liu et al., 2021;
Hao et al., 2020; Saphra & Lopez, 2019; Chiang et al., 2020;
Dodge et al., 2020).
Similarly to § 3, authors in the NLP literature have recommended best practices for statistical testing (Koehn, 2004; Dror et al., 2018; Berg-Kirkpatrick et al., 2012; Card et al., 2020;
Søgaard et al., 2014; Peyrard et al., 2021), many of which are based on existing tests to estimate the uncertainty of test sample.
In concurrent work, Deutsch et al.
(2021) considered bootstrapping methods similar to the Multi-Bootstrap, in the context of summarization metrics evaluation.
Also in concurrent work, the Mistral project (Karamcheti et al., 2021) released a set of 10 GPT-2 models with intermediate checkpoints at different stages of pre-training.
Our work is complementary, focusing on BERT, introducing a larger number of pre-training seeds, and presenting a methodology to draw robust conclusions about model performance.
R ELEASE D ESCRIPTION
We ﬁrst describe the MultiB ERT s release: how the checkpoints were trained and how their performance compares to the original B ERT on two common language understanding benchmarks.
2. T RAINING
Overview.
The MultiB ERT s checkpoints are trained following the code and procedure of Devlin et al.
(2019), with minor hyperparameter modiﬁcations necessary to obtain comparable results on
GLUE (Wang et al., 2019); a detailed discussion of these differences is provided in Appendix E.
We use the B ERT -Base, Uncased architecture with 12 layers and embedding size 768.
We trained the models on a combination of BooksCorpus (Zhu et al., 2015) and English Wikipedia.
Since the
[Figure]
MNLI (acc) MRPC (acc)
QNLI_v2 (acc)
QQP
[Figure]
acc)
RTE (acc)
[Figure]
acc)
[Figure]
[Figure]
[Figure]
# c he ck
[Figure]
i
[Figure]
exact dataset used to train the original B ERT is not available, we used a more recent version that was collected by Turc et al.
with the same methodology. Checkpoints. We release 25 models trained for two million steps each, each training step involving a batch of 256 sequences. For ﬁve of these models, we release 28 additional checkpoints captured over the course of pre-training (every 20,000 training steps up to 200,000, then every 100,000 steps).
In total, we release 165 checkpoints, about 68 GB of data.
Training Details.
As in the original B ERT paper, we used batch size 256 and the Adam optimizer (Kingma & Ba, 2014) with learning rate 1e-4 and 10,000 warm-up steps.
We used the default values for all the other parameters, except the number of steps, which we set to two million, and sequence length, which we set to 512 from the beginning with up to 80 masked tokens per sequence.
We follow the B ERT code and initialize the layer parameters from a truncated normal distribution, using mean 0 and standard deviation 0 .
We train using the same conﬁguration as Devlin et al.
(2019) 3 , with each run taking about 4.5 days on 16 Cloud TPU v2 chips.
Environmental Impact Statement.
We estimate compute costs at around 1728 TPU-hours for each pre-training run, and around 208 GPU-hours plus 8 TPU-hours for associated ﬁne-tuning experiments ( § 2.2, including hyperparameter search and 5x replication).
Using the calculations of
Luccioni et al.
(2019) 4 , we estimate this as about 250 kg CO2e for each of our 25 models.
Counting the 25 runs each of CDA-incr and CDA-full from § 4, associated coreference models (20 GPU-hours per pretraining model), and additional experiments of Appendix E, this gives a total of about 12.
metric tons CO2e before accounting for offsets or clean energy.
Based on the report by Patterson et al.
(2021) of 78% carbon-free energy in Google Iowa (us-central1), we estimate that reproducing these experiments would emit closer to 2.6 tons CO2e, or slightly more than two passengers on a round-trip ﬂight between San Francisco and New York.
By releasing the trained checkpoints publicly, we aim to enable many research efforts on reproducibility and robustness without requiring this cost to be incurred for every subsequent study.
2. P ERFORMANCE B ENCHMARKS
GLUE Setup.
We report results on the development sets of the GLUE tasks:
CoLA (Warstadt et al., 2019), MNLI (matched) (Williams et al., 2018), MRPC (Dolan & Brockett, 2005), QNLI (v2) (Rajpurkar et al., 2016; Wang et al., 2019), QQP (Chen et al., 2018), RTE (Bentivogli et al., 2009), SST-2 (Socher et al., 2013), and SST-B (Cer et al., 2017).
In all cases we follow the same approach as Devlin et al.
For each task, we ﬁne-tune B ERT for 3 epochs using a batch size of 32.
We run a parameter sweep on learning rates [5e-5, 4e-5, 3e-5, 2e-5] and report the best score.
We run the procedure ﬁve times for each of the 25 models and average the results. [Figure]
SQuAD Setup.
We report results on the development sets of SQuAD versions 1.1 and 2.0 (Rajpurkar et al., 2016; 2018), using a setup similar to that of Devlin et al.
For both sets of experiments, we use batch size 48, learning rate 5e-5, and train for 2 epochs.
Results.
Figures 1 and 2 show the distribution of the MultiB ERT s models’ performance on the development sets of GLUE and SQuAD, in comparison to the original B ERT checkpoint.
On most tasks, original B ERT ’s performance falls within the same range as MultiB ERT s (i.e., original B ERT is between the minimum and maximum of the MultiB ERT s’ scores).
However, original B ERT outperforms all MultiB ERT s models on QQP, and under-performs them on SQuAD.
The discrepancies may be explained by both randomness and differences in training setups, as investigated further in Appendix E.
To further illustrate the performance variability inherent to pre-training and ﬁne-tuning, we analyze the instance-level agreement between the models in Appendix C.
H YPOTHESIS T ESTING U SING M ULTIPLE C HECKPOINTS
The previous section compared MultiB ERT s with the original B ERT , ﬁnding many similarities but also some differences (e.g., in the case of SQuAD).
To what extent can these results be explained by random noise?
More generally, how can we quantify the uncertainty of a set of experimental results when there are multiple sources of randomness?
In parallel to the MultiB ERT s release, we propose a more principled and standardized method to compare training procedures.
We recommend a non-parametric bootstrapping procedure, the “Multi-Bootstrap”, which enables us to make inference about model performance in the face of multiple sources of uncertainty: the randomness due to the pre-training seed, the ﬁne-tuning seed, and the ﬁnite test data.
The main idea is to use the average behavior over seeds as a means of summarizing expected behavior in an ideal world with inﬁnite samples.
Although we present Multi-Bootstrap in the context of analyzing the MultiBERTs, the method could be applied in all setups that involve a set of checkpoints pre-trained with the same method, a ﬁnite test set, and (possibly) multiple rounds of ﬁne-tuning.
The Multi-Bootstrap is implemented as a Python library, included with the MultiB ERT s release.
3. I NTERPRETING S TATISTICAL R ESULTS
The Multi-Bootstrap provides an estimate of the amount of remaining uncertainty when summarizing the performance over multiple seeds.
The following notation will help us state this precisely.
We assume access to model predictions f ( x ) for each instance x in the evaluation set.
We consider randomness arising from:
The choice of pre-training seed S ∼ M 2.
The choice of ﬁne-tuning seed T ∼ N 3.
The choice of test sample ( X, Y ) ∼ D
The Multi-Bootstrap procedure allows us to account for all of the above.
Speciﬁcally, MultiB ERT s enables us to estimate the variance due to the choice of pre-training seed (1), which would not be possible with a single artifact.
Note that multiple ﬁne-tuning runs are not required in order to use the procedure.
For each pre-training seed s , let f s ( x ) denote the learned model’s prediction on input features x and let L ( s ) denote the expected performance metric of f s on a test distribution D over features X and labels Y .
For example, the accuracy would be L ( s ) =
E [1 { Y = f
s ( X ) } ] .
We can use the test sample (which we will assume has n x examples) to estimate the performance for each of the seeds in MultiB ERT s, which we denote as b
L ( s ) .
The performance L ( s ) depends on the seed, but we are interested in summarizing the model over all seeds.
A natural summary is the average over seeds, E S ∼ M [ L ( S )] , which we will denote by θ .
Then, using n s independently sampled seeds, we can compute an estimate
[Equation]
Because b θ is computed under a ﬁnite evaluation set and ﬁnite number of seeds, it is necessary to quantify the uncertainty of the estimate.
The goal of Multi-Bootstrap is to estimate the distribution of the error in this estimate, b θ − θ , in order to compute conﬁdence intervals and test hypotheses about θ , such as whether it is above some threshold of interest.
Below, we describe a few common experimental designs in NLP that can be studied with these tools.
Design 1: Comparison to a Fixed Baseline.
In many use cases, we want to compare B ERT ’s behavior to that of a single, ﬁxed baseline.
For instance, does B ERT encode information about syntax as a feature-engineered model would (Tenney et al., 2019;
Hewitt & Manning, 2019)?
Does it encode social stereotypes, and how does it compare to human biases (Nadeem et al., 2021)?
Does it encode world knowledge, similarly to explicit knowledge bases (Petroni et al., 2019)?
Does another model such as RoBERTa (Liu et al., 2019) outperform B ERT on common tasks such as those from the GLUE benchmark?
In all these cases, we compare MultiB ERT s to some external baseline of which we only have a single estimate (e.g., random or human performance), or against an existing model that is not derived from the MultiB ERT s checkpoints.
We treat the baseline as ﬁxed, and assess only the uncertainty that arises from MultiB ERT s’ random seeds and the test examples.
Design 2: Paired Samples.
Alternatively, we might seek to assess the effectiveness of a speciﬁc intervention on model behavior.
In such studies, an intervention is proposed (e.g., representation learning via a speciﬁc intermediate task, or a speciﬁc architecture change) which can be applied to any pre-trained B ERT checkpoint.
The question is whether the procedure results in an improvement over the original B ERT pre-training method: does the intervention reliably produce the desired effect, or is the observed effect due to the idiosyncracies of a particular model artifact?
Examples of such studies include: Does intermediate tuning on NLI after pre-training make models more robust across language understanding tasks (Phang et al., 2018)?
Does pruning attention heads degrade model performance on downstream tasks (Voita et al., 2019)?
Does augmenting B ERT with information about semantic roles improve performance on benchmark tasks (Zhang et al., 2020)?
We refer to studies like the above as paired since each instance of the baseline model f s (which does not receive the intervention) can be paired with an instance of the proposed model f ′ s (which receives the stated intervention) such that f s and f ′ s are based on the same pretrained checkpoint produced using the same seed.
Denoting θ f and θ f ′ as the expected performance deﬁned above for the baseline and intervention model respectively, our goal is to test hypotheses about the true difference in performance δ
= θ f ′ − θ f using the estimated difference b δ
=
b θ f ′ − b θ f .
In a paired study, Multi-Bootstrap allows us to estimate both of the errors b θ f − θ f and b θ f ′ − θ f ′ , as well as the correlation between the two.
Together, these allow us to approximate the distribution of the overall estimation error b δ − δ
=
( b θ f − b θ f ′ )
− ( θ f − θ f ′ ) , between the estimate b δ and the truth δ .
With this, we can compute conﬁdence intervals for δ , the true average effect of the intervention on performance over seeds, and test hypotheses about δ , as well.
Design 3: Unpaired Samples.
Finally, we might seek to compare a number of seeds for both the intervention and baseline models, but may not expect them to be aligned in their dependence on the seed.
For example, the second model may use a different architecture so that they cannot be built from the same checkpoints, or the models may be generated from entirely separate initialization schemes.
We refer to such studies as unpaired .
Like in a paired study, the Multi-Bootstrap allows us to estimate the errors b θ f − θ f and b θ f ′ − θ f ′ ; however, in an unpaired study, we cannot estimate the correlation between the errors.
Thus, we assume that the correlation is zero.
This will give a conservative estimate of the error ( b θ f − b θ f ′ ) − ( θ f − θ f ′ ) , as long as b θ f − θ f and b θ f ′ − θ f ′ are not negatively correlated.
Since there is little reason to believe that the random seeds used for two different models would induce a negative correlation between the models’ performance, we take this assumption to be relatively safe.
Hypothesis Testing.
Given the measured uncertainty, we recommend testing whether or not the difference is meaningfully different from some arbitrary predeﬁned threshold (i.e., 0 in the typical case).
Speciﬁcally, we are often interested in rejecting the null hypothesis that the intervention does not improve over the baseline model, i.e.,
[Equation]
in a statistically rigorous way.
This can be done with the Multi-Bootstrap procedure described below.
3. M ULTI -B OOTSTRAP P ROCEDURE
The Multi-Bootstrap is a non-parametric bootstrapping procedure that allows us to estimate the distribution of the error b θ − θ over the seeds and test instances.
The algorithm supports both paired and unpaired study designs, differentiating the two settings only in the way the sampling is performed.
To keep the presentation simple, we will assume that the performance L ( s ) is an average of a perexample metric ℓ ( x, y, f s ) over the distribution D of ( X, Y ) , such as accuracy or the log likelihood, and b
L ( s ) is similarly an empirical average with the observed n x test examples,
[Equation]
We note that the mapping D 7→ L ( s ) is linear in D , which is required for our result in Theorem 1.
However, we conjecture that this is an artifact of the proof; like most bootstrap methods, the method here likely generalizes to any performance metric which behaves asymptotically like a linear mapping of D , including AUC, BLEU score (Papineni et al., 2002), and expected calibration error.
Building on the rich literature on bootstrap methods (e.g., Efron & Tibshirani, 1994), the MultiBootstrap is a new procedure which accounts for the way that the combined randomness from the seeds and test set creates error in the estimate b θ .
The statistical underpinnings of this approach have theoretical and methodological connections to inference procedures for two-sample tests (Van der
Vaart, 2000), where the samples from each population are independent.
However, in those settings, the test statistics naturally differ as a result of the scientiﬁc question at hand.
In our procedure, we generate a bootstrap sample from the full sample with replacement separately over both the randomness from the pre-training seed s and from the test set ( X, Y ) .
That is, we generate a sample of pre-training seeds ( S ∗ 1 , S ∗ 2 , . .
, S ∗ n s ) with each S ∗ j drawn randomly with replacement from the pre-training seeds, and we generate a test set sample (( X ∗ 1 , Y ∗ 1 ) , ( X ∗ 2 , Y ∗ 2 ) , . . .
, ( X ∗ n x , Y ∗ n x )) with each ( X, Y ) pair drawn randomly with replacement from the full test set.
Then, we compute the bootstrap estimate b θ ∗
[Equation]
∗ ( S
[Equation]
j ) , where
[Equation]
To illustrate the procedure, we present a minimal Python implementation in Appendix A. For sufﬁciently large n x and n s , the distribution of the estimation error b θ − θ is approximated well by the distribution of b θ ∗ − b θ over re-draws of the bootstrap samples, as stated precisely in Theorem 1.
Theorem 1.
Assume that E [ ℓ 2 ( X, Y, f S )] < ∞ .
Furthermore, assume that for each s , E [ ℓ 2 ( X, Y, f s )] < ∞ , and for almost every ( x, y ) pair, E [ ℓ 2 ( X, Y, f S ) |
X = x, Y
= y ] < ∞ .
Let n = n x + n s , and assume that 0 < p s = n s /n < 1 stays ﬁxed (up to rounding error) as n → ∞ .
Then, there exists 0 < σ 2 < ∞ such that √ n ( b θ − θ ) d
→ G with G ∼ N (0 , σ 2 ) .
Furthermore, conditionally on (( X 1 , Y 1 ) , ( X 2 , Y 2 ) , . . . ) , √ n ( [Equation]
The proof of Theorem 1 is in Appendix B, along with a comment on the rate of convergence for the approximation error.
The challenge with applying existing theory to our method is that while the seeds and data points are each marginally iid, the observed losses depend on both, and therefore are not iid.
Therefore, we need to handle this non-iid structure in our method and proof.
For nested sources of randomness (e.g., if for each pre-training seed s , we have estimates from multiple ﬁne-tuning seeds), we average over all of the inner samples (ﬁne-tuning seeds) in every bootstrap sample, motivated by Field & Welsh (2007)’s recommendations for bootstrapping clustered data.
Paired Samples (design 2, continued).
In a paired design, the Multi-Bootstrap procedure can additionally tell us the joint distribution of [Equation]
To do so, one must use the same bootstrap samples of the seeds ( S ∗ 1 , S ∗ 2 , . .
, S ∗ n s ) and test examples (( X ∗ 1 , Y ∗ 1 ) , ( X ∗ 2 , Y ∗ 2 ) , . . .
, ( X ∗ n x , Y ∗ n x )) for both models.
Then, the correlation between the errors b θ f ′ − θ f ′ and b θ f − θ f is well approximated by the correlation between the bootstrap errors
[Equation]
and [Equation]
In particular, recall that we deﬁned the difference in performance between the intervention f ′ and the baseline f to be δ , and deﬁned its estimator to be b δ .
With the Multi-Bootstrap, we can estimate the bootstrapped difference
[Equation]
With this, the distribution of the estimation error b δ − δ is well approximated by the distribution of b δ ∗ − b δ over bootstrap samples.
Unpaired Samples (design 3, continued).
For studies that do not match the paired format, we adapt the Multi-Bootstrap procedure so that, instead of sampling a single pre-training seed that is shared between f and f ′ , we sample pre-training seeds for each one independently.
The remainder of the algorithm proceeds as in the paired case.
Relative to the paired design discussed above, this additionally assumes that the errors due to differences in pre-training seed between b θ f ′ − θ f ′ and b θ f − θ f are independent.
Comparison to a Fixed Baseline (design 1, continued).
Often, we do not have access to multiple estimates of L ( s ) , for example, when the baseline f against which we are comparing is an estimate of human performance for which only mean accuracy was reported, or when f is the performance of a previously-published model for which there only exists a single artifact or for which we do not have direct access to model predictions.
When we have only a point estimate b θ f = b
L ( S 1 ) of θ f for the baseline f with a single seed S 1 , we recommend using Multi-Bootstrap to compute a conﬁdence interval around θ f ′ and reporting where the given estimate of baseline performance falls within that distribution.
An example of such a case is Figure 1, in which the distribution of MultiB ERT s performance is compared to that from the single checkpoint of the original B ERT release.
In general such results should be interpreted conservatively, as we cannot make any claims about the variance of the baseline model.
Hypothesis Testing.
A valid p -value for the hypothesis test described in Equation 1 is the fraction of bootstrap samples from the above procedure for which the estimate b δ is negative.
A PPLICATION : G ENDER B IAS IN C OREFERENCE S YSTEMS
We present a case study to illustrate how MultiB ERT s and the Multi-Bootstrap can help us draw more robust conclusions about model behavior.
The use case is based on gendered correlations.
For a particular measure of gender bias, we take a single B ERT checkpoint and measure a value of 0.35.
We then apply an intervention, foo , designed to reduce this correlation, and measure 0.25.
In an effort to do even better, we create a whole new checkpoint by applying the foo procedure from the very beginning of pre-training.
On this checkpoint, we measure 0.3.
How does one make sense of this result?
As a concrete example, we analyze gender bias in coreference systems (Rudinger et al., 2018) and showing how MultiB ERT s and the Multi-Bootstrap can help us understand the effect of an intervention, counterfactual data augmentation (CDA).
We follow a set-up similar to Webster et al.
(2020), which augments the B ERT pretraining data with counterfactual sentences created by randomly swapping English binary-gendered pronouns.
The goal is to weaken the correlation between gendered pronouns and other words such as occupation terms (e.g., doctor, nurse).
We compare our baseline
MultiB ERT s models to two strategies for CDA.
In the ﬁrst ( CDA-incr ), we continue pre-training each MultiB ERT s model for an additional 50K steps on the counterfactual data of Webster et al.
In the second, we train BERT models from scratch ( CDA-full ) on the same dataset.
The Winogender dataset consists of template sentences covering 60 occupation terms and instantiated with either male, female, or neutral pronouns.
We follow Webster et al.
and train a gold-mention coreference system using a two-layer feedforward network that takes span representations from a frozen BERT encoder as input and makes binary predictions for mention-referent pairs.
The model is trained on OntoNotes (Hovy et al., 2006) and evaluated on the Winogender examples for both per-sentence accuracy and a bias score, deﬁned as the Pearson correlation between the peroccupation bias score (Figure 4 of Rudinger et al. 2018) and the occupational gender statistics from the U.S. Bureau of Labor Statistics.
For each pre-training run, we train ﬁve coreference models, using the same encoder but different random seeds to initialize the classiﬁer weights and to shufﬂe the training data.
4. P AIRED ANALYSIS : CDA- INCR VS . B ASE
We investigate the impact of the intervention on performance and bias.
Overall accuracy is fairly consistent across pre-training seeds, at 62.6 ± 1.2% for the base model, with only a small and not statistically signiﬁcant change under CDA-incr (Table 1).
However, as shown in Figure 3, there is considerable variation in bias correlation, with r values between 0.1 and 0.7 depending on pretraining seed.
The range for CDA-incr overlaps somewhat, with values between 0.0 and 0.4; however, because the incremental CDA is an intervention on each base checkpoint, we can look at the individual seeds and see that in most cases there appears to be a signiﬁcant improvement.
A paired
Multi-Bootstrap allows us to quantify this and further account for noise due to the ﬁnite evaluation sample of 60 occupations.
The results are shown in Table 1, which show that CDA-incr signiﬁcantly reduces bias by
[Table]
δ = − 0 .
162 with p = 0 .
[Table]
[Table]
4. U NPAIRED ANALYSIS : CDA- FULL VS . CDA- INCR
We can also test if we get any additional beneﬁt from running the entire pre-training with counterfactually-augmented data.
Similar to MultiB ERT s, we trained 25 CDA-full checkpoints for 2M steps on the CDA dataset.
Because these are entirely new checkpoints, independent from the base MultiB ERT s runs, we use an unpaired version of the Multi-Bootstrap, which uses the same set of examples but samples pretraining seeds independently for CDA-incr and CDA-full.
As shown in Table 2, overall accuracy does not change appreciably (0.622 vs. 0.623, p = 0 . 416 ), while bias correlation seems to decrease but not signiﬁcantly (0.256 vs 0.192, δ = -0.064 with p = 0 . 132 ).
As an ablation, we also experiment with sampling over either only seeds (taking the set of examples, i.e. occupations, as ﬁxed), or over examples (taking the set of 25 seeds as ﬁxed).
As shown in
Table 2, we ﬁnd lower p -values (0.005 and 0.053) in both cases—showing that failing to account for ﬁnite samples along either dimension could lead to overconﬁdent conclusions.
In Appendix E, we present two additional examples: a paired study where we increase pretraining time from 1M to 2M steps, as well as an unpaired comparison to the original bert-base-uncased checkpoint.
C ONCLUSION
To make progress on language model pre-training, it is essential to distinguish between the properties of speciﬁc model artifacts and those of the training procedures that generated them.
To this end, we have presented two resources: the MultiB ERT s, a set of 25 model checkpoints to support robust research on B ERT , and the Multi-Bootstrap, a non-parametric statistical method to estimate the uncertainty of model comparisons across multiple training seeds.
We demonstrated the utility of these resources by showing how to quantify the effect of an intervention to reduce a type of gender bias in coreference systems built on B ERT .
We hope that the release of multiple checkpoints and the use of principled hypothesis testing will become standard practices in research on pre-trained language models.
A M INIMAL I MPLEMENTATION OF THE M ULTI -B OOTSTRAP
Below, we present a simpliﬁed
Python implementation of the Multi-Bootstrap algorithm presented in Section 3.2.
It describes a single-sided version of the procedure, which could be used, e.g., to test that a model’s performance is greater than 0 .
The input is a matrix of predictions where row indices correspond to test examples and column indices to random seeds.
The functions returns an array of n boot samples [ b θ 1 , . . .
, b θ n boot ] .
1 def multibootstrap(predictions, labels, metric_fun, nboot): """
Generates bootstrap samples of a model’s performance.
Input:
13""" # Checks the data format.
n_samples, n_seeds = predictions.shape assert labels.shape =
= (n_samples,) 18thetas
= np.zeros( ) for boot_ix in range(nboot): # Samples n_samples test examples and n_seeds pre-training seeds. x_samples = np.random.choice(n_samples, size=n_samples, replace=True) s_samples
= np.random.choice(n_seeds, size=n_seeds, replace=True) 24# Computes the metric over the bootstrapping samples.
sampled_predictions = predictions[np.ix_(x_samples, s_samples)]
sampled_labels = labels[x_samples] sampled_metrics =
[ metric_fun(sampled_predictions[:,j], sampled_
We provide the complete version of the algorithm on our repository http://goo.gle/ multiberts .
Our implementation is optimized and supports all the experiment designs described in Section 3, including paired and unpaired analysis as well as multiple ﬁne-tuning runs for each pretraining seed.
B P ROOF OF T HEOREM
Before giving the proof, we deﬁne some useful notation that will simplify the argument considerably.
We let D n be the empirical measure over the n x observations ( Z i =
( X i , Y i )) ni
=1 , and M n be the empirical measure over the n s observations ( S j ) nj =1 .
For a function f : V → R and a distribution
P over V , we will use the shorthand Pf to denote the expectation of f under P ,
[Equation]
For example, this allows us to write
[Equation]
For the bootstrapped distributions, let D ∗ n denote the distribution over the bootstrap data samples ( Z ∗ 1 , Z ∗ 2 , . . .
, Z ∗ n x ) and M ∗ n denote the distribution over the bootstrapped seed samples, ( S ∗ 1 , S ∗ 2 , . .
, S ∗ n s ) , both conditional on the observed samples ( Z i ) n x i =1 and ( S j ) n s j =1 .
Note that the empirical average over a bootstrapped sample
[Equation]
can be written as
[Equation]
where A i is the number of times Z i appears in the bootstrapped sample ( Z ∗ k ) n x k =1 , and B j is the number of times S j appears in the bootstrapped sample ( S ∗ k ) n s k =1 .
With this in mind, we will abuse notation, and also denote D ∗ n as the distribution over the A i and M ∗ n as the distribution over the B j .
Finally, we will use E ∗ and Var ∗ to denote the expectation and variance of random variables deﬁned with respect to D ∗ n or M ∗ n , conditional on D n and M n .
We will use P to denote the distribution P = D × M .
Throughout, all assertions made with respect to random variables made without a note about their probability of occurrence hold P -almost surely.
Proof.
The challenge with applying existing theory to our method is that because the performance metric ( ℓ ( Z i , f S j ) n x i =1 over the n x observations for a given seed S j all depend on the same S j , they are not independent.
Similarly for the performance on a given observation, over seeds.
Therefore, we need to handle this non-iid structure in our proof for the multi-bootstrap.
There are conceptually three steps to our proof that allow us to do just that.
The ﬁrst is to show that b θ has an asymptotically linear representation as
[Equation]
The second is to show that conditional on D n and M n the multi-bootstrapped statistic
[Equation]
has an asymptotically linear representation as
[Equation]
where D ◦ n and M ◦ n are multiplier bootstrap samples coupled to the bootstrap D ∗ n and M ∗ n which we deﬁne formally in the beginning of Step 2.
The third step is to use standard results for the multiplier bootstrap of the mean of iid data to show that the distributions of the above linearized statistics converge to the same limit.
Because we have assumed that ℓ ( Z, f S ) < ∞ , E [ ℓ ( Z, f S ) | S ] < ∞ , and E [ ℓ ( Z, f S ) | Z ] < ∞ , Fubini’s theorem allows us to switch the order of integration over Z and S as needed.
We will assume that DMℓ ( X, Y, f S ) = 0 .
This is without loss of generality, because adding and subtracting √ nDMℓ to the bootstrap expression gives
[Equation]
so if we prove that the result holds with the mean zero assumption, it will imply that the result holds for ℓ with a nonzero mean.
This theorem guarantees consistency of the Multi-Bootstrap estimates.
One question that comes up is whether it is possible to get meaningful / tight rates of convergence for the approximation.
Unfortunately, getting O P (1 /n ) convergence as found in many bootstrap methods (Van der Vaart, 2000) is difﬁcult without the use of Edgeworth expansions, by which the Multi-Bootstrap is not welladapted to analysis.
That said, many of the remainder terms already have variance of order O (1 /n ) , or could easily be adapted to the same, suggesting an O P (1 / √ n ) convergence.
The main difﬁculty, however, is showing rates of convergence for the strong law on separately exchangeable arrays (see the proof of Lemmas 2, 4-5).
Showing a weaker notion of convergence, such as in probability, may perhaps allow one to show that the remainder is O P (1 / √ n ) , however the adaptation of the aforementioned Lemmas is nontrivial.
Step 1 Recalling that
[Equation]
and θ ∆ = DMℓ , we can expand √ n (
[Equation]
as follows,
[Equation]
The following lemma shows that √ n ( D n − D )( M n − M ) ℓ is a lower order term.
Lemma 1.
Under the assumptions of Theorem 1, √ n ( D n − D )( M n −
[Equation]
ℓ =
[Equation]
Therefore,
[Equation]
Step 2 One of the challenges with working with the bootstrap sample D ∗ n and M ∗ n is that the induced per-sample weights { A i } n x i =1 and { B j } n s j =1 do not have independent components, because they each follow a multinomial distribution over n x items and n s items, respectively.
However, they are close enough to independent that we can deﬁne a coupled set of random variables { A ◦ i } n x i =1 and { B ◦ j } n s j =1 that do have independent components, but behave similarly enough to { A i } and { B j } that using these weights has a negligible effect on distribution of the bootstrapped estimator, as described concretely below.
First, we discuss the coupled multiplier bootstrap sample D ◦ n and M ◦ n .
The creation of this sequence, called “Poissonization” is a standard technique for proving results about the empirical bootstrap that require independence of the bootstrap weights (van der Vaart et al., 1996).
We describe this for D ◦ n as the idea is identical for M ◦ n .
Because our goal is to couple this distribution to D ∗ n , we deﬁne it on the same sample space, and extend the distribution P ∗ , expectation E ∗ and variance Var ∗ to be over
D ◦ n and M ◦ n , conditionally on D n and M n , as with D ∗ n and M ∗ n .
To construct the distribution D ◦ n , from the empirical distribution D n and a bootstrap sample D ∗ n , start with the distribution D ∗ n and modify it as follows: We draw a Poisson random variable N n x with mean n x .
If N n x > n x , then we sample N n x − n x iid observations from D n , with replacement, and add them to the bootstrap sample initialized with D ∗ n to produce the distribution D ◦ n .
If N n x < n x , we sample n x − N n x observations from D ∗ n , without replacement, and remove them from the bootstrap sample to produce the distribution D ◦ n .
If N n x = n x , then D ◦ n = D ∗ n .
Recalling that A i is the number of times the i -th sample is included in D ∗ n , similarly deﬁne A ◦ i as the number of times the i -th sample is included in D ◦ n .
Note that by the properties of the Poisson distribution,
A ◦ i ∼ Poisson(1) , and { A ◦ i } n x i =1 are independent.
Note that the natural normalization for D ◦ n would be N n x .
However, it will be useful to maintain the normalization by n x , so abusing notation, for a function f ( z ) , we will say that D ◦ n f = 1 n x
[Equation]
Deﬁne b θ ◦ as the following empirical estimator of θ under the distribution D ◦ n × M ◦ n ,
[Equation]
Lemma 2 shows that √ n ( b θ ∗ − b θ ◦
[Equation]
= o P ∗ (1) , and so √ n ( b
[Equation]
∗
[Equation]
Lemma 2.
Under the assumptions of Theorem 1, and that DMℓ = 0 , √
[Equation] [Equation]
With this, the expansion of √ n ( b θ ◦ − b θ ) begins mutatis mutandis the same as in Step 1, to get that √ n (
[Equation]
As with Step 1, we provide Lemma 3 showing that the remainder term √ n ( D ◦ n − D n )( M ◦ n − M n ) ℓ will be lower order.
Lemma 3.
Under the assumptions of Theorem 1, √ n ( D ◦ n − D n )( M ◦ n − M n ) ℓ =
o P ∗ (1) .
Therefore,
[Equation]
Then, to write √ n ( b θ ∗ − b θ ) in terms of √ n s ( M ◦ n − M n ) Dℓ as wanted in Eq.
(3), instead of √ n s ( M ◦ n −
M n ) D n ℓ , we must additionally show that the functional has enough continuity that the error term √ n s ( M ◦ n − M n )( D n − D ) ℓ is lower order.
The following lemma shows exactly this.
Lemma 4.
Under the assumptions of Theorem 1, conditionally on the sequences Z 1 , Z 2 , . . .
and
S 1 , S 2 , . . . ,
[Equation]
Altogether, these imply that
[Equation]
Step 3 Noting that Mℓ ( · , f S ) =
E D × M [ ℓ ( · , f S )
| Z = · ] is a real-valued random variable with ﬁnite variance (similarly for Dℓ ( Z, · ) ), and recalling that the n x samples used for D n and n s samples for M n satisfy n = n x / (1 − p s ) and n = n s /p s , for 0 <
p s < 1 , the conventional central limit theorem shows that for some positive semi-deﬁnite matrix Σ ∈ R 2 × 2 , and G ∼ N (0 , Σ ),
[Equation]
Note that D n and M n are independent, so G is, in fact, a diagonal matrix.
Additionally, the conditional multiplier CLT (van der Vaart et al., 1996, Lemma 2.9.5, pg. 181) implies that conditionally on Z 1 , Z 2 , . . .
and S 1 , S 2 , . .
[Equation]
Finally, applying the delta method (see Theorem 23.5 from Van der Vaart (2000)) along with the results from Steps 1 and 2 shows that the distributions of √ n ( b θ − θ ) and √ n ( b θ ∗ − b θ ) converge to
N (0 , σ 2 ) , where σ 2 = Σ 11 / (1 − p s ) + Σ 22 /p s .
B. P ROOF OF L EMMA
Fix ǫ > 0 .
Note that E [( D n − D )( M n − M ) ℓ ] = 0 , so by Chebyshev’s inequality,
[Equation]
Therefore, it sufﬁces to show that lim n →∞ Var( √ n ( D n − D )( M n − M ) ℓ ) = 0 .
To do so, we apply the law of total variance, conditioning on D n , and bound the resulting expression by C/n .
[Equation]

[Equation]
B. P ROOF OF L EMMA
First, note the following representation for b θ ∗ − b θ ◦ : b
[Equation]
| {
[Equation]
| {z }
[Equation]
Let ǫ > 0 .
Noting that E ∗ [ I 1 ]
=
E ∗ [ I 2 ] = 0 , applying Chebyshev’s inequality gives
P ∗
[Equation]
It sufﬁces to show that n Var ∗ ( I 1 ) →
0 and n Var ∗ ( I 2 ) → 0 .
The arguments for each term are mutatis mutandis the same, and so we proceed by showing the proof for I 2 .
By the law of total variance,
[Equation]
Because E ∗ [ A i ] = E ∗ [ A ◦ i ] and { B j } n s j =1 ⊥⊥
A
i , A ◦ i , it follows that E ∗
[ I 2 | { B j } n s j =1 ] =
Taking the remaining term and re-organizing the sums in I
[Equation]
Next, we apply the law of total variance again, conditioning on N n x =
[Equation]
◦ i .
First,
[Equation]
and so
Var ∗
[Equation]
Then, conditionally on N n x (and { B j } ), I 2 is the (centered) empirical average of | N n − n | samples from a ﬁnite population of size n , rescaled by | N n − n | /n .
Therefore, applying Theorem 2.2 of Cochran (2007) gives the conditional variance as [Equation] 
[Equation]
| {z
[Equation]
To take the expectation over N n x , notice that because E ∗ [ N n x ] = n x , this is the mean absolute deviation (MAD) of N n x .
Using the expression for the MAD of a Poisson variable from Ramasubban (1958) gives
[Equation]
and using Stirling’s approximation, this is bounded by C √ n x , for some 0 < C < ∞ .
Combining this with the above term for the variance of the conditional expectation, we have
Var ∗
[Equation]
Noting that E ∗ [ B 2 j ]
= E ∗ [ B j B k ] = 1 , we get the following bound:
[Equation]
where
[Equation]
Because of the assumption that DMℓ = 0 , the SLLN adapted to separately exchangeable arrays (Rieders, 1991, Theorem 1.4) implies that
[Equation]
almost surely.
Therefore, the ﬁrst term of (5) is o (1 /n ) .
Note that ¯ V 2 is the empirical variance of the conditional expectation of ℓ ( Z i , f S j ) given { Z i } ni =1 .
Therefore, the law of total variance shows that
[Equation]
By the SLLN adapted to separately exchangeable arrays (Rieders, 1991, Theorem 1.4), both of the terms converge almost surely to DMℓ 2 < ∞ and ( DMℓ ) 2 , respectively.
and therefore,
[Equation]
B. P ROOF OF L EMMA
As with Lemma 1, the main idea of the proof is to apply Chebyshev’s inequality, and show that the variance tends to zero.
Indeed, choosing an arbitrary ǫ > 0 ,
[Equation]
Therefore, it sufﬁces to show that the variance in the above display goes to zero.
To do this, we start by re-writing the expression in terms of A ◦ i and B ◦ j , and then apply the law of total variance.
[Equation]
Because { B ◦ j } n s j =1 are independent of { A ◦ i } n x i =1 , and have mean 1 , the conditional expectation in the ﬁrst term is 0 almost surely.
Expanding out the second term, using that Var ∗ ( B ◦ j ) = [Equation] and that the { B ◦ j } n s j =1 are uncorrelated,
[Equation]
Now, noting that Var ∗ ( A ◦ i ) = 1 , and that the { A ◦ i } n x i =1 are uncorrelated, this simpliﬁes to
[Equation]
Because E D × M [ ℓ 2 ( Z, f S )] < ∞ , the SLLN adapted to separately exchangeable arrays (Rieders, 1991, Theorem 1.4) implies that this converges almost surely to 0 .
B. P ROOF OF L EMMA
We prove (a) of the Lemma, as (b) follows from applying Fubini’s theorem and following mutatis mutandis the same argument.
Without loss of generality, we will assume that ℓ ( Z i , f S j ) ≥ 0 .
Because Var( ℓ ( Z i , f S j )) < ∞ , we can always decompose ℓ ( · , · ) into a positive and negative part, and show that the result holds for each individually.
Once again, we prove (a) by turning to Chebyshev’s inequality.
Fix ǫ > 0 , and observe that
[Equation]
so it is sufﬁcient to show that Var ∗
[Equation]
Writing the above in terms of A ◦ i , we have
[Equation]
Now, we want to show that the last display converges almost surely to 0.
Notice that each term within the outer sum will obviously converge due to the SLLN.
Showing that the outer sum also converges almost surely is technically difﬁcult, but conceptually follows the same argument used to prove
the
SLLN (speciﬁcally, we follow the one done elegantly by Etemadi (1981); Luzia (2018) provides a more detailed account of this proof technique that is helpful for developing a deeper understanding).
We show the following version of almost sure convergence: that for any ǫ > 0 ,
[Equation]
where i.o. stands for inﬁnitely often.
Deﬁne the shorthand L ij = ℓ
( Z i , f S j ) and let ¯ L ij =
L ij 1 { L ij < ij } be a truncated version of
L ij .
The proof of Theorem 2 of Etemadi (1981) implies that P ( ¯ L ij 6 =
L ij i.o. )
= 0 , because the assumption Var( L ij ) < ∞ implies the assumption used in Etemadi (1981), and independence of { L ij } i,j is not needed for this result.
Therefore,
[Equation]
Together, these imply that if we can prove that the truncated sum converges, ie.,
[Equation]
this is sufﬁcient to show that the un-truncated version converges almost surely.
To prove (6), we show two things: ﬁrst, that there is a subsequence k n such that (6) holds when restricted to the subsequence, and then we show that the sequence is a Cauchy sequence, which together imply the result.
Let α > 1 and let k n = α n .
For convenience, denote k nx as the number of data samples and k ns as the number of seed samples when k nx + k ns =
k n total samples are drawn.
We will ignore integer rounding issues, and assume k nx
=
(1 − p s ) α n , and k ns =
p s α n .
The following lemma shows that the subsequence deﬁned by k n converges almost surely.
Lemma 5.
Let α > 1 , and k n = α n .
Under the assumptions of Theorem 1 and that L ij ≥
[Equation]
We now must show that the sequence in (6) is a Cauchy sequence.
Note that the SLLN implies that
[Equation]
and the LLN for exchangeable arrays (Rieders, 1991, Theorem 1.4) implies that
[Equation]
Therefore,
[Equation]
Notice that because ¯ L ij ≥ 0 , the sum
P
[Equation]
L
[Equation]
(cid:17) 2 is monotone increasing in n s and n x .
With this in mind, for any m > 0 , let n be such that k n ≤ m < k n +1 .
Then, by the montonicity, (cid:18) k n
[Equation]
From (7), the left hand side converges to 1 α 3 E [ E [¯ L ij | Z i ] 2 ] , and the right hand side converges to α 3 E [ E [¯ L ij | Z i ] 2 ] .
Because α is arbitrary, this proves that the sequence
[Equation]
is almost surely Cauchy.
Together with Lemma 5, this implies (6).
B.
P ROOF OF L EMMA
We will show that
[Equation]
This, along with the ﬁrst Borel-Cantelli lemma (´Emile Borel, 1909; Cantelli, 1917) implies the result.
Applying Markov’s inequality and using the fact that ¯ L ij and ¯ L ih are independent conditional on Z i gives
[Equation]
where the last line follows from the law of total variance.
To simplify the remaining algebra, we will use a .
b to denote that there is some constant 0 < c < ∞ such that a < cb .
Continuing, we have
[Equation]
where n ( i, j ) is shorthand for n ( i, j )
= log α max { i/ (1 − p s ) , j/p s } is the ﬁrst n such that k nx ≥ i and k ns ≥ j .
Now, deﬁne Q as the distribution of L 11 induced by Z 1 and S 1 .
Additionally, split the inner sum into two pieces, one for when j < i and so max { i, j } = i and one for when j ≥ i and so max { i, j } = j .
[Equation]
 switching the order of the indices over j and k , using that 1 ≤ k ≤ ij and the constraints on j relative to i ,
[Equation]
Switching the order of summation over i and k , and separating out the terms where k/i <
i and k/i ≥ i ,
[Equation]
k
[Equation]
  .ǫ ∞ X k =  √ k Z k k −x 2 d Q
[Equation]
x ) ! .ǫ
∞
{+X k =
 Z k k −x√ x d Q ( x ) ! .ǫ Z ∞ x 1 . 5 d Q ( x ) < ∞ .+}
C I NSTANCE -L EVEL A GREEMENT OF M ULTI B ERTS ON GLUE
We present additional performance experiments to complement Section 2.
Table 3 shows per-example agreement rates on GLUE predictions between pairs of models pretrained with a single seed (“same”) and pairs pre-trained with different seeds (“diff”); in all cases, models are ﬁne-tuned with different seeds.
With the exception of RTE, we see high agreement (over 90%) on test examples drawn from the same distribution as the training data, and note that agreement is 1–2% lower on average for the predictions of models pre-trained on different seeds compared to models pre-trained on the same seed.
However, this discrepancy becomes signiﬁcantly more pronounced if we look at out-of-domain “challenge sets” which feature a different data distribution from the training set.
For example, if we evaluate our MNLI models on the anti-sterotypical examples from HANS (McCoy et al., 2019) [Table] we see agreement drop from 88% to 82% when comparing across pre-training seeds.
Figure 4 shows how this can affect overall accuracy, which can vary over a range of nearly 20% depending on the pre-training seed.
Such results underscore the need to evaluate multiple pre-training runs, especially when evaluating a model’s ability to generalize outside of its training distribution. [Table] [Figure] [Table]
[Figure]
[Table]
[Table]
D C ROSS -S EED V ARIATION
Figure 5 shows variation in Winogender bias correlation (S4) between each MultiB ERT s pretraining seed.
Each box shows the distribution over ﬁve runs, and some of the variation between seeds may simple be due to variation in training the coreference model.
If we average the scores for each seed then look at the distribution of this per-seed average score, we get 0.45 ± 0.11.
What if pretraining didn’t matter?
If we ignore the seed and randomly sample sets of ﬁve runs from this set with replacement, we get scores of 0.45 ± 0.05 - telling us that most of the variance can only be explained by differences between the pretraining checkpoints.
We can conﬁrm this by taking a subset of our pretraining seeds and training additional 25 randomlyinitialized coreference models.
Figure 6 shows the result: seeds 0, 2, 3, and 4 appear closer together than in Figure 5, but seed 1 clearly has different properties with respect to our Winogender metric.
We can conﬁrm this with an unpaired multibootstrap analysis, taking seed 0 as base and seed 1 as experiment: we observe a signiﬁcant effect of δ = 0 .
203 ( p = 0 . 009 ), as shown in Table 4.
[Table]
E C ASE S TUDY : M ULTI BERT S VS . O RIGINAL B ERT
As an additional example of application, we discuss challenges in reproducing the performance of the original B ERT checkpoint, using the Multi-Bootstrap procedure.
The original bert-base-uncased checkpoint appears to be an outlier when viewed against the distribution of scores obtained using the MultiB ERT s reproductions.
Speciﬁcally, in reproducing the training recipe of Devlin et al.
(2019), we found it difﬁcult to simultaneously match performance on all tasks using a single set of hyperparameters.
Devlin et al.
(2019) reports training for 1M steps.
However, as shown in Figure 1 and 2, models pre-trained for 1M steps matched the original checkpoint on SQuAD but lagged behind on GLUE tasks; if pre-training continues to 2M steps,
GLUE performance matches the original checkpoint but SQuAD performance is signiﬁcantly higher.
The above observations suggest two separate but related hypotheses (below) about the B ERT pretraining procedure.
On most tasks, running B ERT pre-training for 2M steps produces better models than 1M steps.
The MultiB ERT s training procedure outperforms
the original B ERT procedure on SQuAD.
Let us use the Multi-Bootstrap to test these hypotheses.
E. H OW MANY STEPS TO PRETRAIN ?
Let f be the predictor induced by the B ERT pre-training procedure using the default 1M steps, and let f ′ be the predictor resulting from the proposed intervention of training to 2M steps.
From a glance at the histograms in Figure 8, we can see that MNLI appears to be a case where 2M is generally better, while MRPC and RTE appear less conclusive.
With the MultiB ERT s, we can test the signiﬁcance of the results.
The results are shown in Table 5.
We ﬁnd that MNLI conclusively performs better ( δ = 0 .
007 with p < 0 .
001 ) with 2M steps; for RTE and MRPC we cannot reject the null hypothesis of no difference ( p = 0 .
14 and p = 0 .
56 respectively).
As an example of the utility of this procedure, Figure 7 shows the distribution of individual samples of L for the intervention f ′ and baseline f from this bootstrap procedure (which we denote as L ′ and L , respectively).
The distributions overlap signiﬁcantly, but the samples are highly correlated due to the paired sampling, and we ﬁnd that individual samples of the difference ( L ′ − L ) are nearly always positive.
E. D OES THE M ULTI B ERTS PROCEDURE OUTPERFORM ORIGINAL B ERT ON SQ U AD?
To test our second hypothesis, i.e., that the MultiB ERT s procedure outperforms original B ERT on
SQuAD, we must use the unpaired Multi-Bootstrap procedure.
In particular, we are limited to the case in which we only have a point estimate of L ′ ( S ) , because we only have a single estimate of the performance of our baseline model f ′ (the original B ERT checkpoint).
However, the MultiBootstrap procedure still allows us to estimate variance across our MultiB ERT s seeds and across the examples in the evaluation set.
On SQuAD 2.0, we ﬁnd that the MultiB ERT s models trained for 2M steps outperform original B ERT with a 95% conﬁdence range of 1.9% to 2.9% and p < 0 .
001 for the null hypothesis, corroborating our intuition from Figure 2.