Efﬁcient transfer learning for NLP with ELECTRA
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility Summary
Scope of Reproducibility
Clark et al.
[2020] claims that the ELECTRA approach is highly efﬁcient in NLP performances relative to computationbudget.
As such, this study focus on this claim, summarized by the following question: Can we use ELECTRA toachieve close to SOTA performances for NLP in low-resource settings, in term of compute cost?
Methodology
This replication study has been conducted by fully reimplementing the small variant of the original ELECTRA model(Clark et al.
All experiments are performed on single GPU computers.
GLUE benchmark dev set (Wang et al.[2018]) is used for models evaluation and compared with the original paper.
Results
My results are similar to the original ELECTRA’s implementation (Clark et al.
[2020]), despite minor differencescompared to the original paper for both implementations.
With only 14M parameters, ELECTRA-Small outperforms,in absolute performances, concurrent pretraining approaches from some previous SOTA, such as GPT, or alternativeefﬁcient approaches using knowledge distillation, such as DistilBERT.
By taking into account compute cost, ELECTRAis clearly outperforming all compared approaches, including BERT and TinyBERT.
Therefore, this work supports theclaim that ELECTRA achieves high level of performances in low-resource settings, in term of compute cost.
Furthermore, with an increased generator capacity than recommended by Clark et al.
[2020], the discriminant cancollapses by being unable to distinguish if inputs are fake or not.
Thus, while ELECTRA is easier to train than GAN(Goodfellow et al.
[2014]), it appears to be sensitive to capacity allocation between generator and discriminator.
The code and a pretrained model will be released.
What was easy
Information provided by the authors of the original paper (Clark et al.
[2020]), either from the paper; within the sourcecode: or from the ofﬁcial Github repository, is very rich and exhaustive to understand the proposed approach.
In addition, as stated with their main claim, ELECTRA can be easily run on a single GPU
What was difﬁcult
By being an aggregation of several tasks and the variance from results, GLUE benchmark requires signiﬁcant amountof effort, in term of implementation and computation.
For models comparison, several tricks can also inﬂuence theresults, which is even more ampliﬁed by the different aggregation formulas and the lack of measure of dispersion forpublished results.
As such, conﬁrming the correctness of this reimplementation was harder than expected.
Communication with original authors
Kevin Clark, one of the original authors, has been helpful by answering some questions.
Unfortunately, breakdown of
GLUE score per tasks have not yet been provided to fully compare this implementation with the original one.
Otherwise, most of questions that I had were already answered though the Github repository or by inspecting the source code
Introduction
Over the recent years, the combination of pretraining task with large unlabelled text corpus has shown a lot of successfor Natural Language Processing (NLP) tasks (Howard and Ruder [2018], Peters et al.
[2018], Devlin et al.
[2019], Liuet al.
[2019], Radford et al.
However, the main drawback is the computation requirements, which limits theadoption of these approaches for more specialized domains for some potential use cases.
Unlike the recent trend of increasing computing requirements, in the original paper ELECTRA (Clark et al.
[2020]), theauthors are taking the opposite research direction to have more efﬁcient approach for transfer learning for NLP tasks.
Thus, the authors provide a way to democratize the use of deep learning for more actors and more use cases.
Scope of reproducibility
The scope for this reproducibility work is related to the following questions, which are part of the original paper (Clarket al.
Can we use ELECTRA pretraining task to achieve close to SOTA performances on low-resource settings, in termof compute cost?
To formalize this claim, we use the GLUE benchmark (Wang et al.
[2018]) as measure of NLP performances andcompare it with large pretrained models, such as GPT (Radford et al.
All experiments are executed on single
GPU computers.
How does the training process behave with different generator size ?
While ELECTRA shares some aspects from GAN (Goodfellow et al.
[2014]), with the use of discriminator and generatornetworks, the training is not adversarial.
Nevertheless, the capacity of the generator compared to the discriminatorremains an important hyper-parameter in the architecture.
Methodology
The approach for this work is to fully reimplement in PyTorch the small variant of ELECTRA (Clark et al.
Forthis, the different libraries from HuggingFace (Wolf et al.
[2020]), namely Transformers; Tokenizers; and Datasets, havebeen used respectively for the Transformer (Vaswani et al.
[2017]) implementation and training loop logic; tokenizationmodels; and to ease the access to relevant datasets.
Due to constraint of compute resources availability initially, some changes have been implemented in this reimplementation, such as the ability to use gradient accumulation.
Furthermore, to ease further explorations, as the preprocessingsteps are time consuming, approximately 3 hours, the reimplementation’s preprocessing has been adapted to be lessdependant on the sequence length, see section 3.2 for the details.
3.1 Model descriptions
ELECTRA-Small is a complete reimplementation in PyTorch of ELECTRA-Small*, original implementation in
TensorFlow from Clark et al.
The architecture consists in two Transformer based neural networks:
• The generator network transforms a sequence of tokens, containing masked tokens, into a new sequence with
the original tokens predictions for these masked tokens.
• The discriminator network transforms a sequence of tokens, containing tokens replaced by a generator, into a
new sequence of binary predictions, true if the token has been replaced, otherwise false.
Similar to ELECTRA-Small* (Clark et al.
[2020]), input embeddings are shared between the generator and thediscriminator models.
Moreover, input token embeddings and output token embeddings are shared for the generatormodel.
Position embeddings are also used similar to BERT (Devlin et al.
[2019]) in both ELECTRA implementations.
For the downstream tasks only, both implementations use a 2 layer MLP heads.
The original implementation (Clarket al.
[2020]) uses the ﬁrst contextualized embeddings for the downstream head.
However, this reimplementation usesaverage pooling across all contextualized embeddings.
For a visual representation, please refer to the ﬁgure 1.
Generator is only used at pretraining.
Average pooling and MLP are only used for downstream tasks.
Note: the original implementation (Clark et al.
[2020]) uses ﬁrst embedding for pooling layer instead of average pooling.
Image inspired by the original paper (Clark et al.
3.2 Datasets
Pretraining task: The original paper (Clark et al.
[2020]) uses the same datasets and preprocessing steps as BERT(Devlin et al.
[2018]), namely Wikipedia and BookCorpus (Zhu et al.
[2015]), also referred as Wikibooks.
In theofﬁcial GitHub repository 1 , the authors use also OpenWebText, (Gokaslan and Cohen [2019]), referred as OWT.
Thepreprocessing steps consist to extract text segments satisfying the maximum sequence length from large text corpus.
Segments are then tokenized using WordPiece (Wu et al.
[2016]) with a vocabulary size 30,522.
For each training input,two tokenized segments are concatenated with the special token <
SEP> and the special token <
CLS> is added in theﬁrst position.
In addition of this input, a sequence id is provided to identify the segment, either 0 or 1, and a position idis also provided to give spacial information into inputs.
For this reproducibility study, the dataset for the pretraining task is OpenWebText (Gokaslan and Cohen [2019]).
Forthe preprocessing, the reimplementation uses a slight variation from the original paper.
Indeed, all documents aretokenized ofﬂine using Byte-level Byte Pair Encoding (Radford et al.
[2019]) with a vocabulary size 30,522.
Duringtraining, for each document, a random segment with the maximum sequence length allowed by the model is selected.
Inother words, this reimplementation uses a dynamic segmentation instead of precomputed static segments in the originalimplementation.
Downstream task: Similar to the original paper (Clark et al.
[2020]), the downstream tasks are the GLUE benchmark(Wang
et al.
The preprocessing steps are mostly similar to the original paper.
For single sentence tasks, inputsconsist of the single sentence for each training input.
For double sentences tasks, both sentences are concatenated withthe special token <
SEP>.
Finally, inputs are potentially truncated to ensure the maximum sequence length.
3.3 Hyperparameters
The selected hyperparameters are the same as the original paper, except for the optimization algorithm and the use ofgradient accumulation.
For the optimizer, AdamW (Loshchilov and Hutter [2019]) has been used instead of Adam(Kingma and Ba [2014]).
In addition, the original implementation is actually using a lower layer-wise learning rate thandescribed in Clark et al.
For more information, please refer to the tables 5 and 6.
3.4 Experimental setup
Pretrained models were trained on a single GPU computer, with Nvidia RTX 3090.
Regarding the downstream tasks,experiments were either run on Google Colab Pro instances, with Nvidia P100 or V100.
Due to the heterogeneous
The overall total is less than sum due
[Table]
infrastructure, a key element of the experimental setup is the centralization of results with Weights & Biases (Biewald[2020]).
For further details, please refer to the Github repository 2 .
3.5 Computational requirements
The pretraining task and the downstream tasks can be performed with a relative limited GPU memory budget and asingle GPU, thanks to gradient accumulation and mixed precision.
For example, pretraining can be done with a single
GTX 1060 6GB in 21 days; or on GTX 1080 Ti 11GB in 13 days instead of 3.75 days on RTX 3090 24GB.
Interestingly, by constraining the GPU memory usages to 16GB, minimum amount for the GPU used in the originalpaper (Clark et al.
[2020]), the duration for the pretraining task is similar to the original implementation (Clark et al.[2020]), meaning 3.75 days on RTX 3090 instead of 4 days on V100.
Using all GPU memory for the RTX 3090 wouldhave decrease the computation requirements, but it would have complicated the comparison between reimplementations.
Computational requirements, per experiments and overall, are summarized in the table 1.
Experiments
4.1 Reproducing ELECTRA on GLUE benchmark
In this experiment, one model is pretrained with 1 million steps on OpenWebText (Gokaslan and Cohen [2019]) withthe similar approach as the original paper (Clark et al.
With the weights initialization from this pretrainedmodel, the downstream task, the GLUE benchmark (Wang et al.
[2018]), is executed 5 times with different seeds.
Theresults are summarized in the table 2.
Reimplementation compared to the original: This reimplementation provides results, which are fairly similar to theoriginal implementation (Clark et al.
[2020]) on OpenWebText dataset for individual GLUE tasks.
Furthermore, thetraining dynamics, captured by the different metrics from the generator and the discriminator, are very similar to the oneprovided in the ofﬁcial ELECTRA’s Github repository, see ﬁgure 3 in appendix.
Thus, this reimplementation behavessimilarly to the original implementation.
Discrepancies for the aggregated GLUE score:
By comparing results in table 2 and table 1 from Clark et al.
[2020],this study reports lower GLUE scores for each baseline and ELECTRA models.
For baseline models, I collectedthe different information from different sources, mainly from the original papers (see notes from table 2 for exactsources).
Then, I recomputed the GLUE score on dev set as per Wang et al.
[2018] from the individual task results.
Thediscrepancy appears to be related to a difference of reporting methodology, surely related to the treatment of the WNLItask.
Therefore, table 1 from Clark et al.
[2020] is not reporting the actual GLUE score on dev set but a meta score,from which I cannot fully reproduce.
While this discrepancy involves an overstatement for all models in Clark et al.[2020], it doesn’t change the main claim about the high performance of ELECTRA relative to other approaches.
Robustness: Despite the reported bug 3 from the original implementation regarding layer-wise learning rate decay; thespeciﬁc data augmentation procedure for STS and MRPC task 4 , which is not used in this reimplementation; and theminor differences in preprocessing steps, see section 3.2, the results are very close between this reimplementation and
[Table] [Table]
[Table]
83 .± 1 .80 .±
.83 .±
.74 .±
.84 .±
.57 .±
2 .74 .±
.72 .± 0 .
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
0 .59 .± 2 .75 .±
.73 .±
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
.86 .±
[Table] [Table]
.74 .± 0 .
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
.75 .±
[Table] [Table]
53 .± 1 .89 .
[Table]
[Table] [Table]
.85 .±
[Table]
0 .87 .±
.61 .± 1 .79 .±
.76 .± 0 .
[Table] [Table]
2 .88 .±
.87 .±
1 .85 .± 0 .86 .±
.80 .±
[Table] [Table]
Figures from Sanh et
[Table] [Table]
Figures
[Table]
Figures from the ofﬁcial ELECTRA’s Github repository.
The associated GLUE score cannot be computed as F scores and Person correlation are unavailable.
† Figures are the average of the 2 metrics (F1 and accuracy for MRPC and QQP; and Person and Spearman correlation for STS) and are not comparable.
GLUE scores are recomputed with the required metrics for each GLUE tasks.
For WNLI.
as the majority class predictor beats all models, the GLUE scores use 56.34% accuracy for this speciﬁc task.
the original one (Clark et al.
Thus, we can conclude the ELECTRA approach is robust and generalizable asresults are similar despite different implementations and different datasets (Wikibooks and OWT).
Efﬁciency: I also compared results from this reimplementation to the literature with the hardware requirements.
Findings are summarized in table 3.
By looking at the computation requirements, estimated by heuristics (see OpenAIblog on AI and Compute), relative to GLUE scores, ELECTRA (Clark et al.
[2020]) is outperforming all models.
Interestingly, this observation is also valid for models using knowledge distillation such as DistilBERT (Sanh et al.[2020]) or TinyBERT (Jiao et al.
[2020], even if we don’t take into account the initial cost for the teacher model.
Asimilar conclusion can be made by looking only at number of parameters, which is somehow an approximate proxy forinference time.
Therefore, ELECTRA is outperforming all compared approaches in term of efﬁciency.
[Table]
0 x) ≈ 0 .
x)
[Table] [Table]
Formula from
[Table]
blog on AI and Compute.
Assumptions: 8.9TFLOPS
GTX 1080; 12TFLOPS for P600; 16TFLOPS for V100; 45TFLOPS for TPU; 35TFLOPS for RTX 3090; 0.33 utilization rate.
See table 2.
4.2 Sensitivity to generator size
In this experiment, different models, with different generator capacity and with the same discriminator capacity, arepretrained like in section 4.1 with that the exception the training was stopped at 250k steps ( 25% ).
The results aresummarized in the table 4.
Training stability: Interestingly, in case of larger generator model than recommended in the original paper (Clark et al.[2020]), in this reimplementation, the discriminator network can collapse by not being able to distinguish which tokensare fake or not.
Therefore, even if ELECTRA is not using an adversarial setting (Goodfellow et al.
[2014]), the trainingprocedure may collapse if we allocate too much capacity to the generator compared to the discriminator.
Please refer toﬁgure 2 for a visual representation.
Discussion
5.1 What was easy
Ability to run on small GPU: One claim of ELECTRA (Clark et al.
[2020]) is the relative low computation requirements.
In this work, only runs with a single GPU have been executed.
Furthermore, the GPU memory requirement,roughly 15GB, can be easily decreased with the use of gradient accumulation.
As example, with 2 gradient accumulationsteps, the pretraining can be performed on a 11GB GPU.
[Table]
Clarity of the original paper: The original paper (Clark et al.
[2020]) contains a lot of detailed information, includingthe hyperparameters used for the pretraining and downstream tasks.
The proposed pretraining task is also well explainedand the replication was fairly easy as such.
The only exception was the unavailability of all GLUE metrics, such as F1score or Person correlation.
The section about negative results is also very beneﬁcial to better understand some earlyexplorations which has lead the Authors to propose the ELECTRA approach.
It would be beneﬁcial for all if suchopenness about negative results would be more systematically present in research papers, along the positive results.
Original source code and documentation: Authors of the original paper (Clark et al.
[2020]) have released theirsource code in their Github repository 5 .
While the source code was developed in Tensorﬂow instead of the targetframework, PyTorch, for this reimplementation, the original source code was relatively easy to understand.
Plenty ofdocumentation and answers were already present in this Github repository.
5.2 What was difﬁcult
GLUE benchmark: The GLUE benchmark is composed on 9 different tasks.
While each task is fairly simple toimplement, implementing all of them to use the same pipeline requires some software engineering effort, at least furtherthan expected.
More importantly for comparison, results are subject to high variance and therefore, most papers usesome statistics over several runs, usually mean or median.
The multitude of runs increase signiﬁcantly the computationbudget.
It would also be beneﬁcial to also report the variance of the results to have a better sense of this variance, and to ease comparisons between models.
Finally, papers use different way to report results, such as meta score like in
Devlin et al.
[2019], or different metrics for MRPC, STS or QQP, for example Clark et al.
[2020] and Sanh et al.
More practically, the validation of this ELECTRA reimplemention, compared to the original one, was harder due thesereporting differences.
Additional tricks for GLUE benchmark: Submissions for the GLUE benchmark have been using additional tricks toimprove performances, such as ensembling, further pretraining, multi-task training vs single task training, differenthyperparameters between tasks.
For example, while this has been mentioned in the original paper, see appendix from
Clark et al.
[2020], one speciﬁc task data augmentation procedure is also used for the results submissions for MRPCand STSB tasks.
All these tricks make harder to compare different models, as described in Aßenmacher and Heumann[2020].
Communication with original authors
Kevin Clark, one of the original authors (Clark et al.
[2020]), has been helpful by answering some questions.
Unfortunately, breakdown of GLUE score per tasks have not yet been provided to fully compare this implementation withthe original one.
Otherwise, most of questions that I had were already answered though the Github repository or byinspecting the source code.
Appendix
[Table] [Table]
[Figure]