[RE] ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility
Summary
In this report, in-depth comparisons between BERT and ALBERT are made using various open-source interpretabilityand testing tools to verify the claim that ALBERT achieves better performance and faster inference compared to BERT.
ALBERT achieves higher benchmarks across many different downstream tasks and demonstrates appropriate sentence embeddings visualization.
However, experiment results from behavioral testings and adversarial attacks suggest that
ALBERT has relatively worse capabilities, particularly in terms of Robustness and Fairness.
Scope of Reproducibility
Lan et al.
[2] attributes improved parameter efﬁciency as the most important advantage of ALBERTs design choicesand claims substantial improvements for several GLUE and other downstream evaluation tasks.
Thanks to the prevalence of established full re-implementations and standardized models, model interpretability testing tools were usedto perform in-depth comparisons beyond traditional benchmarks.
Methodology
Fine-tuned models were obtained from HuggingFace [9] and those released by Morris et al.
[4] were used to ensureconsistency across models and experiments.
Most notably, SentenceTransformers [5], UMAP [3], CheckList [6],
TextAttack [4], and LIT (Language Interpretability Tool) [8], were used for each experiments.
Results
Based on several experiments in this report, ALBERT is qualitatively worse than BERT and has consistently slowerinference speed.
Therefore, this work does not support the broad conclusion that ALBERT outperforms BERTin terms of performance and inference speed in most cases.
What was easy
With the standardized models from the HuggingFace’s
Transformers [9], the use of plug-and-play models in variousexperiments was trivial.
Because all of the tools used in the experiments were open-source, it was fairly easy to debugerrors and modify the source code.
What was difﬁcult
Reviewing the literature and writing this reproducibility report alone took a signiﬁcant amount of time, notwithstandingthe absence of re-implementation or training. Communication with original authors Initially,
Introduction
Lan et al.
[2] identiﬁed four major ways to "lighten" Devlin et al.
[1]’s original BERT architecture: cross-layer parameter sharing, sentence-order-prediction auxiliary loss, factorized embedding parameterization, and dropout removal.
Typically, a replication study reproduces one or more of these claims by means of a full re-implementation.
However,it did not seem reasonable to pursue the same undertaking when many re-implementations had already been carriedout.
In addition, full re-implementation is a time-consuming and error-prone procedure that ultimately results in the veriﬁcation of the same set of metrics that prompted the need for reproducibility.
With the recent developments in easy-to-use interpretability and testing tools for NLP models, state-of-the-art modelscan be evaluated in terms of qualitative performance in addition to existing benchmarks.
Scope of reproducibility
Lan et al.
[2] claimed signiﬁcant improvements over BERT for several GLUE and other downstream evaluation tasksbased on the improvement of ALBERT in parameter efﬁciency.
Simply put, Lan et al.
[2] claimed that ALBERT isbetter than BERT based on benchmarks.
However, comparing the performance of models based on a single aggregated statistic is problematic because it isdifﬁcult to ﬁgure out why and where the models are fail (Wu et al.
By considering each model as a black-box, aqualitative comparison of model capabilities for various models can be made, even though they have been trained ondifferent datasets (Ribeiro et al. During our preliminary review, we found seven pre-existing full re-implementations of ALBERT on GitHub alone,as well as over hundreds of pre-trained and ﬁne-tuned models via Huggingfaces’s transformers (Wolf et al. Themajority of these models indicated that the original paper’s results were reproducible. However, we found no resultsbeyond the paper.
Rather than replicating ALBERT from the scratch, this report uses some of the 82 models provided by Morris et al.
[4] to determine if ALBERT is actually better BERT in terms of both benchmarks and capabilities. 2.1 Addressed claims
from the original paper • Fine-tuned ALBERT on STS-B provides similarly or or more meaningful representation of sentence embeddings compared to BERT.
• Fine-tuned ALBERT on sentiment analysis and paraphrase identiﬁcation tasks have similar or better behav-
ioral testing results compared to BERT.
• Fine-tuned ALBERT on RTE is equivalent to or more robust against adversarial attacks than BERT.
Methodology
Due to the prevalence of existing re-implementations, the available ﬁne-tuned models have been used for an in-depthcomparison of BERT and ALBERT models.
Beyond traditional benchmarks, visualization of sentence embeddings,behavioral testing, adversarial attacks, and counterfactual explanation experiments were carried out to verify if claimsby Lan et al.
[2] still apply under different circumstances.
3.1 Model descriptions
All models were obtained from TextAttack Model Zoo via the HuggingFace website ( https://huggingface.co/textattack ), re-implemented and ﬁne-tuned by Morris et al.
3.2 Datasets
All datasets were obtained from the HuggingFace’s Datasets library ( https://github.com/huggingface/datasets ).
3.3 Hyperparameters
Hyperparamters for the ﬁne-tuning varies from model to model.
According to Morris et al.
[4], the best out of a gridsearch over a bunch of possible hyperparameters was selected for each of the models.
All experiments used the following additional parameters, unless otherwise speciﬁed.
• SentenceTransformers: models.
Transformers(max_seq_length=128)
• UMAP: umap.
UMAP(random_state=0, transform_seed=0, metric=’cosine’)
• CheckList: TestSuite.run(seed=1)
TextAttack: textattack attack –num-examples
3.4 Experimental setup
All experiments were performed on Google Colab with a single Tesla T4 GPU (NVIDIA-SMI 450.32.03 Driver Version: 418.67 CUDA Version: 10.1).
Notebooks and other artifacts are available on GitHub ( https://github.com/ ).
3.5 Computational requirements
The computational requirements for each experiment ranged from a few minutes to an hour on a single GPU due tothe different heuristics used for each experiment.
Results
ALBERT achieves higher benchmarks across many different downstream tasks and demonstrates appropriate sentenceembeddings visualization.
However, experiment results from behavioral testings and adversarial attacks suggest that
ALBERT has relatively worse capabilities, particularly in terms of Robustness and Fairness.
4.1 Benchmarks
[Table]
Morris et al.
[4] provides ﬁne-tuned textattack/bert-base-uncased and textattack/albert-base-v2 models via HuggingFace [9].
Based on Table 4.1, the ALBERT models clearly outperforms the BERT models across many differentdownstream tasks, as Lan et al.
[2] claimed.
4.2 Embeddings visualization
bert-base-uncased-STS-B and albert-base-v2-STS-B models were chosen for this experiment because sentence embeddings of ﬁne-tuned models have semantically more meaningful representation compared to the original pre-trainedmodels [1] [5] [7].
The embeddings were generated using the AG News training dataset without further ﬁne-tuning.
Tvisualizations in Figure 1 were then created using UMAP [3].
Given that UMAP is agnostic to rotation or reﬂection of the ﬁnal layout [3], the results are essentially the same as thereﬂection in the x-axis and the 90-degree counter-clockwise rotation for the BERT visualization results in almost thesame layout as the ALBERT.
However, ALBERT (10min 30s) took noticeably longer to produce the embeddings compared to BERT (8min 49s). Itshould be noted that the UMAP, which took 3min 52s and 4min 4s respectively, was not included. [Table]
4.3 Behavioral testing
In Table 4.3, ﬁne-tuned models on several sentiment analysis datasets were tested with CheckList [6].
The sentimentanalysis test suite (curated by Ribeiro et al.
[6]) consists of Minimum Functionality test (MFT), INVariance, and
DIRectional capability tests.
Apart from six Negation test cases, Table 4.3 includes all test cases for Vocabulary ,
Robustness , NER , Fairness , Temporal , Negation , and SRL capability tests.
For Vocabulary , Negation , and SRL , it’s difﬁcult to make a reasonable conclusion due to the variance across differentdatasets.
In addition, certain test cases ( Single negative words and Reducers ) have extreme outliers, which rendersthese tests unreliable.
For Robustness , NER , and Fairness , ALBERT consistently result in higher fail rates than BERT.
More importantly,
ALBERT have signiﬁcantly worse fail rates for Fairness in all cases except Race on SST-2.
In Table 4.3, ﬁne-tuned models on two paraphrase identiﬁcation datasets (QQP and MRPC) were tested with CheckList[6].
Apart from Robustness , only the MFT test cases from the QQP test suite (curated by Ribeiro et al.
[6]) are includedin Table 4.3.
For QQP, similar behavior is observed for Robustness , but not for NER .
For MRPC, ALBERT achieves lower fail ratesthan BERT across all test cases for Robustness .
However, it seems inappropriate to use the QQP test suite for MRPCsince both BERT and ALBERT have either 0.0% or 100.0% for the most of the capability tests in MRPC.
On average, ALBERT (4m 48s and 10m 47s) took consistently longer than BERT (4m 14s and 9m 30s) to completethe test suites (sentiment analysis and QQP).
[Table]
Change numbers 3.
Fairness
Race 31.
Sexual 69.
Religion 32.
Nationality 8.
Temporal
Used to, but now 51.
"Used to" should reduce 56.
Negation
Negative 1.
Not negative 93.
Not neutral is still neutral 98.
SRL
My opinion is what matters 53.
Q & A: yes 47.
Q & A: yes
[Table]
neutral) 99.
Q & A: no 53.
Q & A: no
[Table]
neutral) 100.
Notes: CheckList sentiment analysis test suite results for bert-base-uncased and albert-base-v2 models.
The reported numbers are fail rates in % (lower the better).
For each dataset, lower fail rates
[Table]
What was person’s life before becoming X !
=
What was person’s life after becoming X 100.
Do you have to X your dog before Y it !
=
Do you have to X your dog after Y it.
Is it {ok, dangerous, ...} to {smoke, rest, ...} after !
= before 99.
[Table]
can
[Table]
become
[Table]
person !
=
How can I become a person who is not X 6.
Is it {ok, dangerous, ...} to {smoke, rest, ...} in country !
=
Is it {ok, dangerous, ...} not to {smoke, rest, ...} in country 17.
What are things a {noun} should worry about !
= should not worry about.
How can I become a X person
[Table]
How can I become a person who is not antonym(X) 77.
[Table]
she 96.
[Table]
coref: his and her 99.
[Table]
[Table] [Table]
[Table] [Table]
Traditional SRL: wrong active / passive
[Table] [Table] [Table]
Logic
[Table]
or B is not
[Table] same as C and D
A or B is not the same as A and B
A and / or B is the same as B and / or A
a {nationality} {profession} = a {profession} and {nationality}
Reﬂexivity: (q
q)
should be duplicate 0.
[Table]
4.4 Adversarial attack
bert-base-uncased-RTE and albert-base-v2-RTE models were chosen for this experiment to perform adversarial attacksusing TextAttack [4].
Out of fast-alzantot , iga , and textfooler attack recipes, iga was the most effective in executingvalid attacks (3 out 10 examples).
Due to the limited number of adversarial examples and lack of explanation, it cannotbe argued that one is more or less robust against adversarial attacks compare to another.
Example Asentence1:
Mrs. Bush’s approval ratings have remained very high, above 80%, even as her husband’shave recently dropped below 50%.sentence2: 80% approve of Mr. Bush.
Not entailment (98%) –> Entailment (52%)
[BERT]sentence1: Mrs. Bush’s endorsement ratings have persevere very haut, above 80%, even as herhusband’s have recently plummeted below 50%.sentence2: 80% endorsement of Mr. Bush.
Not entailment (95%) –> Entailment (57%)
[ALBERT]sentence1: Mrs. Bush’s approval punctuation have remains very superior, above 80%, even as herhusband’s have recently dropped below 50%.sentence2: 80% approval of Monsieur.
Bush.
Example A demonstrates that both BERT and ALBERT are vulnerable to this adversarial attack.
While haut is frenchand superior has a different meaning in this context, it was enough to make both models to classify the example as
Entailment when it is clearly not. Furthermore, this particular example was successful in all above mentioned recipes. Example B sentence1:
Two British soldiers have been arrested in the southern Iraq city of Basra, sparkingclashes outside a police station where they are being held.sentence2: Two British tanks, sent to the police station where the soldiers are being held, were setalight in clashes.
Not entailment (97%) –> Entailment (71%)
[BERT]sentence1: Two British solider have been captured in the southern Iraq
city of Basra, sparking
clashes outboard a police station where they are being held.
sentence2: Two British tanks, sent to the policing station where the soldiers are being held, were set
alight in clashes.
Not
entailment (
> [FAILED] [ALBERT] In Example B, iga recipe fails to produce a valid adversarial example for ALBERT. However, the lack of an successfulattack does not mean that ALBERT is more robust against than BERT in this particular example since the textfoolerrecipe was capable of generating a valid adversarial attack.
C sentence1:
U.S. forces have been engaged in intense ﬁghting after insurgents launched simultaneous attacks in several Iraqi cities, including Fallujah and Baqubah.
sentence2: Fallujah and Baqubah are Iraqi cities.
Entailment (90%) –> Not entailment (98%) [BERT]
Sentence 1: U.S. forces have been engaged in intense ﬁghting after insurgents launched simultaneous attacks in several Iraqi cities, including Fallujah and Baqubah.
sentence2: Fallujah and Baqubah are Iraqi townships.
Entailment (96%) –> Not entailment (96%)
[ALBERT] sentence1: U S. forces have been engaged in intense ﬁghting after insurgents launched simultaneous attacks in several Iraqi townships, including Fallujah and Baqubah.
sentence2: Fallujah and Baqubah are Iraqi cities.
Example C shows that both BERT and ALBERT are vulnerable to this very simple adversarial attack (cities <–>townships).
Hwoever, this particular example was only successful in iga recipe.
On average, ALBERT (20m 11s, 4m 27s, and 15s) took longer time than BERT (15m 46s, 6m 12s, and 17s) tocomplete each attack ( fast-alzantot , iga , and textfooler ).
However, attack time does not reﬂect inference speed as itdepends more on the attack recipes than the models.
4.5 Counterfactual explanation
[Figure]
[Table]
[Table] [Table] [Table] Similar to section 4.4, the same bert-base-uncased-RTE and albert-base-v2-RTE models were used in this experimentto address the issues found in section 4.4.
With LIT (Language Interpretability Tool) by Tenney et al.
[8], Example Cfrom section 4.4 was used to generate counterfactual examples by swapping cities and townships in either sentence1or sentence2.
Based on Figure 3, BERT could be considered more robust than ALBERT for this particular example.
DiscussionEmbeddings visualization
The Language Interpretability Tool (LIT) comes with the Embedding Projector that aims to help machine learningdevelopers and researchers to investigate semantically meaningful vectors in embedding space (Tenney et al.
Itprovides UMAP, t-SNE, and PCA projections to visualize layer-wise embeddings of any machine learning models.
Wechose UMAP because it has demonstrably better run time performance than t-SNE and preserves more of the globalstructure of larger datasets (McInnes and Healy [3]).
Although qualitative analysis of visualization should not be interpreted as a deﬁnitive evidence (Rogers et al. UMAP visualization of embeddings can be quite useful for exploring and comparing clusters and global structures of NLP models (Tenney et al.
Furthermore, Sentence-BERT overcomes the short comings of layer-wise embeddingsby using siamese / triplet network architecture to derive semantically meaningful sentence embeddings.
Nonetheless,visualizations in Figure 1 are the exceptions rather than the rules when it comes to embeddings visualization, sincemost of the ﬁne-tuned models result in widely different embeddings that have far less meaningful representations(Reimers and Gurevych [5]).
Behavioral testing and adversarial attack
One of the primary goals of training NLP models is generalization (Morris et al. and these models are extremelygood at ﬁnding correlations and patterns that are consistent across their training dataset. However, many of thesescorrelations and patterns are actually spurious and do not hold for other distributions. While performance on indistribution is a useful indicator, in-distribution performance is often not comprehensive, and contain the same biasesas the training data (Ribeiro et al.
Out of existing behavorial and adversarial tools, we chose Checklist and TextAttack because both tools can be directlyintegrated with HuggingFaces transformers.
CheckList is a comprehensive behavioral testing of NLP models thatguides users in what to test, by providing a list of linguistic capabilities (Ribeiro et al.
TextAttack aims toimplement adversarial attacks that, given an NLP model, ﬁnd a perturbation of an input sequence that satisﬁes theattack’s goal while adhering to certain linguistic constraints (Morris et al.
These tools make it easier to reasonabout the behavior of NLP models under distribution shift and adversarial settings, as well as their tendencies to behavebased on social biases or shallow heuristics.
According to Ribeiro et al.
[6], CheckList should not be treated as yet another set of challenge or benchmark datasets;instead, it should complement benchmarks to systematically evaluate the precise capabilities of a model that are notcaptured in benchmarks.
Based on Table 4.1, it can be somewhat vague and possibly misleading how BERT and
ALBERT models fare in respect to Robustness and Fairness .
With behavioral testing approach, model capability canbe evaluated more precisely across different models and datasets (Table 4.3 and Table 4.3).
Much of the adversarial attacks reported successful by TextAttack [4] were often invalid because the search constraintswere not properly optimized.
Regardless, TextAttack [4] is highly effective for identifying weakness of each model.
More precisely, it is useful to identify which token to attack, but often fails to generate valid tokens to substitute it.
Counterfactual explanation
While CheckList [6] and TextAttack [4] provide an easy way of evaluating NLP models beyond the typical benchmarks,the Language Interpretability Tool (LIT) provides a more comprehensive set of tools for developers and researchersto debug performance of NLP models (Tenney et al.
With LIT, users can easily hop between visualizationsto test local hypotheses and validate them over a dataset, add new data points on the ﬂy, and compare two modelsside-by-side.
LIT [8] consists of a wide variety of interpretability and explainability components.
One of which is the Salience
Map .
This one is particularly interesting because there seems to be a correlation between the attack targets chosen by
TextAttack [4] recipes and the token gradient norm weights in the Salience Map .
However, tokens with the highestweights are not always the best target for generating insightful counterfactual examples.