E lite PLM : An Empirical Study on General Language Ability Evaluation of Pretrained Language Models (2022)

Abstract: Nowadays, pretrained language models (PLMs) have dominated the majority of NLP tasks. While, little research has been conducted on systematically evaluating the language abilities of PLMs. In this paper, we present a large-scale empirical study on general language ability evaluation of PLMs (ElitePLM). In our study, we design four evaluation dimensions, memory, comprehension, reasoning, and composition, to measure ten widely-used PLMs within five categories. Our empirical results demonstrate that: (1) PLMs with varying training objectives and strategies are good at different ability tests; (2) fine-tuning PLMs in downstream tasks is usually sensitive to the data size and distribution; (3) PLMs have excellent transferability between similar tasks. Moreover, the prediction results of PLMs in our experiments are released as an open resource for more deep and detailed analysis on the language abilities of PLMs. This paper can guide the future work to select, apply, and design PLMs for specific tasks. We have made all the details of experiments publicly available at https://github.com/RUCAIBox/ElitePLM .

## introduction recent years have featured a trend towards transformer @xcite based pretrained language models (plms) in natural language processing (nlp) systems. by being pretrained on massive unlabeled text, plms can be directly fine-tuned on downstream tasks, entirely removing the need for task-specific architectures @xcite . this paradigm has led to significant progress on many challenging nlp tasks such as reading comprehension @xcite ) and text generation @xcite . with rising new state-of-the-art results that approach or surpass human performance on several tasks, it is a non-trivial research topic about how * corresponding author to systematically evaluate the language abilities of plms from a wide range of perspectives. given a wide range of publicly released plms, it is particularly useful to derive principles or guidelines for selecting suitable plms for specific downstream tasks. however, existing works either target some single ability @xcite , or consider a simple mixture of multiple (small-scale) tasks that lack a comprehensive design and test @xcite . there has been no detailed and systematic analysis of plm's abilities in large-scale nlp tasks. to fill the gap of plms evaluation, we introduce the general language ability evaluation (eliteplm) for empirically and systematically assessing the general language abilities of plms. the ideal goal behind plms is to create a humanlike machine learner where it can understand the language and then perform any specific task related to language. in cognitive science, wechsler adult intelligence scale (wais) @xcite is the most commonly used intelligence quotient (iq) test for measuring the intelligence and cognitive ability of humans. this test would assess the level of individuals on verbal comprehension, perceptual reasoning, working memory, and processing speed. thus, by imitating the intelligence test on humans, we design four evaluation dimensions in eliteplm for measuring the abilities of plms, including memory, comprehension, reasoning, and composition. following previous works @xcite , for each ability in eliteplm, we elaborate and select multiple representative tasks (e.g., question answering for the comprehension ability) and commonly-used benchmarks (e.g., glue and squad) to quantitatively evaluate the performance of plms. these results can serve as numerical explanations of plms at a specific ability. in human intelligence tests, the background of participants (e.g., gender, race, and occupation) should be as much as diverse. thus, in eliteplm, we select a diversity of plms to conduct generalized and meaningful comparisons. according to training objectives, plms can be divided into three types: bidirectional lms (e.g., bert @xcite ) for natural language understanding (nlu), unidirectional lms (e.g., gpt @xcite ) for natural language generation (nlg), and hybrid lms (e.g., unilm @xcite ) for combining these two paradigms. furthermore, knowledge-enhanced lms (e.g., ernie @xcite ) and text-to-text lms (e.g., t5 @xcite ) also emerge as important branches of plms. considering the variety, we finally select ten widely-used plms within the above five categories and evaluate their abilities on four dimensions. we show the comparisons of these plms in from the ability test results, we have three salient findings. first, plms with varying pretraining objectives and strategies are good at different kinds of downstream tasks. specifically, we observe that bidirectional lms like bert and pretraining strategies like larger training batches in roberta are helpful for memorizing pretraining corpora; permutation language modeling in xlnet is beneficial for modeling the bidirectional context in language comprehension; inter-sentence coherence objective in albert is suitable for sentence-level reasoning tasks; text-to-text lms using denoising objective like bart perform better in short text generation. second, when fine-tuning plms in downstream tasks, their performance is typically sensitive to the data distribution in fine-tuning stage, which can be addressed by incorporating intermediate datasets or tasks to alleviate such a discrepancy. third, plms have excellent transferability between similar tasks, especially reasoning tasks. this finding will inspire future researchers to leverage data-rich tasks for improving data-scarce tasks. for more clarity, we illustrate the impact level of each factor for plms' abilities in besides eliteplm being an evaluation benchmark of plms' language ability, more importantly, the predicted results of eliteplm can be used as an open resource for more depth and granularity in analyzing plms performance on each ability. for example, we further analyze the comprehension test results of plms across answer types in qa tasks. the analysis shows that plms are good at simple single-token answers such as dates but more challenged on intricate phrase answers. moreover, by analyzing human test and turing test results on composition, we observe that summaries with high accuracy are more likely to pass the turing test while rich information is more important for story generation. overall, eliteplm can act as an analysis tool to gain more insight into plms. we show the details of our used datasets and predicted outputs of plms in appendix b. this paper is intended to help establish sound principles for choosing, applying, interpreting and designing plms for nlp tasks in practical settings. we have released the code and predicted results of each ability experiment, providing the research and industry community with off-the-shelf tools to evaluate and analyze their plms. ## eliteplm in this section, we will detail these four kinds of language abilities, i.e., memory, comprehension, reasoning, and composition, in eliteplm. ## experiments in this section, we first set up baselines, and then report the results and analysis on four ability tests. ## discussion based on the above four ability tests, we intend to provide a guideline for helping researchers choose, apply, interpret and design plms for nlp tasks. in section 3.3, we observe that the improvement in memory ability is likely to be helpful for the performance of comprehension ability. hence, designing plms with special objectives like bidirectional language modeling in bert and strategies like larger training batches in roberta for larger memory capacity will further benefit plms in the downstream comprehension tasks. besides, when applying plms to downstream tasks, the similarity of data distribution between pretraining and finetuning has a great impact on plms performance. possible solutions such as introducing intermediate tasks or datasets can alleviate such a discrepancy. moreover, we further find some limitations in plms' comprehension ability, where plms are good at simple single-token answer types in qa such as dates but perform worse in complex phrase answers. compared to comprehension, reasoning in section 3.4 is much more intricate and usually involves inferring the semantic relationships among multiple sentences. therefore, plms such as al-bert trained with sentence-level objectives can be more suitable for conducting reasoning tasks. intuitively, incorporating sentence-level objectives during pretraining will help plms learn the correlation among different sentences. note that plms have better reasoning transferability between similar tasks, thus data-scarce reasoning tasks can be improved by first training on data-rich tasks. for composition ability, plms with denoising training objectives perform much better on short text composition, while plms with left-to-right objectives or larger model size are more suitable for long text composition. this might be because plms with different training objectives can finally capture different ranges of semantic dependency between tokens and phrases. moreover, to obtain a higher pass rate of turing test, different text generation tasks will be concerned with varying factors, such as informativeness is much more critical for story generation. language model evaluation. many efforts have studied the evaluation of language model performance. @xcite evaluate bert @xcite , gpt @xcite , and elmo @xcite on a variety of linguistics tasks. their findings indicate that the features generated by plms are sufficient for good performance on a board set of tasks but fall short on tasks requiring fine-grained linguistic knowledge. @xcite evaluate similar models on a range of sub-sentence linguistic analysis tasks, showing that plms encode both syntax and semantics into parameters. @xcite also report that plms can learn rich knowledge but focus on evaluating the commonsense. however, these studies only look at one dimension of plms ability evaluation. other work such glue @xcite and clue (liang xu, 2020) just consider a simple mixture of multiple tasks lacking comprehensive evaluation. to the best of our knowledge, this is the first work to systematically evaluate plms by defining various kinds of language abilities and performing extensive comparison. ## conclusion this paper investigates the general language ability evaluation of pretrained language models. we design four kinds of language abilities of plms, including memory, comprehension, reasoning, and composition, and measure ten widely-used plms within five categories. for each language ability, we select multiple representative tasks to quantitatively evaluate the performance of plms. our experimental results demonstrate that plms with varying objectives and strategies are good at different ability tests. note that our final predicted outputs of plms can also be reused as an open resource for more depth and granularity in analyzing plms' language abilities. as a result, it is believed that this study will benefit future work about choosing or designing suitable plms for the target nlp tasks based on their properties.