Pre-trained language models evaluating themselves - A comparative study (2022)

Abstract: Evaluating generated text received new attention with the introduction of model-based metrics in recent years. These new metrics have a higher correlation with human judgments and seemingly overcome many issues of previous n-gram based metrics from the symbolic age. In this work, we examine the recently introduced metrics BERTScore, BLEURT, NUBIA, MoverScore, and Mark-Evaluate (Petersen). We investigate their sensitivity to different types of semantic deterioration (part of speech drop and negation), word order perturbations, word drop, and the common problem of repetition. No metric showed appropriate behaviour for negation, and further none of them was overall sensitive to the other issues mentioned above.

## introduction alongside with the current developments in natural language generation (nlg), evaluating the quality of artificially generated text is an equally important (and ever harder) task in the field. ngram based metrics, like bleu @xcite or rouge @xcite , come with severe drawbacks @xcite and given the increasing versatility of modern nlg systems, they are assumed to struggle even more @xcite . architectures based on the transformer @xcite , like bert @xcite or the complete gpt series @xcite @xcite , have increased the quality of artificially generated text to an extent that even humans tend to struggle distinguishing natural from artificial texts @xcite . based on these models, new metrics have been introduced, such as bertscore @xcite , bleurt @xcite , nubia @xcite , mover-score @xcite , or mark-evaluate (mordido and meinel, 2020), claiming to increase correlation with human judgment. we examine the latter introduced metrics using synthetic data. the examination will include several practical problems commonly observed in nlg systems. the code to reproduce our experiments is publicly available on github. foot_0 2 related work @xcite compared different metrics, including bertscore, regarding their sensitivity to specific impairments. their experiment (related, but not similar to ours) indicated that bertscore is more sensitive to the semantic integrity than ngram based metrics. another analysis by @xcite provides an evaluation of model-based metrics based on linguistic properties of their input. they showed that even model-based metrics tend to behave differently regarding specific modifications to their input. some metrics showed a higher sensitivity to semantics, while others showed higher sensitivity to syntactic issues. eventually, ensembling methods were proposed to combine the strengths of metrics. based on the checklist library @xcite introduced a library for assessing nlg metrics via different perturbations to the input data. multiple metrics, including model-based ones, were assessed, and neither of them did show a proper overall sensitivity to all modifications. the most severe issue was found in an overall insensitivity to negation. in contrast to @xcite , our work focuses on examining different degrees of perturbations and how metrics reflect these modifications towards maximal impairment. @xcite further underline the criticism of evaluating metrics according to their correlation with human judgments, which was already criticized in an in-depth analysis by mathur et al. (2020) about applying correlation as an evaluation measure. furthermore, our work does not focus on correlation but solely on the scores which the different metrics report when confronted with specific impairments to various degrees, how metrics behave in contrast to bleu when a particular part of speech is dropped, and how these metrics react to negated sentences. ## materials and methods the metrics examined in this work are bertscore @xcite , bleurt @xcite , nubia @xcite , mark-evaluate petersen (me-p) @xcite , and moverscore @xcite . as a baseline, the bleu score is always computed as well. the examined metrics can be subdivided into modelbased metrics and metrics as trained models. nu-bia and bleurt are trained models for evaluating generated text, while the other metrics are computed using specific formulas incorporating language models. detailed descriptions of the metrics are provided in appendix a. additionally to describing the respective metric, an exact specification of the setup and model-specific details are reported in appendix b. ## experiments for all our experiments we used the cnn/daily mail data set @xcite from huggingface.datasets as a reference corpus. since it represents a corpus of high-quality news articles, it is ideally suited to use the scores of its original sentences as an upper bound for the evaluated metrics. the data set is in english entirely, i.e. all our findings do not necessarily transfer to other languages. we randomly sampled 2000 texts from this corpus for all of the models, except for nubia and me-p. 2 resulting scores from artificial impairments of different degrees can subsequently be compared to this upper bound. the modifications foot_2 include the following different commonly observed flaws in nlg systems and the underlying language models: word swap random word pairs are chosen and swapped. the higher the intensity, the more random the sequence of tokens becomes, such that the original sequence should not be recognizable anymore. this approach was inspired by @xcite and @xcite . word drop a random drop of words mimics general quality deterioration. the larger the intensity, the larger the drop probability gets. at the highest level, only a few tokens are left. similar to word swap, this task was inspired by @xcite and @xcite . repetition as shown by @xcite , repetition remains a problem in text generated by nlg systems. a sequence at the end of the sentence is chosen and repeatedly added to the sentence to mimic this issue. with increasing intensity, the chosen sequence is repeated more often and the overall sentence becomes longer. at the maximum degree, the sequence is repeated as many times as there are tokens in the reference sentence. negation sentences were negated to change the semantics severely. a simple syntactic change of the sentence has the power to shift the semantics in an entirely different direction. the checklist library's (ribeiro et al., 2020) experimental foot_3 negation function was utilized to apply this change. specifically, the root of the dependency grammar tree is negated. this task was also used in the work of @xcite . pos-drop words with different part-of-speech (pos) tags were dropped to examine how the metrics behave when different kinds of words are removed. we assume for our experiment that some part-of-speech units like determiners have less influence over the semantic integrity than the removal of verbs, nouns, or adjectives. spacy @xcite and nltk @xcite were used to execute the different pos drops. the semanticinvariant and n-gram-based bleu score is computed for each impairment, which we then use for displaying the changes relative to modern metrics. (cf. fig. 2 ). ## results the loss of different words should be reasonably comparable to bleu. results for continuous impairments (word drop, word swap and repetition) are displayed in figure 1 , while negation and pos drop are shown in figure 2 . for each type of impairment, we will report the most striking observations. word swap while bleu exhibits, as expected, a steady drop to almost zero, some metrics tend to report higher values even when all words are swapped and the order is essentially random. nu-bia and bleurt both have minimum values above 0.4, while moverscore and bertscore yield values above 0.2 for the highest degree of impairment. in contrast to this behavior, me petersen is most sensitive to word order perturbation and shows a sharp decline. it already drops to 0.47 at the first level of word order perturbation and reports a score of 0.01 for the random permutation. word drop in this task, bleu, moverscore, bertscore, and me-p drop continuously until they eventually all (nearly) reach zero. me-p again drops the fastest, similar to the word swap but stops at 0.05. a different behavior, however, can be observed for bleurt and nubia, which again exhibit higher values compared to the rest. bleurt eventually drops to 0.14, and nubia even increases from its lowest value at the third level of impairment of 0.24 to 0.36 at the last level. repetition a less uniform behavior is observed for the repetition impairment, where the values strongly diverge at the highest level. both bertscore metrics monotonically decrease until they eventually reach zero, me-p also finally drops to a value near zero (0.06). however, it does not monotonically decrease, but drops sharply after the first level. bleu and moverscore both monotonically decrease strictly but end up way above zero at around 0.2. bleurt and nubia behave entirely different, such that bleurt seems to converge to 0.76 from the second level onward and does not show proper sensitivity to this issue, while nubia again increases after the third level from 0.5 to 0.52. ## discussion regarding word order perturbation, repetition, and word drop, it was expected to see a strict monotonous decline in the reported scores, which was not met by a single metric in every task (although me-p came close to meeting the expectations). however, at least one metric dropped to a value of zero or close to zero for every task. a crucial result is a metric-dependent sensitivity to word order perturbations and repetition. especially for nubia and bleurt, two trained metrics, the adj det verb noun 0.4 0.2 0.0 0.2 0.4 deviation from bleu pos-drop negated 0.0 0.2 0.4 0.6 0.8 1.0 value negation brtscr-idf brtscr moverscore me-p bleu nubia blrt-b-128 our pos-drop task showed that some tokens influence scores more than others. notably, the removal of determiners, which was expected not to influence the semantic integrity, did not lower the scores of most metrics compared to bleu. however, the syntactic integrity is affected, which must be considered when interpreting respective metrics. semantic-focused behavior like this was also shown in @xcite and was indicated by @xcite regarding bertscore. no uniform behavior in most metrics was seen for removing verbs, nouns, and adjectives. however, sensitivity to semantic integrity is bound by the underlying model's capabilities, as observed in our negation task. no metric reported a proper value for the severe semantic modification of negation, which aligns with @xcite . the work of @xcite and @xcite already examined bert regarding its understanding of negation, and they showed a general lack of understanding of the concept of negation. the most significant limitation of this work is the lack of expected ideal behavior when metrics are confronted with modified samples. it should be suspected that metrics show a higher drop in quality over more severe modifications, though it is unclear how humans would evaluate these specific cases. this issue is especially crucial in the task of negation since on the one hand side, it is not clear how severe the metrics are intended to reflect the impaired input, and on the other hand side it is also unclear how humans would rate negated sentences compared to the original sample. consequently, the lack of human evaluation has to be considered when interpreting the results of this work. the same issue must be stated for pos-drop tasks, in which human evaluation also becomes crucial. further, it has to be taken into consideration that we use a feature described as experimental by its creators foot_4 for negating the sentences. another arising issue, in this case, might be the rather long and detailed sentence structure of news article sentences, where the algorithm might be prone to negate only parts of the sentences. this issue might also arise for the pos-drop case, since some pos units might occur more often in this data set than in other text. ## conclusion & future work our results additionally underline that model-based metrics should be used with caution. the most severe drawback is the lack of sensitivity to negation, for which no metric reported a proper value. hence further research in natural language understanding is necessary to overcome this issue. furthermore, state-of-the-art metrics like bleurt and nubia lacked sensitivity to repetition, which is a severe issue in nlg. although many metrics deviated from the expected behavior, some others did not. thus, we endorse the proposal of @xcite to ensemble metrics, since some showed strengths where others showed weaknesses, and validate against the perturbation checklist package @xcite .