A Fine-Grained Analysis of BERTS core (2021)

Abstract: BERTScore, a recently proposed automatic metric for machine translation quality, uses BERT, a large pre-trained language model to evaluate candidate translations with respect to a gold translation. Taking advantage of BERT’s semantic and syntactic abilities, BERTScore seeks to avoid the flaws of earlier approaches like BLEU, instead scoring candidate translations based on their semantic similarity to the gold sentence. However, BERT is not infallible; while its performance on NLP tasks set a new state of the art in general, studies of specific syntactic and semantic phenomena have shown where BERT’s performance deviates from that of humans more generally. This naturally raises the questions we address in this paper: what are the strengths and weaknesses of BERTScore? Do they relate to known weaknesses on the part of BERT? We find that while BERTScore can detect when a candidate differs from a reference in important content words, it is less sensitive to smaller errors, especially if the candidate is lexically or stylistically similar to the reference.

## introduction while manual, human evaluation of machine translation (mt) systems is still the gold standard, automatic evaluation metrics have long been used for their relative speed and inexpensiveness. early automatic metrics were easy to implement and somewhat correlated with human judgements, but have clear limitations: bleu @xcite relies on n-gram overlap, and is thus not robust to differing word order or choice. in contrast, me-teor @xcite requires training, but depends on token alignment, which is also a fraught task. with the advent of deep learning, new automatic metrics have arisen, both in response to and making use of the technical advances brought by deep learning. in particular, metrics like comet @xcite and bertscore use large pre-trained language models (llms) to generate scores for candidate sentences. the use of these llms allows for metrics that take advantage of the linguistic capabilities of these llms, and no longer rely solely on surface-level features such as n-grams. the expressiveness of these models is both a boon and a danger. while they can (and do, based on correlation with human judgments) generate more useful scores for translations, how they arrive at the score, and which types of sentences they will score accurately is not immediately obvious. moreover, these llms are known to have flaws. bert in particular has been shown to be, in certain scenarios, insensitive to negation @xcite and word order @xcite . bert also has inexact representations of numbers @xcite and fails to be robust to named entities @xcite . all of these phenomena could result in poor-quality scores from bertscore. however, it is difficult to say for certain how these issues might manifest in bertscore, as it employs bert in an unsupervised scenario distinct from that of these analyses. thus, in this paper, we analyze bertscore. we first formally define desiderata for a mt metric. then, we consider how bertscore fulfills these requirements under conditions of interest. we find that bertscore violates some of these requirements, specifically the requirement that incorrect translations be rated below correct ones; this occurs most often when the incorrect translation is lexically similar to the reference, or especially if the difference is only in function (not content) words. ## desiderata for mt metric quality the most common method of measuring the quality of a mt metric is correlation with human judgments @xcite ; however, these correlations provide little information regard-ing when and why an mt metric differs from human judgment. in this paper, we consider three ways of examining mt metric quality, with the aim of determining the failure cases of mt metrics. in all of our experiments, we assume the following setup. we have a mt metric, which we take to be a function m that takes as input a reference and candidate translation, and outputs a score in [0, 1]. we also have a dataset d, consisting of triples (x, y, b) where x is a source sentence, y is a list of at least two reference translations, and b is a list of at least one "bad" translation, which contains errors. then we state that a good mt metric foot_0 m fulfills, for any triple (x, y, b) ∈ d, where @xmath0, y 2 , . . . , y n }, and @xmath1, b 2 , . . . , b m }, the following conditions: put simply, (i) reference translations should be scored near 1 when compared to each other, (ii) bad candidate translations should be scored worse than reference translations, and (iii) the scoring of bad candidate translations should reflect their relative quality. to use this framework to investigate the failure points of mt metrics, we simply need a dataset that contains phenomena of interest; for example, we might be interested in knowing if a mt metric is able to distinguish between translations that do and do not correctly render negation. then, we simply compute the quantities discussed in conditions (i) through (iii) for each example, and see which, if any, conditions are violated. if, for example, condition (i) is violated when two equivalent references employ different types of negation, this might imply that our metric is not robust to this sort of negation phenomenon. note that these desiderata only concern the scores given to the reference and candidate sentences; the source-language sentence is ignored. this is because we define these desiderata keeping in mind that many mt metrics (including bertscore) operate only in the target language foot_1 . we can thus avoid assuming the existence of a source sentence whatsoever, allowing the construction of datasets consisting only of reference and bad candidate translations that exhibit phenomena of interest. however, it may be desirable to use real translations, so that the dataset reflects the distribution of real-world translation errors. ## experiments for our experiments, we utilize the framework described in section 2 to investigate the questions about bert described in section 3. as the number of datasets that fit our framework is few, we limit ourselves to three primary experiments. ## results in the following section, we detail the experiments performed and their results. in all experiments, the original authors' implementation of bertscore foot_2 is used, with default baseline rescaling. ## discussion from our experiments on three datasets, we draw three main findings. first, we find that the performance of bertscore with respect to conditions (i) and (ii) can vary based on linguistic phenomena. second, while bertscore is generally capable, it has difficulties on a challenge dataset that tasks it with penalizing lexically similar incorrect translations and preferring translations that are lexically different but more correct. third, bertscore does, in certain circumstances, respect our third condition-it ranks worse bad translations below better bad translations. with respect to the first finding, penalizing translations that incorrectly render function words seems to be the most difficult for bertscore. in tq-autotest, this includes sentences with tag questions; in one example, the reference is "you're crazy, aren't you?", and secondary good translation is "you're crazy, right?", while incorrect sentences are "you're crazy, or?" and "you're crazy, are not you?". that is, the differences do not affect the main content of the sentence. in contrast, sentences with incorrectly resolved word ambiguity or larger, multi-word errors were easily penalized. the second finding also confirms that bertscore can more easily detect bad translations when there is less lexical overlap. in the easy problem setting, where both the good and bad candidate translations were lexically distinct from the reference, bertscore easily distinguished the good translation from the bad. but, in the hard problem setting, where the bad translation has high lexical overlap with and was stylistically similar to the reference, bertscore struggled. due to this style sensitivity, bertscore may be better-suited to scoring candidates from widely-differing systems, as opposed to closely-related systems, or multiple candidates from one system. our third finding, provides more positive results. in the gec scenario, bertscore was able to fulfill all three of the conditions we defined. it not only gave high scores to similar sentences and worse scores to sentences with errors; it also gave better scores to more grammatically correct sentences, even when they were not perfectly correct. unfortunately, the gec dataset does not necessarily reflect the kinds of errors that occur in machine translation. it focuses primarily on grammatical errors, and thus contains fewer semantic errors. moreover, since the gec annotators had only the incorrect text, and no source text to work with, their annotations are occasionally in disagreement, as they each independently inferred the intended meaning of the incorrect text. ## related work the original paper introducing bertscore @xcite naturally compared bertscore's correlations with human judgments to that of other metrics. however, various other surveys of mt metrics, as well as datasets and methodologies have been conducted, offering insights into how mt system and metric performance should be measured. naturally, the wmt metrics task, most recently run in 2020 @xcite ) is one such forum for the evaluation of metrics. in this last iteration, metrics were evaluated based on their correlation with human judgment scores on the sentence, paragraph, and document level. bertscore was not included even in the most recent iteration of the metrics task. more recently, @xcite run a large-scale comparison of mt metrics, including bertscore using a large dataset of translations with human judgments; they find that bertscore's performance is middle-of-the-road, though better than bleu, and recommend comet @xcite for general use. unfortunately, while these studies evaluate mt metrics, using human judgments alone cannot tell us when or why they may succeed or fail. in response to the results of 2019 wmt metrics task @xcite , @xcite note that correlations of metrics with human judgment can be highly sensitive to the number of systems in question, as well as outliers. @xcite propose moving beyond correlation with human judgment alone as a standard for mt metric evaluation. to this end, they conduct a comprehensive study of mt metrics, and review datasets that have more finegrained error and quality annotations. despite this, datasets for mt metric evaluation with linguistic annotations or other annotations regarding sentence content are few. ## conclusion bertscore is a new metric for mt evaluation that uses bert, and is as a result difficult to interpret. we define desiderata for bertscore and other such metrics' performance, and use targeted datasets to find when bertscore fails. we find that bertscore fails to assign low scores when a bad candidate sentence has high lexical overlap with the reference in terms of content words. despite this, in less challenging scenarios, bertscore does well, and is able to rank sentences in order of their quality. moreover, bertscore outperforms bleu score across the datasets and conditions we tested. however, these experiments are limited in scope, due to limited available data with appropriate annotations. development of datasets for mt metric evaluation with linguistic annotation would aid in further work on this topic.