MENLI : Robust Evaluation Metrics from Natural Language Inference (2023)

Abstract: Recently proposed BERT-based evaluation metrics for text generation perform well on standard benchmarks but are vulnerable to adversarial attacks, e.g., relating to information correctness. We argue that this stems (in part) from the fact that they are models of semantic similarity. In contrast, we develop evaluation metrics based on Natural Language Inference (NLI), which we deem a more appropriate modeling. We design a preference-based adversarial attack framework and show that our NLI based metrics are much more robust to the attacks than the recent BERT-based metrics. On standard benchmarks, our NLI based metrics outperform existing summarization metrics, but perform below SOTA MT metrics. However, when combining existing metrics with our NLI metrics, we obtain both higher adversarial robustness (15%–30%) and higher quality metrics as measured on standard benchmarks (+5% to 30%).

## introduction proper evaluation is key to fields such as machine learning and natural language processing (nlp). evaluation is particularly challenging for natural language generation (nlg) tasks, as there many be an infinitude of correct solutions (e.g., translations or summaries) for a given source text. while human evaluation is often considered the gold standard, it is slow and costly, thus researchers resort to automatic evaluation. previously, this was done using simple lexical overlap metrics such as bleu and rouge, but these exhibit low correlations with human judgments, particularly for state-of-the-art nlg systems @xcite . thus, a popular recent trend is to design automatic evaluation metrics based on large language models such as bert and its many extensions @xcite @xcite . nonetheless, these novel metrics also have key limitations. for example, @xcite and @xcite show that they are not robust to various adversarial attacks including lexical overlap and factuality errors. taking the currently most popular metric-bertscore foot_0 -as an example, this adversarial vulnerability is unsurprising. bertscore computes the semantic similarity between a reference and a system output (the candidate), using a simplified token matching procedure. however, a good candidate is typically not appropriately identified by semantic similarity. for example, a candidate ''5 ukrainian soldiers wounded in russia'' is not an adequate translation of a source corresponding to the reference ''50000 russian soldiers killed in ukraine'', although the two texts are of course semantically very similar. 2 while there have been many attempts to improve bertscore using better token matching, e.g., using word mover distance @xcite @xcite , we argue that this line of research is a dead-end, as the underlying model of semantic similarity, originally proposed to address issues of lexical variation in bleu/rouge, is simply not (fully) appropriate. an intuitively more suitable idea to model evaluation metrics is via natural language inference (nli) @xcite . for example, in reference-based settings, in which candidates are compared to human references, a candidate is intuitively good if it is equivalent to a human reference via the concept of bi-implication. nli systems are also promising alternatives because nli is one of the most researched upstream tasks in nlp, where a lot of emphasis has been placed on concepts such as biases, generalization and adversarial conditions @xcite . in this paper, we ask whether we can directly use pre-trained nli models as evaluation metrics, thereby establishing a new paradigm (but with predecessors, as indicated in §2). our contributions: we point out that some current metrics already leverage nli systems-thus, we do not include new information with respect to them-but indirectly and thus (we argue) inadequately: e.g., moverscore @xcite leverages bert representations fine-tuned on nli. @xcite train (pre-bert) nli-inspired architectures on mt datasets. in contrast, we show that by directly leveraging nli systems, much better adversarial and standard benchmark performances can be obtained. we call our novel metrics menli (metrics from nli). 3 3 code+data: http://github.com/cyr19/menli . ## adversarial setup following @xcite and others, we consider an array of adversarial attacks on evaluation metrics-we will give a motivation of our attacks from the perspective of errors committed by real text generation systems below. in contrast to @xcite and similar to the later published work of @xcite , we implement a preference-based setup, which does not need human annotators. the advantages of the preference-based setup are: (i) lower cost (e.g., no annotation costs), (ii) which can be especially relevant for non-english languages (e.g., in ref-free situations for mt), and (iii) which allows adversarial evaluation at larger scale, yielding more robust estimates of performance. the challenge of the preference setup is to cleverly determine text pairs to compare. in our design, we use an anchor text (either the reference ref or the source src), a paraphrase cand para of the anchor text, and an adversarial text cand adv which is maximally similar to the anchor text, but contains an adversarial attack. we expect a good metric m to prefer cand para over cand adv : for ref-based metrics, we apply the perturbation templates to ref to construct cand adv . in contrast, for ref-free mt metrics, we first translate the source src using google translate to a translation r and then perturb r to obtain cand adv . we introduce r to increase the similarity of cand adv to src; e.g., we assume that google translate translates more literally, i.e., closer to word-by-word translations, than human translators. this may be important to construct challenging test cases, cf. §6 and our above discussion. for ref-free summarization, we apply the perturbation templates to a document r which is maximally similar to src; details follow. cand para we use different ways to obtain cand para , because different kinds of paraphrases may yield more/less difficult test cases for metrics. we will analyze this in §6. in particular, we use data from ( 1 ) paws @xcite , (2) paws-x @xcite , (3) wmt20-news-commentary-v15 german-to-english @xcite erate cand para for mt evaluation metrics, and (4) summeval for summarization metrics. a summary with attributes is shown in we select the data for two closer language pairs: german-to-english and french-to-english, and two more distant language pairs: chinese-to-english and japanese-to-english. accordingly, we create error source mt hypothesis mismatch/verb pay attention to (follow) suning.com service account mismatch/adj. not bad, the picture quality of playing games is really fragrant (good) pronoun/addition bought it for his (my) son, he said it was good. name on the same day, us secretary of transportation zhao xiaolan (elaine lan chao), us congressman meng zhaowen (grace meng) and dong jiling (chiling tong), founding president of the international leaders foundation, spoke at the meeting respectively. omission i'll review your account, one moment, please. ich werde ihr konto [...] ( überpr üfen), einen moment bitte. mismatch/noun listen, i don't want to make my people mad," she said. ,,hör zu, ich will mein volk (meine leute) nicht verrückt machen'', sagte sie. pronoun williams wasn't the only one who received a fine at this year's wimbledon, though hers was the most costly. williams war nicht die einzige, die beim diesjährigen wimbledon eine geldstrafe erhielt, obwohl sie (ihre) die teuerste war. we obtain cand para as in the case of paws back . we refer to the adversarial dataset induced from summeval as se adv in the remainder. we obtain cand para as in the case of paws back . 5 real-world motivation of attacks modern text generation systems are prone to many of the errors we investigate in this work. for example, @xcite show, based on fine-grained human error annotations 5 as we generate our adversarial test instances fully automatically from backtranslation or automatic tools, they may contain some errors (including upper-/lower-case). for example, we note that in cand para , ''. . . billion dollars'' is sometimes incorrectly formulated as ''. . . dollars billion''; however, such cases occur only in ∼1% of all test cases for number error, which we argue is still on an acceptable noise level. @xcite , that translations generated by state-of-the-art mt models still contain many accuracy-related errors (e.g., addition and omission of information, inappropriately informal pronouns) and sometimes even fluency-related errors (e.g., wrong spelling). negation handling is also frequently discussed as an issue of modern mt systems @xcite @xcite . in summarization, system summaries are often factually inconsistent with source documents in terms of numbers, named entities and assigning quotations to a particular person, etc. @xcite @xcite . more generally, hallucination (of which addition/mismatches/etc. may be considered special cases) is a particular worrisome limitation of recent large language models @xcite . in task metrics mt ref-based moverscore (zhao et al., 2019), bertscore (zhang et al., 2020), bartscore (yuan et al., 2021), sentsim (song et al., 2021), comet (rei et al., 2020b), bleurt (sellam et al., 2020) ref-free comet, sentsim, xmoverscore (zhao et al., 2020) summarization ref-based bartscore, discoscore (zhao et al., 2023), moverscore, bertscore ref-free bartscore, supert (gao et al., 2020) task datasets mt segment-level wmt15-17, wmt20-21 system-level wmt20-21 adversary ref-based: paws ori/back , wmt20 de , xpaws de ; ref-free: xpaws de/fr/zh/ja , wmt20 de summarization summary-level realsum (bhandari et al., 2020) system-level realsum, summeval adversary se adv , rank19 (falke et al., 2019) (ref-free only) ## experiment results before outlining our main results in §5.1 (mt) and §5.2 (summarization), we first discuss good pooling strategies for nli metrics. pooling strategy we determine the pooling strategy for nli metrics in mt evaluation from (1) the accuracy on the adversarial datasets and ## discussion & analysis selected failure cases of metrics: nevertheless, reagan and i were able to create a constructive climate through constantcontact and personal interaction. nicole and i were nonetheless able to create a reservoir of constructive spirit through constant outreach and face-to-face interaction. no metric is good everywhere: across distinct dimensions, different metrics perform differently, indicating that they capture varying aspects. for example, nli metrics are not so good on fluency adversarial attacks, e.g., typos. this may be unsurprising, given that fluency is a low-level phenomenon while nli concerns high-level logical relationships between sentences (some fluency phenomena would best be treated by switch-ing to a lower-level representation space, such as character-level @xcite ; this could seamlessly be integrated in existing nli models). the nli metrics are also weaker concerning segment-level mt evaluation on standard benchmarks. however, nli metrics alone perform surprisingly well: in ref-based mt, they win on 7 out of 19 dimensions (12 adversarial phenomena and 7 standard datasets, evaluated segment-and system-level), only beaten by bleurt (8 wins); ref-free, they win 5 out of 19 dimensions, second only to comet (11 wins). in ref-based summarization, they are clearly ahead of all standard metrics, winning not only 8 out of 12 adversarial dimensions, but also system-level litepyramid, consistency and fluency (thus, 11 out of 18 wins), clearly ahead of bartscore-p (4 of 18); ref-free, they are also best and win 13 out of 18 dimensions. the best overall metrics, measured as average performance over standard and adversarial datasets, always include nli: for ref-based mt, this is bleurt+0.2×nli-r, for ref-free mt, it is comet+0.3×nli-d. for summarization, nli-r alone and combined with bartscore-f perform best on average. rescaling: the min-max normalization we used (a standard technique for normalizing data in machine learning, typically applied to input features) for metric combination requires batch processing. it is necessary to account for the different ranges of metrics, e.g., some metrics take negative values. an alternative would include to enforce more formal constraints on evaluation metrics, i.e., that they should take outputs in [0,1]. when applying our combined metrics in practice, one could also replace them by surrogate metrics trained on the outputs of the original combined metrics or simply take the min-max values inferred from the datasets already evaluated on-the larger these datasets the more reliably are min and max estimated. sensitvity to w nli : having different weights w nli for different tasks is undesirable, because it requires considering each task individually. however, in our experiments, we found that all small w nli (below 0.5) yield good performances and are thus safe choices: they increase adversarial robustness and also lead to better metrics on standard benchmarks. ## concluding remarks in this work, we explored nli as a general paradigm for evaluation metrics. we showed that nli metrics yield adversarial robustness, and are also strong-though not always state-ofthe-art-when it comes to standard metric evaluation benchmarks. by linearly interpolating established (bert-based) metrics with our nli metrics, we obtained high-quality metrics along both axes: adversarial robustness and standard benchmarks, with substantial gains over recent bert-based metrics. a potential reason why nli based metrics perform subpar on some standard benchmarks (especially in mt) is the training data mismatch, i.e., typical nli datasets contain many artificial sentences of the type ''a girl is playing on a piano''. a further limitation is that cross-lingual nli models are not yet high-quality enough and that most current nli models are sentence-level, not document-level-with a few recent exceptions @xcite . once these limitations of nli are overcome, we believe that even better performances from nli based metrics can be expected, which, we believe, is one of the most promising directions for future high-quality and robust evaluation metric design. future work should also consider nli metrics for other text generation tasks; the nli paradigm looks especially promising for tasks that require comparison with human references, which oftentimes involve the concept of logical equivalence.