M u LER : Detailed and Scalable Reference-based Evaluation (2023)

Abstract: We propose a novel methodology (namely, MuLER) that transforms any reference-based evaluation metric for text generation, such as machine translation (MT) into a fine-grained analysis tool. Given a system and a metric, MuLER quantifies how much the chosen metric penalizes specific error types (e.g., errors in translating names of locations). MuLER thus enables a detailed error analysis which can lead to targeted improvement efforts for specific phenomena. We perform experiments in both synthetic and naturalistic settings to support MuLER’s validity and showcase its usability in MT evaluation, and other tasks, such as summarization. Analyzing all submissions to WMT in 2014-2020, we find consistent trends. For example, nouns and verbs are among the most frequent POS tags. However, they are among the hardest to translate. Performance on most POS tags improves with overall system performance, but a few are not thus correlated (their identity changes from language to language). Preliminary experiments with summarization reveal similar trends.

## introduction reference-based evaluation of text generation plays a uniquely important role in the development of machine translation @xcite , summarization @xcite , and simplification @xcite among many other sub-fields of nlp. it allows a scalable, cheap evaluation that often correlates at the system-level with human evaluation. however, reference-based evaluation metrics tend to produce a bottom line score, allowing little to no ability for a fine-grained analysis of the systems' strengths and weaknesses. such an analysis is important, for example, for targeted development efforts that focus on improving specific phenomena, or for better identifying scenarios in which the 1 our codebase is found here: https://github.com/tai314159/muler figure 1 : illustration of muler for the feature noun. two masking strategies are employed on the reference and the candidate -oracle masking max(r, c), and anti-oracle masking min(r, c). σ is the task's metric (e.g. bleu, rouge). muler thus yields a decomposition of any evaluation metric, to more focused measurements of the system's performance on span-level and word-level features, such as pos tags, named entity types, sentence sentiment etc. moreover, the methodology and code can be expanded to features of choice. in providing a per-phenomenon picture of system performance, muler is similar to challenge set approaches to evaluation (see §6). however, muler takes a more naturalistic approach, and narrows the evaluation to the test examples that contain a particular feature. given an evaluation metric (e.g., bleu) for a text generation task (e.g., mt) and a feature of interest of the system's output (e.g., performance on adjectives), muler operates as follows (see §2.1): it masks the feature in both the reference and the prediction by the same token (e.g., replace each adjective with a placeholder "adj"). this can be seen as an oracle adaptation to the output, that changes the span with the feature to agree with the reference. muler's score is the (normalized) difference between the metric score over the masked texts and the score over the original ones. we present results from mt as well as summarization and synthetic paraphrasing. in addition, we perform synthetic experiments to validate muler's effectiveness and usability. our experiments show that muler can measure performance on a particular feature ( §5), and reveal some previously unreported patterns in established mt systems ( §4). for example, while translation of nouns and verbs improved over the years, translation of named entities improve only for some categories §4.2. ## methodology the muler methodology seeks to gain insight as to the performance of a text generation system s according to a given metric σ on instances with the feature f . the feature is a dimension along which the system is evaluated that can be automatically detected given text. examples of features here may be pos tags, named entity types, morphological categories, among others. muler operationalizes this notion as improvement in the score of s according to σ, if s would have correctly predicted all instances of this feature. for scale, this improvement is compared to the overall possible improvement (the score is defined in §2.3). to assess that, muler creates an oracle where the feature f is perfect and an antioracle where it is fully wrong (cf. §2.2). ## experimental setup evaluation metrics. as reference-based metrics, we consider bleu @xcite , bertscore @xcite and rouge @xcite . bleu was developed to measure machine translation quality, and focuses on precision. rouge is made for summarization and focuses on recall. both are based on overlapping n-grams, while bertscore, a metric for text generation quality, is based on similarity between contextualised embeddings. for these metrics, the basic unit of evaluation is a sentence, as it compares between a reference sentence (a human translation) and a candidate sentence (an output of a system). features. we experiment with several feature types, each separated into different features: pos tagging, ner and dependency features (see app. §9 -for full description). sentence scorers. as dedicated scorers, we look at sentiment analysis, concreteness, valence, dominance and arousal (cf. app. a.) released library specifications. upon acceptance, we will share a library of code. the library allows using the metrics used in this paper as well as easily defining new ones. it reports muler variants as well as discrepancy breakdowns ( §2.5). ## validation experiments in this section, we perform various synthetic experiments to check the validity of muler. for a given feature f , let f be the set of words tagged as f (e.g., nouns) under τ , and α ∈ [0, 1]. ## related work automatic metrics are useful to assess systems and we base our work on them (see §3). other lines of work study a specific property and propose evaluation measures for it. for example, addressing hallucinations @xcite or measuring grammaticality @xcite . we share the aspiration to a more fine-grained form of evaluation with these works. there are methods for analyzing performance in a more fine-grained manner. for example, evaluation with minimal changes to the input @xcite and challenge sets @xcite . few methods highlight patterns rather than predefined properties, by contrasting texts (e.g. reference and output) @xcite . in a sense, muler stands in the middle between those, it highlights a closed set of traits, but it is extendable. ## conclusion we presented a novel methodology (muler) to decompose any reference-based score into its finegrained components, making it possible to obtain a detailed picture of text generation systems' performance, instead of a bottom-line score. muler filters and dissects naturalistic data to highlight phenomena in the generated text. we validated muler using a set of synthetic experiments ( §5). applying muler to off-the-shelf systems shows ( §4) that different systems' strengths and weaknesses are varied, even when their overall performance is alike, and detect interesting trends over the years. our work creates an avenue for further research into more fine-grained evaluation metrics and provides a tool to understand system behaviour. in future work, we plan to extend muler to more complex features such as long-distance syntactic dependencies.