Training and Meta-Evaluating Machine Translation Evaluation Metrics at the Paragraph Level (2023)

Abstract: As research on machine translation moves to translating text beyond the sentence level, it remains unclear how effective automatic evaluation metrics are at scoring longer translations. In this work, we first propose a method for creating paragraph-level data for training and meta-evaluating metrics from existing sentence-level data. Then, we use these new datasets to benchmark existing sentence-level metrics as well as train learned metrics at the paragraph level. Interestingly, our experimental results demonstrate that using sentence-level metrics to score entire paragraphs is equally as effective as using a metric designed to work at the paragraph level. We speculate this result can be attributed to properties of the task of reference-based evaluation as well as limitations of our datasets with respect to capturing all types of phenomena that occur in paragraph-level translations.

## introduction automatic evaluation metrics have always been a critical component to the progress of research on machine translation (mt). as the field of mt moves beyond translating individual sentences to translating full paragraphs, book chapters, or documents @xcite @xcite @xcite , automatic metrics need to be designed to work on these longer texts. currently, how well automatic metrics agree with human judgments of paragraph translation quality is an open question. 1 few studies have meta-evaluated metrics on longer texts, and those that have are focused on the literary domain and are limited in the size of the evaluation dataset 1 translation beyond the sentence level is often referred to as document-level mt. however, there is no clear definition for the term "document." we use "paragraph" in this work because we feel it most accurately describes the length of text in our datasets. see §2 for more details on this. @xcite @xcite . in this work, we investigate training and meta-evaluating metrics for scoring paragraph translations using the benchmark workshop on machine translation (wmt) datasets that are widely used for metric development @xcite . due to the scarcity of human ratings of paragraph translations, we propose a method to create paragraph-level training and meta-evaluation datasets from the existing wmt sentence-level datasets ( §3). although these ratings are typically only used at the sentence level, they were collected on contiguous paragraphs and performed with document context, so they can be used as paragraphlevel datasets. we repurpose these datasets to benchmark existing sentence-level metrics as well as train new paragraph-level metrics for scoring paragraph translations ( §4). our experimental results are somewhat surprising. we find that there appears to be little evidence that training on paragraph-level data is beneficialat least given the limitations of our experimental setup. using metrics trained on sentence-level data only to directly score full paragraphs achieves comparable agreement to human ratings as metrics trained on paragraph-level data ( §6.1). sentencelevel metrics appear to generalize well to inputs much longer than they were trained on ( §6.2). we hypothesize these observations can be explained by the nature of evaluating translations and characteristics of our paragraph-level dataset ( §7). we speculate that long range dependencies-which paragraph-level metrics can model but sentencelevel likely do not-may not be too important for achieving high agreement with human ratings. further, due to the fact that our training and evaluation datasets assume a sentence alignment between the reference and hypothesis paragraphs, certain translation phenomena that sentence-level metrics may struggle to handle, like sentence or information reordering, are not well represented in the dataset, limiting our ability to show the benefits of training on paragraph-level ratings. the contributions of our work include (1) a method for constructing paragraph-level training and meta-evaluation datasets from sentence-level ratings, (2) an experimental study that demonstrates the comparable performance of sentenceand paragraph-level metrics, and (3) an analysis that aims to provide an explanation for our experimental observations. ## terminology throughout this paper, we use terms like segment, sentence, paragraph, and document to refer to different lengths of text. to the best of our knowledge, there are no agreed upon definitions for these terms in the mt literature, so here we define how they are used for the rest of the paper. we refer to the input text to an mt system or evaluation metric as a segment, irrespective of its length. traditionally, segments in mt have been roughly equivalent to one sentence, although sometimes they can be short phrases or even longer than a single sentence. regardless, we use sentence to refer to this unit of text since it accurately describes the most common text length that is widely used in mt. our work investigates evaluating paragraphs of text, which we define to be multi-sentence segments. we do not require that the paragraphs used in this work obey the traditional definition of a paragraph (i.e., a unit of text separated by a newline character). we refrain from calling this unit of text a document-which we consider to be all of the possible input text-since each document can be broken down into multiple paragraphs and the term paragraph more accurately describes the length of text we use. ## paragraph-level datasets the two main sources for training and metaevaluating mt metrics are the direct assessment (da) and multi-dimensional quality metrics (mqm; @xcite datasets that the workshop on machine translation (wmt) has collected as part of the yearly metrics shared task @xcite . the da ratings were done by a mixture of expert and non-expert raters (depending on whether the translation direction is into or out of english) who assigned a quality score in the range 0-100 to translated sentences. because of differences in rater behavior, the da scores are z-normalized per rater. 2 in mqm, expert raters identify error spans in translated sentences and assign each error a category and severity level, which are used to calculate a score for that error. a sentence's mqm score is defined as the sum of the errors' scores. training and meta-evaluating metrics at the paragraph level requires a collection of translated paragraphs and paragraph-level quality scores. luckily, the da data since 2019 and the mqm data can be considered to be paragraph-level ratings. the ratings were performed on contiguous blocks of sentences that were translated by the same system (e.g., the first k sentences per document are rated for a system). although the scores were collected at the sentence level, the ratings were done in context, meaning the raters had access to the document context for a sentence, so the scores should reflect paragraph-or document-level phenomena like discourse errors. therefore, we use the sentence-level da and mqm data to construct paragraph-level datasets as follows. for each document translated by a system, we run sliding window of size k sentences from the start to the end. if all k sentences in the window have been rated, those k sentences are concatenated together to become a paragraph instance and the window shifts by k. otherwise, the sliding window shifts by 1 and the process repeats. to maintain consistency between the sentence scores within a paragraph, we additionally require that every sentence is scored by the same rater. then, we define the paragraph-level scores to be the average da z-score or sum of mqm scores for each sentence in the paragraph. 3 the result is a dataset of rated paragraph translations of k sentences each. we apply this dataset construction approach to the da and mqm data for @xmath0, 2, . . . , 10 sentences per paragraph. the number of paragraphs is shown in figure 1 and the distribution of the lengths of the new translated paragraphs is shown in figure 2 . as k increases, the number of paragraphs decreases because there are fewer candidate paragraphs, while the length of the paragraphs increases, roughly by an expected factor of k. these paragraph-level da and mqm datasets are used to train and meta-evaluate paragraph-level metrics for the rest of this paper. ## paragraph-level metrics we explore two different methods for creating paragraph metrics: directly applying sentence-level metrics to paragraphs ( §4.1) and training metrics on paragraph-level data ( §4.2). ## results first, we directly evaluate how well metrics perform when used to directly score paragraphs ( §6.1), then we further examine the behavior of different paragraph-level metrics by analyzing their performances with the context of their sentence-level counterparts ( §6.2). ## discussion in theory, training on paragraph-level data should have advantages compared to training on sentencelevel data. the metric (1) should be able to handle longer input sequences, (2) it should be able to capture long range dependencies, and (3) it should be able to model different paragraph-level phenomena like information or sentence reordering. however, we were not able to demonstrate these advantages in practice, and we theorize why as follows. figure 7 : an english-to-spanish translation example where the reference translation does not have enough information to correctly evaluate the hypothesis. gender in spanish is marked on pronouns, and spanish is a pro-drop language, which means the pronoun can be omitted if the context is clear. in this example, the pronoun is dropped from the reference, so determining whether the pronoun used in the hypothesis requires taking into account the previous reference sentence. we suspect such examples are not frequent, and if they do exist, the information required to resolve the ambiguity is relatively local to the reference sentence. first, the analysis in §6.2 shows that sentencelevel metrics generalize well to significantly longer input, so advantage (1) may not be so relevant. we hypothesize that the scoring function learned by sentence-level metrics like sent-base or comet could score a token in the hypothesis based on some alignment to the reference using its relative position in the translation. this function would be agnostic with respect to the global positioning, and thus the scoring function would generalize well to longer inputs. if this were true, training on paragraphlevel data would not be necessary to obtain good performance on long sequences. second, evaluating translation quality seems to be a very "local" problem in the sense that modeling long range dependencies is not frequently necessary for evaluation. often, the reference phrase that aligns to a hypothesis phrase has enough information to accurately evaluate the hypothesis. if it does not, the information is likely nearby, not several sentences away (see figure 7 ). although the sentence-level metrics were not trained on multiple sentences, we suspect they are able to capture nearby dependencies across sentences when evaluating paragraphs. in theory, a paragraph-level metric would have the ability to model long range dependencies since it could observe them during training. however, if they are infrequent, advantage (2) over sentence-level metrics may be small. finally, the ability for our learned paragraph metrics to capture phenomena like sentence reordering is limited by our dataset construction method. since the paragraphs in our training and test sets come from mt systems that translated one sentence at a time, there are no phenomena like sentence reordering present in the datasets. therefore, the paragraph-level metric cannot learn to model such cases, and the metrics are never evaluated on them either. thus, the limitations of the dataset mean that we cannot demonstrate advantage (3). we believe that paragraph-level metrics are necessary for evaluating true paragraph translations, where mt systems can be more creative with how a full paragraph is translated, rather than paragraph translations that are created by translating individual sentences. we hypothesize that sentence-level metrics will not generalize well when there is no sentence alignment or there is significant information reordering. to accurately evaluate actual paragraph translations, metrics need to be trained on similar data. future work should invest in collecting human ratings for paragraph-level translations so that new metrics can be trained and evaluated. ## related work the vast majority of research on mt evaluation has worked at the sentence level @xcite @xcite @xcite @xcite @xcite @xcite , although there has been recent interest in moving beyond sentence-level evaluation. @xcite propose a method to incorporate document-level context into a sentencelevel metric by using the additional context when computing the representations for the hypothesis and reference sentences. although they use document context in their metric, it is still scores single sentences at a time, in contrast to the paragraphlevel metrics in our work that predict a score for entire paragraphs at once. then @xcite propose a document-level metric called blonde that targets evaluating discourse phenomena as opposed to overall translation quality (i.e., they do not model translation accuracy errors). to the best of our knowledge, ours is the first study aimed at training a learned metric that directly scores entire paragraphs. other studies that have evaluated sentence-level metrics beyond the sentence-level have done so in the literary domain. @xcite show that automatic metrics prefer mt output over human translations, and @xcite show that metrics prefer actual translations of paragraphs over sentence-by-sentence translations. our work is complementary to theirs as we focus on the news domain, train metrics on paragraph-level data, and evaluate on a much larger set of human ratings. it is not clear whether conclusions reached about metrics in the news domain will apply to the literary domain or vice versa. some researchers have developed challenge sets that can be used to probe how well metrics capture discourse phenomena that appear when translating more than one sentence at a time @xcite @xcite . however, these challenge sets can be trivial for reference-based metrics because the reference often resolves the ambiguity in the translation. to the best of our knowledge, a challenge set that forces reference-based metrics to use context outside of a single reference sentence during evaluation (see figure 7 ) does not exist. research on generating translations of text longer than single sentences directly use sentencelevel metrics to score translations @xcite @xcite @xcite . our work can be viewed as a justification for doing so. ## conclusion in this work, we proposed a method for constructing paragraph-level datasets for training and metaevaluating mt evaluation metrics from sentencelevel data. our experimental results showed that metrics trained on paragraph-level data do not necessarily out-perform those trained on sentencelevel data, potentially due to the fact that sentencelevel metrics seem to generalize well to longer inputs and limitations of our paragraph-level datasets. future work should invest in collecting human judgments for paragraph translations generated by mt systems that directly translate full paragraphs instead of translating one sentence at a time. such a dataset would be more likely to contain phenomena that do not exist at the sentence level, which we hypothesize would be more likely to require metrics designed to work at the paragraph level.