Multi-Hypothesis Machine Translation Evaluation (2020)

Abstract: Reliably evaluating Machine Translation (MT) through automated metrics is a long-standing problem. One of the main challenges is the fact that multiple outputs can be equally valid. Attempts to minimise this issue include metrics that relax the matching of MT output and reference strings, and the use of multiple references. The latter has been shown to significantly improve the performance of evaluation metrics. However, collecting multiple references is expensive and in practice a single reference is generally used. In this paper, we propose an alternative approach: instead of modelling linguistic variation in human reference we exploit the MT model uncertainty to generate multiple diverse translations and use these: (i) as surrogates to reference translations; (ii) to obtain a quantification of translation variability to either complement existing metric scores or (iii) replace references altogether. We show that for a number of popular evaluation metrics our variability estimates lead to substantial improvements in correlation with human judgements of quality by up 15%.

## introduction translation is an open-ended task with multiple valid solutions. there are often multiple equivalent translations for the same source sentence. this is due to inherent differences between languages and various sources of ambiguity, which is often impossible to solve without access to additional context. furthermore, the source might suffer substantial changes in translation due to translator's need to adapt it to the target audience. with rare exceptions, translations are not literal, they can differ from the source text at any linguistic level -lexical, syntactic, semantic or even discourseand still be considered correct. the ability to produce non-literal, more natural translations is one of the goals in the field of machine translation (mt). neural mt (nmt) approaches have certainly made significant progress in this direction. however, the diversity of possible outcomes makes it harder to evaluate mt models. evaluation metrics (or humans in the case of monolingual manual evaluation) are given a single reference translation against which to compare the mt output. @xcite found differences of up to 1 point on a 1-5 point quality scale (i.e. 20%) between groups of annotators who use different references for manual evaluation. in automatic evaluation, which computes a similarity score between mt output and human reference, they found differences of up to 6 bleu points depending on the reference used, showing that metrics strongly penalise perfectly correct translations that happen to be different from the reference provided. @xcite showed that if multiple human translations are used, any automatic mt evaluation metric achieves a substantially higher correlation with human judgments. however, multiple translations are hardly ever available in practice due to the cost of collecting them. alternatives strategies for modelling linguistic variation in automatic mt evaluation include using paraphrasing, synonyms, or comparing linguistic structures of mt output and the reference translation (e.g. semantic role labels) instead of surface forms ( §2). it is worth noticing that this line of work focuses on varying the reference translation. no existing work accounts for the diversity of possible mt outputs. instead of using multiple references or relaxing the string matching process, we use the mt system to generate multiple additional hypotheses representing potentially valid translation variations. we do so by exploring model uncertainty in output probability distributions. 1 to generate a diverse set of hypotheses from neural mt (nmt) systems, we leverage recent work on uncertainty quantification for neural networks ( §3). the additional hypotheses produced for a given source sentence are then used for evaluation with or without human references. intuitively, if some of the hypotheses match the reference, it is probable that the mt output under evaluation is also of high quality. 2 furthermore, we posit that the differences between system hypotheses produced for the same source capture uncertainty. the more similar they are among themselves, the higher the confidence of the model. as illustrated in figure 1 , this could provide additional information for discriminating translation quality when measuring the distance to the reference translation does not suffice. we devise various new metrics based on this intuition and obtain large improvements in correlation with human judgments over traditional reference-based metrics. our main contributions are as follows: ## related work meteor @xcite was the first mt evaluation metric to relax the exact match constraint between mt system output and reference translation by allowing matching of lemmas, synonyms or paraphrases. however, this requires linguistic resources which do not exist for most languages. character-based metrics @xcite also relax the exact word match constraint by allowing the matching of characters. however, ultimately they still assume a surfacelevel similarity between reference and mt. a more recent direction compares mt and reference sentences in the embedding space. @xcite extract word embedding representations for the two sentences and measure the (cosine) similarity between them. similarly, in @xcite @xcite two words are considered to match if their cosine distance in the embedding space is above a certain threshold. the embeddings are thus used to provide a binary decision. meant 2.0 @xcite and yisi @xcite also relies on matching of words in the embedding space, but this is only used to score the similarity between pairs of words that have already been aligned based on their semantic roles, rather than to find the alignments between words. finally, @xcite and @xcite perform the alignment in the embedding space using earth mover's distance with some special treatment for word order. all of these metrics are however still limited to variance in the words used (even in the continuous space), rather than more general stylistic or structural variations which can only be captured with multiple references. another way of incorporating linguistic variation is pseudo-reference approach by @xcite . they leverage various off-the-shelf mt systems to generate additional imperfect references and use them instead or alongside the original reference during evaluation. evaluation scores obtained using each of the pseudo references and the available human references are combined as features by training a classifier to predict human judgments. thus, this line of work implicitly learns the quality of the mt systems used to generate pseudo references. we revisit this idea in our paper by having pseudo-references as one type of diverse mt output. ## generating diverse hypotheses we posit that using multiple mt hypotheses can help automatic mt evaluation in two ways. first, the difference between them may reflect model confidence and potential ambiguity or complexity of the source. second, they provide an additional point of comparison with the reference, such that if the initial mt output is different from the provided reference due to acceptable linguistic variation, the risk of over-penalising this translation is lower. ## scoring with multiple hypotheses using the methods described above we are able to produce a set of mt hypotheses for each given source segment. the final dataset which we use for evaluation contains a human reference translation (r), the top mt output (o) and this set of alternative n mt hypotheses (@xmath0..h n }). we devise the following ways of combining similarities between possible translations and between these and the reference to obtain more accurate evaluation. this accuracy will be measured by pearson correlation with a direct assessment (da) score collected for the o translation, as is common practice in the evaluation metrics field @xcite . ## experimental settings to test whether our methods improve correlation with human judgments, we need to have access to the nmt model and human judgments for the translations generated by this model. this data is not generally readily available in evaluation campaigns such as metrics task at wmt conferences. below we describe two datasets that satisfy these conditions. they cover two different language pairs and two different domains. news english-czech dataset. we use available data from the wmt19 news translation task. we focus on the university of edinburgh's submission @xcite to the english-czech translation task, since its nmt model is available. the system was trained using the marianmt toolkit with a standard transformer architecture @xcite . details on model training and architecture are described in @xcite . for producing pseudo-references, we use all five "online" systems whose submissions were provided as part of the wmt19 translation task. human judgments were collected in the form of direct assessments (da) following the methodology proposed by @xcite , which suggests that 15 segment-level da judgements are required for trustworthy correlation analysis. however the number of da judgements in the wmt19 metrics task was much smaller. we select segments with at least two da annotations (795 segments with an average da score of 80.22) to minimise this issue, but the results reported here for english-czech should be interpreted with caution. wikipedia estonian-english dataset. this is a new dataset we collected which contains 1k sentences randomly selected from wikipedia articles in estonian and translated into english. two human reference translations were generated independently by two professional translators. all the nmt models were trained using the fairseq toolkit based on the standard transformer architecture @xcite and the training settings described in @xcite . we used publicly available parallel datasets for training the models: the rapid corpus of eu press releases @xcite and europarl @xcite , which amount to around 4m parallel sentences in total. a set of 400 segments were translated by the model variants described in §3 to assess the impact of uncertainty types. the following settings were used for model variants. for mc dropout we use dropout rate of 0.3, same as for training the basic transformer model. additional hypotheses were produced by performing n stochastic forward passes through the network with dropout, as described in §3. for this analysis we use @xmath1, which was shown to perform well for uncertainty quantification @xcite . we also test how the number of hypotheses affects the results (see appendix b). for moe we use hard mixture model with uniform prior and @xmath2. to produce the translations we generate from a randomly chosen component with greedy search following the settings in @xcite . for generating additional hypotheses with beam search the top-k sentences @xmath3..5] from the beam were used (@xmath4. for pseudo-reference approach we use three online systems: systran, google and bing. human judgements were given by professional translators following the flores setup @xcite which presents a form of da judgements @xcite . the annotators were asked to rate each sentence from 0-100 according to the perceived translation quality. specifically, the 0-10 range represents an incorrect translation; 11-29, a translation with few correct keywords, but the overall meaning is different from the source; 30-50, a translation with major mistakes; 51-69, a translation which is understandable and conveys the overall meaning of the source but contains typos or grammatical errors; 70-90, a translation that closely preserves the semantics of the source sentence; and 90-100, a perfect translation. each segment was annotated by up to 6 translators. raw scores were converted into z-scores, i.e. standardised according to each individual annotator's overall mean and standard deviation. the scores collected for each segment were averaged to obtain the final score. the judgments were collected for the 1k segments translated by the standard transformer model and for the 400 segments produced by four mt model variants in §3, resulting in a total of 1000 + 4 * 400 = 2600 source-mt pairs annotated with da judgments. the distribution of da scores for english-czech and 1k estonian-english datasets is shown in the appendix a. 4 ## results in this section, we present the results of our experiments for generating additional mt hypotheses ( §6.1) and the methods for exploiting similarities between them ( §6.2). ## conclusions we proposed to explore nmt model uncertainty to generate additional hypotheses for mt evaluation. we showed that by exploiting similarities in the space of translation hypotheses generated by the model, along with methods to effectively combine information from these multiple hypotheses, we can achieve more accurate estimation on the quality of mt output than standard referencebased comparison, including cases with multiple references. this suggests that model uncertainty alone can be more reliable for assessing mt quality than standard reference-based evaluation. this work can be extended in numerous ways. first, we plan to test whether similar observations will hold for more language pairs and text domains. second, the score combination strategies could be improved by learning weights for each component. finally, we would like to test this approach for comparing different mt systems.