P ar BLEU : Augmenting Metrics with Automatic Paraphrases for the WMT ’20 Metrics Shared Task (2020)

Abstract: We describe parBLEU, parCHRF++, and parESIM, which augment baseline metrics with automatically generated paraphrases produced by PRISM (Thompson and Post, 2020a), a multilingual neural machine translation system. We build on recent work studying how to improve BLEU by using diverse automatically paraphrased references (Bawden et al., 2020), extending experiments to the multilingual setting for the WMT2020 metrics shared task and for three base metrics. We compare their capacity to exploit up to 100 additional synthetic references. We find that gains are possible when using additional, automatically paraphrased references, although they are not systematic. However, segment-level correlations, particularly into English, are improved for all three metrics and even with higher numbers of paraphrased references.

## introduction one of the major challenges faced when automatically evaluating machine translation (mt) outputs is that there are almost always multiple correct translations of a sentence, and an automatic metric should be able to reward them all. some of the most widely used mt metrics, including bleu @xcite and chrf++ @xcite , rely on a surface-form comparison of mt outputs to a human-produced reference translation. both metrics support the use of multiple references. however, even for metrics that support multiple references, human-produced references are expensive to produce and so are rarely available. to overcome this problem, metrics that do not rely on the surface form of reference translations have been developed. one example is esim @xcite , which uses contextual embeddings with the aim of creating an abstract meaning representation of the reference, with the potential of covering all translations with the correct meaning. we explore an alternative way of increasing the capacity of mt metrics to reward multiple valid translations: create additional references by automatically paraphrasing the original reference. there have been previous efforts to provide some sort of paraphrase support, mostly concentrating on synonyms @xcite @xcite . however, we base our work on a more recent attempt to improve bleu using diverse automatic parahrasing with high quality mt-style sentential paraphrasing @xcite . we put this to the test in the wmt'20 metrics shared task by applying @xcite approach to three different metrics: bleu, chrf++ and esim. we compare the different metrics' capacity to exploit automatically generated multiple references. we choose to use diverse paraphrases produced using prism @xcite , since they are available in multiple languages, including most languages of the wmt shared task. we find that gains in correlation are possible, but this depends largely on the language direction and on whether the metric is system-or segment-level. the most positive gains are seen at the segment level, especially for into-english and even at higher numbers of additional paraphrases. this holds for all three metrics, despite esim relying on more abstract semantic representations. ## overview of base metrics in an extension of @xcite , we augment three base metrics with automatic paraphrasing. the metrics vary in the basic units of comparison between mt outputs and the reference. bleu and chrf++ compare surface representations, bleu at the token level, whereas chrf++ also takes into account character n-grams. esim is an embedding-based metric, which aims to cap-ture the semantic relatedness of the sentences. a description of each base metric can be found below. ## experiment setup paraphrase generation we use the prism system to generate paraphrases. prism is a manymany multilingual nmt system covering 39 languages, including all of those of wmt 2019, except gujarati. in their submission to the wmt 2020 metrics task, @xcite retrained prism with five additional languages: gujarati (for wmt'19), and inuktitut, khmer, pashto, and tamil (for wmt'20). this provided almost complete coverage of the wmt 2019 and 2020 languages. we use this same model. by design, prism approaches paraphrasing as a zero-shot translation task. as a result, while good for scoring, it is not a particularly good generative model, in terms of being able to produce diverse outputs. @xcite have tried to address this, but their implementation does not produce n-best lists. we therefore produce n-best lists from the model using fairseq's built-in diverse beam search tool. for every reference in the wmt19 and wmt20 news test sets, we generate a 100-best list (vijayakumar et al., 2016). 4 figure 1 shows examples of paraphrases of two fi-en wmt'20 references. note that the paraphrases are divere and generally of high quality. however, the later paraphrases may be noisier. ## results the results are reported in the following three sections for each paraphrase-augmented metric: parbleu (section 4.1), parchrf++ (section 4.2) and paresim (section 4.3). we report results for up to 100 additional paraphrased references, except for paresim, where we report up to 50 additional references due to the length of time needed to calculate results. there are some general trends: ## conclusions and future work the goal with any metric is to balance accuracy with ease-of-use. for our submission to the wmt20 metrics task, we extended our work investigating paraphrased english references @xcite , by using a multilingual paraphraser. one component of ease-of-use, particularly for a metric, is to avoid highly language-specific parameter searches. our work here used a single model and diversity parameter setting. it is possible that other approaches would yield more success: for example, varying the number of references based on reference length or complexity, or looking at other diverse generation techniques. however they are not guaranteed to work and raise questions about the usefulness of extending surface-based metrics in the neural age. bleu is appealing because of its simplicity and universality, but the emerging evidence (cf. @xcite ) suggest that the most promising approach for future work in mt evaluation is in model-based deep-learning approaches. what is encouraging and also somewhat surprising is that the embedding-based esim also seems to benefit from the addition of automatically paraphrased references at the segment level, especially into english.