A Study in Improving BLEU Reference Coverage with Diverse Automatic Paraphrasing (2020)

Abstract: We investigate a long-perceived shortcoming in the typical use of BLEU: its reliance on a single reference. Using modern neural paraphrasing techniques, we study whether automatically generating additional *diverse* references can provide better coverage of the space of valid translations and thereby improve its correlation with human judgments. Our experiments on the into-English language directions of the WMT19 metrics task (at both the system and sentence level) show that using paraphrased references does generally improve BLEU, and when it does, the more diverse the better. However, we also show that better results could be achieved if those paraphrases were to specifically target the parts of the space most relevant to the MT outputs being evaluated. Moreover, the gains remain slight even when human paraphrases are used, suggesting inherent limitations to BLEU’s capacity to correctly exploit multiple references. Surprisingly, we also find that adequacy appears to be less important, as shown by the high results of a strong sampling approach, which even beats human paraphrases when used with sentence-level BLEU.

## introduction there is rarely a single correct way to translate a sentence; work attempting to encode the entire translation space of a sentence suggests there may be billions of valid translations @xcite . despite this, in machine translation (mt), system outputs are usually evaluated against a single reference. this especially affects mt's dominant metric, bleu @xcite , since it is a surface metric that operates on explicit n-gram overlap (see. (1) showing two ade1 our code and outputs are available at https:// github.com/rbawden/paraphrasing-bleu . mt1: this didn 't bother anybody . mt2: nobody was bothered by this . almost since its creation, bleu's status as the dominant metric for mt evaluation has been challenged (e.g., @xcite , @xcite ). such work typically uses only a single reference, however, which is a deficient form of the metric, since one of bleu's raisons d'être was to permit the use of multiple references, in a bid to represent "legitimate differences in word choice and word order." unfortunately, multiple references are rarely available due to the high cost and effort of producing them. one way to inexpensively create them is with automatic paraphrasing. this has been tried before @xcite , but only recently have paraphrase systems become good enough to generate fluent, high quality sentential paraphrases (with neural mt-style systems). moreover, it is currently unclear (i) whether adding automatically paraphrased references can provide the diversity needed to better cover the translation space, and (ii) whether this increased coverage overlaps with observed and valid mt outputs, in turn improving bleu's correlation with human judgments. we explore these questions, testing on all into-english directions of the wmt19 metrics shared task @xcite at the system and segment level. we compare two approaches: (i) generating diverse references with the hope of covering as much of the valid translation space as possible, and (ii) more directly targeting the relevant areas of the translation space by generating paraphrases that contain n-grams selected from the system out-puts. this allows us to compare the effects of diversity against an upper bound that has good coverage. we anchor our study by comparing automatically produced references against humanproduced ones on a subset of our data. our experiments show that adding paraphrased references rarely hurts bleu and can provide moderate gains in its correlation with human judgments. where it does help, the gains are correlated with diversity (and less so adequacy), but see diminishing returns, and fall short of the nondiverse method designed just to increase coverage. manual paraphrasing does give the best systemlevel bleu results, but even these gains are relatively limited, suggesting that diversity alone has its limits in addressing weaknesses of surfacebased evaluation metrics like bleu. the use of mt systems to produce paraphrases has also been studied previously. @xcite create pseudo-references by using out-of-the-box mt systems and see improved correlations with human judgments, helped by the systems being of better quality than those evaluated. this method was extended by @xcite , who filter the pseudo-references for quality. an alternative strategy is to use mt-style systems as paraphrasers, applied to the references. @xcite show that additional (paraphrased) references, even noisy ones, reduce the number of human references needed to tune an smt system, without significantly affecting mt quality. however their aim for coverage over quality means that their paraphrases are unlikely to be good enough for use in a final evaluation metric. despite the attention afforded to the task, success has been limited by the fact that until recently, there were no good sentence-level paraphrasers @xcite showed that neural paraphrasers can now outperform humans for adequacy and cost). attempts (e.g. @xcite using earlier mt paradigms were not able to produce fluent output, and publicly available paraphrase datasets have only been recently released @xcite . moreover, most works focus on synonym substitution rather than more radical changes in sentence structure, limiting the coverage achieved. structurally diverse outputs diverse generation is important to ensure a wide coverage of possible translations. diversity, both lexical and structural, has been a major concern of text generation tasks @xcite . state-of-the-art neural mt-style text generation models used for paraphrasing @xcite typically suffer from limited diversity in the beam. techniques such as sampling from the model distribution or from noisy outputs have been proposed to tackle this @xcite but can harm output quality. an effective strategy to encourage structural diversity is to add syntactic information (which can be varied) to the generated text. the constraints can be specified manually, for example by adding a parse tree @xcite or by specifying more abstract constraints such as rewriting embeddings @xcite . a similar but more flexible approach was adopted more recently by @xcite , who augment target training sentences with cluster pseudotokens representing the structural signature of the output sentence. when decoding, the top cluster codes are selected automatically using beam search and for each one a different hypothesis is selected. we adopt shu et al.'s approach here, due to the automatic nature of constraint selection and the flexibility afforded by constraint definition, allowing us to test different types of diversity by varying the type of sentence clustering method. ## generating paraphrased references we look at two ways to produce paraphrases of english references using english-english nmt architectures. the first (sec. 3.1) aims for maximal lexical and syntactic diversity, in a bid to better cover the space of valid translations. in contrast, the second (sec. 3.2) aims to produce paraphrases that target the most relevant areas of the space (i.e. that are as close to the good system outputs as possible). of course, not all outputs are good, so we attempt to achieve coverage while maintaining adequacy to the original reference by using information from the mt outputs. while less realistic practically, this approach furthers the study of the relationship between diversity and valid coverage. ## experiments our goal is to assess whether we can generate paraphrases that are representative of the translation space and which, when used with bleu, improve its utility as a metric. we therefore carry out experiments to (i) evaluate the adequacy and diversity of our paraphrases (sec. 5.2) and (ii) compare the usefulness of all methods in improving bleu's correlation with human judgments of mt quality (sec. 4.1). bleu is a corpus-level metric, and our primary evaluation is therefore its systemlevel correlation. however, it is often also used at the segment level (with smoothing to avoid zero counts). it stands to reason that multiple references would be more important at the segmentlevel, so we also look into the effects of adding paraphrase references for sentbleu too. ## metric correlation results the correlation results for each of the metrics (both system-and segment-level) for different numbers of additional references foot_3 (aggregated full results) are shown in tab. 3a and tab. 3b (for the de-en 500-sample subset). we aggregate the main results to make them easier to interpret by averaging over all into-english test sets (the ave. column) and we also provide the gains for the language pairs that gave the smallest and greatest gains (min and max respectively). full raw results can be found in app. d. system-level adding paraphrased references does not significantly hurt performance, and usually improves it; we see small gains for most languages (ave. column), although the size of the gain varies, and correlations for two directions (fien and gu-en) are degraded but non-significantly (shown by the small negative minimum gains). fig. 1 @xcite shows that for the diverse approaches, the average gain is positively correlated with the method's diversity: increased diversity does improve coverage of the valid translation space. this positive correlation holds for all directions for which adding paraphrases helps (i.e., all except fi-en and gu-en). for these exceptions, none of the methods significantly improves over the baseline, and random gives as good if not marginally better results. the constraints approach achieves the highest average gain, suggesting that it is more efficiently targeting the space of valid translations, even though its paraphrases are significantly less diverse (tab. 2). finally, and in spite of these improvements, we note that all systems fall far short of the best wmt19 metrics, shown in the last row. automatic paraphrases do not seem to address the weakness of bleu as an automatic metric. segment-level similar results can be seen at the segment level, with most diverse approaches showing improvements over the baseline (this time sentbleu) and a minority showing nonsignificant deteriorations (i.e., no change). the diversity of the approaches is again positively correlated with the gains seen (fig. 1 , bottom), with the exception of zh-en, for no easily discernable reason. the best result of the diverse approaches is again achieved by the sampled baseline. the constraint-based approach achieves good scores, comparable to sampled, despite an anomalously poor score for one language pair (for kk-en, with a degradation of 0.097. this approach also had the highest bleu scores, however, suggesting that the targeted paraphrasing approach here missed its mark. de-en 500-sentence subset the general pattern shows the same as the averages over all languages in tab. 3a, with the more diverse methods (especially sampled) resulting in the greatest gains. the human results also follow this pattern, resulting in the highest gains of all at the system level. interestingly, the constrained system yields higher average bleu scores than human (tab. 2) yet a comparable system correlation gain, indicating it targets more of the invalid translation space. for this particular subset, the constraints-based approach helps slightly more at the segment level than the system level, even surpassing the human paraphrases in terms of relative gains, despite it having remarkably less diversity. ## discussion does diversity help? in situations where adding paraphrases helps (which is the case for a majority however, diversity is not everything; although human has nearly twice the lexical diversity of sampled, it improves bleu only somewhat and harms sentence bleu. on the other side, targeted constraints have relatively low diversity, but higher correlation gains. diversity itself does not necessarily result in coverage of the space occupied by good translation hypotheses. what effect do more references have? diversity increases the more paraphrases there are and it is positively correlated with gains for most language directions. however, improvements are slight, especially with respect to what we would hope to achieve (using human references results in much more diversity and also greater improvements). the relationship between the number of extra references and system-level correlations shown in fig. 2 suggests that increasing the number of references results in gains, but for most test sets, the initial paraphrase has the most impact and the subsequent ones lead to lesser gains or even occasional deteriorations. similar results are seen at the segment level. de-en fi-en gu-en kk-en lt-en ru-en zh-en 0 4 8 % degraded beam sample random laser treelstm 0 4 8 % improved figure 3: % improved and degraded (with respect to singlereference sentence-bleu) for methods with +5 references. why are gains only slight? with respect to the sentbleu baseline, we calculate the percentage of comparisons for which the decision is improved (the baseline scored the worse translation higher than the better one and the new paraphraseaugmented metric reversed this) foot_5 and for which the decision is degraded (opposite reversal). the results (fig. 3 ) show that although all the systems improve a fair number of comparisons (up to 9.6%), they degrade almost as many. so, while paraphrasing adds references that represent the space of valid translations, references are indeed being added that match with the space of invalid ones too. interestingly, the same pattern can be seen for human paraphrases, 6.46% of comparisons being degraded vs. 8.30% improved, suggesting that even when gold standard paraphrases are produced, the way in which the references are used by sentbleu still rewards some invalid translations, though the balance is shifted slightly in favour of valid translations. this suggests that at least at the segment level, bleu is a balancing act between rewarding valid translations and avoiding rewarding invalid ones. some of these effects may be smoothed out in system-level bleu but there is still likely to be an effect. it is worth noting that for the two languages directions, fi-en and gu-en, for which diversity was negatively cor-related with correlation gain (i.e., diversity could be harming performance), the most conservative approach (random) leads to some of the best results. what is the effect on individual n-grams? we study which new n-grams are being matched by the additional references for the two language directions with the largest system-level correlation gain (ru-en and de-en). for each sentence, we collect and count the n-grams that were not in the original reference but where in the five paraphrased references of beam (missing n-grams), foot_6 accumulated across all test set sentences. we also looked at the most frequent n-grams not found at all, even with the help of the paraphrases (i.e., the unrewarded n-grams from sec. 3.2). the results are in unsurprisingly, most 1-grams are common grammatical words (e.g., a, of, to, in, the) that may be present (or not) in any sentence; it is hard to draw any conclusions. for 4-grams, however, we see some interesting patterns. present in both lists are acronym variants such as u . s . for 'united states' and p . m . for 'afternoon' or the 24-hour clock; their presence on both sides indicates success in sometimes grabbing this variant as well as failure to do so consistently. we also see phrasal variants such as , according to and , " he said. these last points corroborate a point made by @xcite 2) that references may omit these common variants. it also suggests a more focused method for generating paraphrases: identify a high-precision set of common variants, and ensure their presence in the set of references, via constrained decoding or other means (in the spirit of meteor's @xcite synonym-based matching). we note however, that our paraphrasing methods do seem to contain complementary information as they also tend to improve meteor too (see results in app. f). ## conclusion we studied the feasibility of using diverse automatic paraphrasing of english references to improve bleu. although increased diversity of paraphrases does lead to increased gains in correlation with human judgments at both the system and segment levels, the gains are small and inconsistent. we can do a slightly better job by using n newly matched ngrams missing ngrams 1 a (494) of ( 480 ) , ( 442 ) to ( 370 ) in ( 364 ) the ( 315 ) the ( 273 ) is ( 204 ) for ( 196 ) has ( 196 ) on ( 193 ) was ( 179 ) have ( 171 ) that ( 166 ) be ( 155 ) at (145) been ( 140 ) with ( 138 ) and ( 134 ) to ( 921 ) in ( 921 ) on ( 870 ) is ( 802 ) of ( 798 ) a ( 786 ) for ( 568 ) the ( 556 ) with ( 509 ) it ( 508 ) has ( 505 ) are ( 482 ) by ( 480 ) was ( 478 ) have ( 449 ) -( 443 ) at ( 437 ) as (426) which (386) 4 u . s . (63) the u . s (39) , as well as (19) p . m . (15) for the first time (13) in accordance with the (12) the united states , (11) in the united states (10) a member of the (10) of the united states (9) the u . s (9) . m . on (9) , in order to (9) the united states and (8) , of course , (8) . s . navy (8) . m . , (8) the chinese academy of (8) chinese academy of engineering (8) the renaming of the (7) u . s . (136) , according to the (99) , " he said (77) the u . s (55) of the united states (48) of the ministry of ( 39 ) the end of the (38) , " said the (37) same time , the (36) , such as the (36) as well as the (35) ( xinhua ) -( 34 ) and so on . (33) , he said . (32) the head of the (32) , the head of ( 31 ) , as well as (30) on the basis of (30) , and so on (29)