NMTS core: A Multilingual Analysis of Translation-based Text Similarity Measures (2022)

Abstract: Being able to rank the similarity of short text segments is an interesting bonus feature of neural machine translation. Translation-based similarity measures include direct and pivot translation probability, as well as translation cross-likelihood, which has not been studied so far. We analyze these measures in the common framework of multilingual NMT, releasing the NMTScore library. Compared to baselines such as sentence embeddings, translation-based measures prove competitive in paraphrase identification and are more robust against adversarial or multilingual input, especially if proper normalization is applied. When used for reference-based evaluation of data-to-text generation in 2 tasks and 17 languages, translation-based measures show a relatively high correlation to human judgments.

## introduction measures of paraphrastic similarity aim to quantify the degree to which text segments mean the same thing. such measures can be used to identify paraphrases, and also to automatically evaluate text generation by estimating the similarity of model outputs to human-written references. neural machine translation (nmt) enables several similarity measures as a by-product of learning to estimate the probability of translations @xcite @xcite . these measures are promising given that they naturally leverage parallel corpora and might pay more attention to in this paper, we consider three distinct subtypes of translation-based similarity, as visualized in figure 1 . first of all, a straightforward approach is to estimate the direct translation probability of sentence a when translating from sentence b (junczys-dowmunt, 2018; @xcite . secondly, pivot translation probability is an estimate of how probable it is to arrive at sentence a when pivoting through an intermediary language @xcite . finally, we propose to estimate translation cross-likelihood: the likelihood that a translation of b into some language could also be a translation of a. we release the nmtscore library, using manyto-many multilingual nmt to implement these measures. 1 all three measures are competitive in paraphrase identification across 9 languages, compared to other general-purpose similarity measures, and especially on adversarial examples. we find, however, that normalizing the measures with reconstruction probability is important. they also work well cross-lingually, where pivot translation probability performs best while cross-likelihood has the advantage of not requiring an explicit specification of the input's languages. with respect to reference-based evaluation, we show that nmtscore is a competitive evaluation metric for data-to-text, according to human judgments from the webnlg challenge @xcite and from a multilingual amr-to-text evaluation @xcite . taken together, multilingual nmt offers a relatively precise and often complementary perspective on paraphrastic similarity, with little correlation to other metrics. in summary, we make the following main contributions: ## translation-based similarity measures the similarity of two sentences a and b can be measured in several ways by using a multilingual translation model θ. such a model accepts multiple source languages and can translate into multiple target languages. usually only the target language needs to be specified, e.g. with a target language token, and thus we use θ ℓ to denote a model that is conditioned on a target language ℓ. ## paraphrase identification in this section we compare the similarity measures using paraphrase identification test sets in multiple languages. the test sets contain pairs of sentences that have been annotated with whether the sentences are paraphrases or not, yielding a binary classification problem. for datasets with a validation split, we determine thresholds that optimally separate the validation set for each measure; we then apply the thresholds to the test set to compute the accuracy of the measures. if there is no validation set, we report the area under the curve (auc) on the test set. ## evaluation of data-to-text generation we now turn to a different application of sentence similarity, namely the reference-based evaluation of data-to-text generation. while the good performance of the translation-based measures on paraphrase identification is encouraging, this setting poses slightly different requirements on text similarity metrics. specifically, it is not necessary that the similarities of paraphrases and non-paraphrases are completely separable, but only that multiple hypotheses are correctly ranked with respect to a shared reference. moreover, it is relevant in which direction the measure is calculated. below we separately evaluate both directions: sim(hyp|ref) and sim(ref|hyp), as well as their average. ## related work various strategies have been suggested to leverage translation for paraphrastic similarity. most related to translation cross-likelihood is perhaps the work of @xcite , who extracted multiple translations from a bilingual parallel corpus assuming that sentences that are aligned to the same counterpart in the other language have similar meaning. in this paper, we revisit this idea in the context of nmt. a different approach was pursued by @xcite , who applied phrase-based statistical mt to estimate round-trip translation probability. this corresponds to the nmt approach of @xcite , who also explore the use of multiple translation variants (multi-pivoting) or multiple pivot languages (multilingual pivoting). in a variation of this approach, @xcite and @xcite used round-trip translation to generate training data for a paraphrastic sentence embedding model. multilingual mt allows to avoid pivoting by using zero-shot paraphrasing. this has been exploited for model analysis @xcite and reference-based evaluation @xcite . @xcite investigate alternative techniques to estimate direct translation probability for quality estimation. in the context of parallel corpus filtering (junczys-dowmunt, 2018), @xcite propose trie-constrained decoding to improve the efficiency of pairwise comparisons. future work could apply their method to the other translation-based measures. ## conclusion our analysis highlights theoretical and empirical properties of translation-based text similarity measures in a multilingual setting. direct translation probability is the most straightforward measure (an empirical comparison of inference times is found in appendix a). however, it treats inputs as target sequences, and we show that accuracy on paraphrase identification can be clearly improved by normalizing with reconstruction probability. pivot translation probability is advantageous especially when performing cross-lingual comparisons. finally, translation cross-likelihood has the advantage that it achieves symmetry with a single translation direction, and that the input languages need not be specified. the latter property also has interesting consequences for reference-based evaluation: the metric is expected to ignore whether the generated text matches the language of the reference. this can be seen as a rigorous disentanglement of adequacy from fluency. in comparison to baseline measures, translationbased measures are generally slower but show high accuracy on multilingual paraphrase identification, comparatively good reliability on reference-based evaluation of data-to-text generation, and little correlation to alternative measures. our findings thus show the usefulness of nmt translation probabilities for similarity tasks that require high attention to detail.