US core: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation (2023)

Abstract: The vast majority of evaluation metrics for machine translation are supervised, i.e., (i) are trained on human scores, (ii) assume the existence of reference translations, or (iii) leverage parallel data. This hinders their applicability to cases where such supervision signals are not available. In this work, we develop fully unsupervised evaluation metrics. To do so, we leverage similarities and synergies between evaluation metric induction, parallel corpus mining, and MT systems. In particular, we use an unsupervised evaluation metric to mine pseudo-parallel data, which we use to remap deficient underlying vector spaces (in an iterative manner) and to induce an unsupervised MT system, which then provides pseudo-references as an additional component in the metric. Finally, we also induce unsupervised multilingual sentence embeddings from pseudo-parallel data. We show that our fully unsupervised metrics are effective, i.e., they beat supervised competitors on 4 out of our 5 evaluation datasets. We make our code publicly available.

## introduction evaluation metrics are essential for judging progress in natural language generation (nlg) tasks such as machine translation (mt) and summarization, as they identify the state-of-the-art in a key nlp technology. despite their wide dissemination, classical lexical overlap evaluation metrics like bleu @xcite and rouge @xcite have difficulties judging the quality of modern nlg systems @xcite , necessitating novel metrics that correlate better with humans. lately, this has been a very active research area @xcite @xcite @xcite . @xcite recently, more and more supervised metrics are being proposed. e.g., bleurt @xcite 1github.com/potamides/unsupervised-metrics 2of course, the search for high quality metrics dates back at least to the invention of bleu and its predecessors. figure 1 : relationship between metrics m, vector spaces, parallel data, and mt systems: metrics build on (potentially deficient) multilingual vector spaces (a), and can be used to mine (pseudo-)parallel sentences (b), which in turn can be used to improve deficient vector spaces (c). @xcite parallel data can also be used to train mt systems (d), which can generate pseudo-references (e). conversely, metrics can also be optimization criteria for mt systems, which in turn can generate additional pseudo-parallel data through translation (f & g; not explored in this work). 2020) trains on human annotated datasets ranging from 5k-150k pairs, the comet @xcite ) models regress on 12k-370k data points and unite @xcite , before fine-tuning on the same data as comet, pre-trains on 5m-10m parallel sentences. of course, training on larger amounts of data leads to better metrics (measured on in-domain data), but also increases the risk of learning biases from the data @xcite and limits the applicability to domains and language pairs where supervision is available. here, we go the opposite route and try to minimize the amount of supervision as much as possible. we classify existing metrics making use of different types of supervision as follows (cf. training references parallel data type-1 ✓ (✓) (✓) type-2 ✗ ✓ (✓) type-3 ✗ ✗ ✓ unsupervised ✗ ✗ ✗ in contrast, we aim for fully unsupervised evaluation metrics (for mt) that do not use any form of supervision (cf. to achieve our goals, we employ selflearning @xcite and in particular, we leverage the following dualities to make our metrics maximally effective, cf. figure 1: 3we use the term sample efficiency in a generalized sense to denote the amount of supervision required. our contributions are: (i) we show that effective unsupervised evaluation metrics can be obtained by exploiting relationships with parallel corpus mining approaches and mt system induction; (ii) to do so, we explore ways to (a) make parallel corpus mining efficient (e.g., overcome cubic runtime complexity) and (b) induce unsupervised multilingual sentence embeddings from pseudo-parallel data; (iii) we show that pseudo-parallel data can rectify deficient vector spaces such as mbert; (iv) we show that our metrics beat three state-of-the-art supervised metrics on four of five datasets we evaluate on. ## background we take inspiration from three recent supervised (reference-free; type-3) metrics: xmover-score @xcite , distilscore @xcite , and sentsim @xcite . below, we review key aspects of them, and show where supervision plays a role. ## methods in this section, we introduce our fully unsupervised metric uscore. uscore builds upon the existing metrics xmoverscore, distilscore, and sentsim, but eliminates all supervision signals and instead leverages the dualities shown in figure 1 . in particular, we mine pseudo-parallel data from unsupervised metrics, which we use (iteratively) to (a) rectify deficient vector spaces (for xmoverscore) and to (b) train unsupervised mt systems which can generate pseudo-references (as pseudo-references are in the same language as the hypothesis, this eliminates problems of cross-lingual deficiency). furthermore, we use pseudo-parallel data to (c) induce an unsupervised sentence embedding model analogous to distilscore, which we can then (d) integrate with the unsupervised word based model analogous to sentsim. we now give details. ## experiments in this section, we evaluate all uscore variants at the segment level6 and compare them to type-1/2/3 upper bounds. we detail additional hyperparameters in appendix d. ## discussion throughout, we have presented a mix of results and analysis, which we now summarize and discuss. in figure 4 , we conducted an ablation study on the individual components of uscore wrd (on de-en). this showed that all three components (pseudoreferences, language model, wmd) matter; by itself, the lm is more important than the pseudoreferences which are more important than wmd. however, in combination, the lm is least important. we also showed that pseudo-parallel data can successfully rectify deficient multilingual vector spaces, similar to real parallel data, see figure 3 . we note, however, that pseudo-parallel data may introduce an important bias in our data sampling: namely, it may mine factually incorrect parallel we finally note that combining word and sentence-level models is meaningful, because they offer complementary views @xcite . @xcite also show that they capture orthogonal linguistic factors to varying degrees. such complementarity may also stem from different underlying vector spaces, i.e., mbert vs. xlm-r that we use in sentence-and word-level metrics. ## related work all metrics in this work presented so far treated the mt model generating the hypotheses as a blackbox. there also exists a recent line of work of so-called glass-box metrics, which actively incorporate the mt model under test into the scoring process @xcite . in particular, @xcite explore whether the mt model under test can be used to generate additional hypotheses @xcite . a crucial difference to our metrics is the required availability of the original mt model, which we are agnostic about. the mt models used in @xcite are all trained on parallel data, which makes their approach a supervised metric in our sense. other recent metrics that leverage the relationship between metrics and (mt) systems are prism @xcite and bartscore @xcite . we do not classify them as unsupervised, however, as prism is trained from scratch on parallel data and bartscore uses a bart model fine-tuned on labeled summarization or paraphrasing datasets. there are also multilingual sentence embedding models which are highly relevant in our context. @xcite , for example, fine-tune xlm-r on synthetic data translated with an unsupervised mt system. similar to our contrastive learning approach, the resulting embedding model is completely unsupervised. important differences are that our sentence embedding model can be improved iteratively and does not rely on an mt system. we leave a comparison to future work. finally, the idea of fully unsupervised text generation systems has originated in the mt community @xcite @xcite . given the similarity of mt systems and evaluation metrics, designing fully unsupervised evaluation metrics is an apparent next step, which we take in this work. ## conclusion in this work, we aimed for sample efficient evaluation metrics that do not use any form of supervision. in addition, our novel metrics should be maximally effective, i.e., of high quality. to achieve this, we leveraged pseudo-parallel data obtained from fully unsupervised evaluation metrics in three ways: we (i) remapped deficient vector spaces using the pseudo-parallel data, (ii) trained an unsupervised mt system from it (yielding pseudo references), and (iii) induced unsupervised multilingual sentence embeddings. to enable our approach, we also explored efficient pseudo-parallel corpus mining algorithms based on our metrics as an orthogonal contribution. finally, we showed that our approach is effective and can outperform three supervised upper bounds (making use of parallel data) on 4 out of 5 datasets we included in our comparison. in future work, we want to aim for algorithmic efficiency, include pseudo source texts as additional components (using the mt system in backward translation), and address the missing dualities discussed in figure 1 (i.e., use of metrics as optimization criteria and mt systems to generate additional pseudo-parallel data). further, our approach has substantial room for improvement given that we selected hyperparameters completely unsupervised or based on one high-resource language pair (de-en). thus, it will be particularly intriguing to explore weakly-supervised approaches which leverage minimal forms of supervision.