CLIR eval: Evaluating Machine Translation as a Cross-Lingual Information Retrieval Task (2020)

Abstract: We present CLIReval, an easy-to-use toolkit for evaluating machine translation (MT) with the proxy task of cross-lingual information retrieval (CLIR). Contrary to what the project name might suggest, CLIReval does not actually require any annotated CLIR dataset. Instead, it automatically transforms translations and references used in MT evaluations into a synthetic CLIR dataset; it then sets up a standard search engine (Elasticsearch) and computes various information retrieval metrics (e.g., mean average precision) by treating the translations as documents to be retrieved. The idea is to gauge the quality of MT by its impact on the document translation approach to CLIR. As a case study, we run CLIReval on the “metrics shared task” of WMT2019; while this extrinsic metric is not intended to replace popular intrinsic metrics such as BLEU, results suggest CLIReval is competitive in many language pairs in terms of correlation to human judgments of quality. CLIReval is publicly available at https://github.com/ssun32/CLIReval .

## introduction machine translation (mt) is the task of automatically translating sentences from a source language to a target language. a natural question that arises is how do we determine whether an mt system is translating sentences well? one answer is that we can engage human translators to evaluate the translated sentences manually. unfortunately, evaluating translations can be relatively time-consuming and worse, the fact that the quality of translation is inherently subjective can lead to variations among different human translators. the desire for fast and consistent evaluation has led to the emergence of a plethora of automatic evaluation metrics such as bleu @xcite , ter @xcite , metor @xcite and beer @xcite . out of the aforementioned metrics, bleu has become the de facto evaluation metric for machine translation. it calculates the weighted average of n-gram precision between a translated sentence and a reference sentence. nevertheless, bleu, too, has its problems. for example, @xcite showed that an improved bleu score does not represent an actual improvement in translation quality. there are also some proposals to evaluate the quality of translations with the help of extrinsic proxy tasks. @xcite collected short english documents from various domains and created yes and no questions in czech. they then translated the english documents into czech and evaluated the quality of the mt systems based on human performances on the documents and questions in czech. @xcite translated a dataset of german reading comprehension tests into english with various mt systems such as google translate and bing translate and judged the quality of translations based on human performances on the translated reading comprehension datasets. unfortunately, these external tasks suffer from the same scalability and consistency issues as manual evaluation. one downstream task that relies heavily on mt but has not been used as a method to evaluate mt systems is the task of cross-lingual information retrieval (clir). clir is a task in which search queries are issued in one language, and the retrieved relevant documents are written in a different language. two commonly used methods in clir are query translation, where queries are translated into the same language as the documents and document translation where documents are translated into the same language as the queries @xcite @xcite . a monolingual ir system is then used to obtain search results. clir is an active field of research, and previ-ous works suggest that the performance of clir correlates highly with the quality of the mt @xcite @xcite . therefore, we expect ir metrics to be good indicators of the quality of translations. unfortunately, there is currently no publicly available tool to facilitate research in this area, and this motivates us to design and implement clireval. clireval is a lightweight python-based mt evaluation toolkit that consumes the same inputs as other automatic mt evaluation tools such as multibleu.perl and sacrebleu @xcite and does not require any additional annotated clir data. instead, it automatically transforms inputs into a synthetic clir dataset on the fly with the help of an information retrieval (ir) system. it implements the document translation approach to clir, where mt translations are viewed as documents and indexed using a commonly-used search engine (elasticsearch). as a case study, we test clireval on the metrics shared task of wmt2019 @xcite , which measures the pearson correlations (r) between automatically generated mt metrics and human judgments. results show that clireval consistently performs at the level of @xmath0.9 and is on par or even outperforms popular metrics such as bleu on multiple language directions. further, this is achieved without using external data or doing domain-based parameter tuning. these promising results highlight the potential of clir as a proxy task for mt evaluation, and we hope clireval can facilitate future research in this area. our key contributions in this work can be summarized as follows: ## approach given a set of source documents s, an mt system φ converts s into a set of translated documents, @xmath1. intrinsic mt metrics directly calculate an aggregated score between the sentences in t and sentences in r, where r is a set of reference documents. 2 we propose an alternative way to evaluate φ by first converting it into a proxy clir task and then evaluate the mt system with extrinsic ir metrics. first, clireval extracts a set of synthetic search queries q from r. second, given a monolingual information retrieval (ir) engine ρ, we can run these queries q over the document collection r to obtain a set of "relevant" documents for q. we use the notation ρ(q, r) to refer to this set of desired returned search results. now, our goal is to evaluate the quality of the translation @xmath2. we index the documents t into the ir engine and submit the same queries to obtain the search results ρ(q, t ). finally, we can measure the performance of the clir system by comparing ρ(q, t ) to ρ(q, r), and calculating ir metrics such as mean average precision. this approach makes several assumptions. first, clireval implements the document translation approach to clir and evaluates mt quality in that context; additionally, we assume that ρ is a robust and reasonable ir engine that can be used across a wide range of situations. second, we assume r contains the "correct" translations of s, and that ρ(q, r) is a good approximation of the optimal search results. third, we assume that automatically-generated q can mimic that actual information needs of manually-crafted queries. if these caveats are acknowledged, then clireval is a reasonable tool for mt evaluation. 3. query both instances of ir system with the same set of generated search queries. ## design and implementation details we emphasize that the above steps are achieved with a single easy-to-use script: clireval is as simple as executing the following command: ## conclusions we present clireval, an open-source pythonbased evaluation toolkit for machine translation. rather than directly evaluating translated sentences against reference sentences, clireval transforms the inputs into the closely related task of clir, without the need for annotated clir dataset. the aim of this project is not to replace current automatic evaluation metrics or fix the limitations in those metrics, but to bridge the gap between machine translation and cross-lingual information retrieval and to show that clir is a feasible proxy task for mt evaluation. our case study on the wmt2019 metrics shared task further highlights the potential of clir as a proxy task for mt evaluation, and we hope that clireval can facilitate future research in this area.