Dealing with the Paradox of Quality Estimation (2021)

Abstract: In quality estimation (QE), the quality of translation can be predicted by referencing the source sentence and the machine translation (MT) output without access to the reference sentence. However, there exists a paradox in that constructing a dataset for creating a QE model requires non-trivial human labor and time, and it may even requires additional effort compared to the cost of constructing a parallel corpus. In this study, to address this paradox and utilize the various applications of QE, even in low-resource languages (LRLs), we propose a method for automatically constructing a pseudo-QE dataset without using human labor. We perform a comparative analysis on the pseudo-QE dataset using multilingual pre-trained language models. As we generate the pseudo dataset, we conduct experiments using various external machine translators as test sets to verify the accuracy of the results objectively. Also, the experimental results show that multilingual BART demonstrates the best performance, and we confirm the applicability of QE in LRLs using pseudo-QE dataset construction methods.

## introduction in the field of machine translation (mt), most of the representative metrics such as bleu @xcite and meteor @xcite are used to measure the quality of mt output by comparing it with the reference sentence. however, these evaluation metrics limit the amount of datasets owing to the need for a reference sentence @xcite . in cases where end users use mt, they do not have sufficient knowledge of the source or target languages. specifically, in the case of low-resource languages (lrls), people are often unfamiliar with such languages. in such cases, it is difficult to determine whether the translation results derived using mt have been translated well. recently, studies on quality estimation (qe) have been actively conducted to address this problem @xcite @xcite . in qe, the source sentence and the mt output are referenced to predict the quality of translation result. qe can be used to express the quality of mt output numerically, rank the results of several mt systems @xcite , and inform end users on mt system's level of trust. quality annotations resulting from the qe system also allows individuals who are unfamiliar with the translation languages to verify the quality of mt outputs @xcite . additionally, post-editing efforts can be reduced by filtering out poor-quality mt outputs @xcite . as a result, the importance of qe research has been emphasized in the field of mt. we found one paradox pertaining to this useful qe task. qe has an advantage in that it can make predictions about mt results without using a reference sentence. however, efforts to build datasets that require more expertise than building a parallel corpus must be made to ensure the progress of qe. these requirements also limit the construction of large qe datasets. we refer to this paradox as the paradox of qe, and we use methods for generating a pseudo-qe dataset to address this paradox. because it is difficult to obtain a parallel corpus for lrls and hinders to build a qe dataset for such languages, there are few qe studies on lrls, except for those provided by the conference on machine translation (wmt). based on these limitations, we conduct a study on sentence-level qe with a main focus on lrls. we construct a pseudo-qe dataset by automatically expanding korean-based monolingual or parallel corpora without using extra human labor. we conduct a comparative analysis between qe models based on various multilingual pretrained language models (mplms), and we confirm the possibility of creating a qe model for lrls through the experimental results. the contributions of this study are as follows: ## conclusion and future work this study points out a paradox in terms of the construction of data for qe tasks. to address this limitation, we propose two methods for generating a pseudo dataset. first, considering the limitations of data construction in low-resource language settings, we generate an rttbased pseudo-qe dataset using monolingual corpora, and second, we construct pseudo data using parallel data. the experiments are conducted using mplms that support korean, and mbart demonstrated the highest performance. by conducting tests using various external machine translators, we further confirm that the model trained using a pseudo dataset is not significantly skewed on a specific external translator. therefore, by leveraging pseudo-qe generation methods, we confirm that qe is also available in lrls, and induce the use of various applicability of qe in lrls. in our future studies, as we have seen the possibility of sufficient performance improvement for the result of experimenting with monolingual corpora, we plan to conduct further experiments to expand the amount of data to large-scale. we also plan to expand the proposed methodology to various language pairs and conduct detailed verification of the proposed methodology.