T rans Q uest at WMT 2020: Sentence-Level Direct Assessment (2020)

Abstract: This paper presents the team TransQuest’s participation in Sentence-Level Direct Assessment shared task in WMT 2020. We introduce a simple QE framework based on cross-lingual transformers, and we use it to implement and evaluate two different neural architectures. The proposed methods achieve state-of-the-art results surpassing the results obtained by OpenKiwi, the baseline used in the shared task. We further fine tune the QE framework by performing ensemble and data augmentation. Our approach is the winning solution in all of the language pairs according to the WMT 2020 official results.

## introduction the goal of quality estimation (qe) systems is to determine the quality of a translation without having access to a reference translation. this makes it very useful in translation workflows where it can be used to determine whether an automatically translated sentence is good enough to be used for a given purpose, or if it needs to be shown to a human translator for translation from scratch or postediting @xcite . quality estimation can be done at different levels: document level, sentence level and word level @xcite . this paper presents transquest, a sentence-level quality estimation framework which is the winning solution in all the language pairs in the wmt 2020 sentence-level direct assessment shared task @xcite . in the past, high preforming quality estimation systems such as quest @xcite and quest++ @xcite were heavily dependent on linguistic processing and feature engineering. these features were fed into traditional machine-learning algorithms like support vector regression and randomised decision trees @xcite , which then determined the quality of a translation. even though, these approaches provide good results, they are no longer the state of the art, being replaced in recent years by neural-based qe systems which usually rely on little or no linguistic processing. for example the best-performing system at the wmt 2017 shared task on qe was postech, which is purely neural and does not rely on feature engineering at all @xcite . in order to achieve high results, approaches such as postech require extensive pre-training, which means they depend on large parallel data and are computationally intensive @xcite . transquest, our qe framework removes this dependency on large parallel data by using crosslingual embeddings @xcite ) that are already fine-tuned to reflect properties between languages @xcite . @xcite show that by using them, transquest eases the burden of having complex neural network architectures, which in turn entails a reduction of the computational resources. that paper also shows that transquest performs well in transfer learning settings where it can be trained on language pairs for which we have resources and applied successfully on less resourced language pairs. the remainder of the paper is structured as follows. the dataset used in the competition is briefly discussed in section 2. in section 3 we present the transquest framework and the methodology employed to train it. this is followed by the evaluation results and their discussion in section 4. the paper finishes with conclusions and ideas for future research directions. ## dataset the dataset for the sentence-level direct assessment shared task is composed of data extracted from wikipedia for six language pairs, consisting of high-resource languages english-german (en-de) and english-chinese (en-zh), medium-resource languages romanian-english (ro-en) and estonian-english (et-en), and lowresource languages sinhala-english (si-en) and nepalese-english (ne-en), as well as a a russian-english (ru-en) dataset which combines articles from wikipedia and reddit @xcite . each language pair has 7,000 sentence pairs in the training set, 1,000 sentence pairs in the development set and another 1,000 sentence pairs in the testing set. each translation was rated with a score between 0 and 100 according to the perceived translation quality by at least three translators @xcite . the da scores were standardised using the z-score. the quality estimation systems have to predict the mean da z-scores of the test sentence pairs @xcite . ## methodology this section presents the methodology used to develop our quality estimation methods. our methodology is based on transquest our recently introduced qe framework @xcite . we first briefly describe the neural network architectures transquest proposed, followed by the training details. more details about the framework can be found in @xcite . ## evaluation, results and discussion this section presents the evaluation results of our architectures and the fine tuning strategies that can be used to improve the results. we first evaluate the transquest framework with the default setting (section 4.1). next we evaluate an ensemble setting of transquest in section 4.2. we finally assess the performance of transquest with augmented data. we conclude the section with a discussion of the results. the evaluation metric used was the pearson correlation (r) between the predictions and the gold standard from the test set, which is the most commonly used evaluation metric in wmt quality estimation shared tasks @xcite . we report the pearson correlation values that we obtained from codalab, the hosting platform of the wmt 2020 qe shared task. as a baseline we compare our results with the performance of openkiwi as reported by the task organisers @xcite . ## conclusion in this paper we evaluated different settings of transquest in sentence-level direct quality assessment. we showed that ensemble results with xlm-r-base and xlm-r-large with data augmentation techniques can improve the performance of transquest framework. the official results of the competition show that transquest won the first place in all the language pairs in sentence-level direct assessment task. transquest is the sole winner in en-zh, ne-en and ru-en language pairs and the multilingual track. for the other language pairs (en-de, ro-en, et-en and si-en) it shares the first place with another system, whose results are not statistically different from ours. the full results of the shared task can be seen in @xcite . in the future, we plan to experiment more with the data augmentation settings. we are interested in augmenting the training file with semantically similar sentences to the test set rather than augmenting with random sentence pairs as we did in this paper. as shown in the error analysis in section 4.4 the future releases of the framework need to handle named entities properly. we also hope to implement transquest in document level quality estimation too.