Explainable Quality Estimation: CUNI E val4 NLP Submission (2021)

Abstract: This paper describes our participating system in the shared task Explainable quality estimation of 2nd Workshop on Evaluation & Comparison of NLP Systems. The task of quality estimation (QE, a.k.a. reference-free evaluation) is to predict the quality of MT output at inference time without access to reference translations. In this proposed work, we first build a word-level quality estimation model, then we finetune this model for sentence-level QE. Our proposed models achieve near state-of-the-art results. In the word-level QE, we place 2nd and 3rd on the supervised Ro-En and Et-En test sets. In the sentence-level QE, we achieve a relative improvement of 8.86% (Ro-En) and 10.6% (Et-En) in terms of the Pearson correlation coefficient over the baseline model.

## introduction quality estimation (qe) or confidence estimation (ce) is a task of assessing the quality of machinetranslated text given the source without accessing the reference @xcite . qe can be assessed on sentence-level, wordlevel granularity or even document-level @xcite . sentence-level scores predict what score would a human annotator assign to the whole sentence; most commonly, direct assessment @xcite or hter @xcite serve as the golden standard. word-level qe indicates word-level errors in machine translation output or incorrectly translated words in the source. while automatic word-level scores are usually continuous, the gold truth is binary: some words are labeled as correct while some are labeled as wrong. this estimation provides an aid in the translation workflow. for instance, it can help to determine if the machine-translated sentence is good enough to be used as-is or if it requires a human translator for post-editing or translating from scratch @xcite . in this paper, we present our submission to the shared task of explainable quality estimation of 2nd workshop on evaluation & comparison of nlp systems @xcite ). foot_0 our solution is based on the xlm-r multilingual pretrained model @xcite . we first build a word-level quality estimation model. then we finetune this model for sentence-level qe. we make our code publicly available. foot_1 ## related work in the past decade, most of the quality estimation systems depended heavily on feature engineering, linguistic information, and machine learning algorithms such as support vector machines or randomized decision trees @xcite . in recent years, emerging neural-based qe have been outperforming earlier approaches on leaderboards of mt quality estimation @xcite . for instance, postech @xcite , a purely neural system based on encoder-decoder recurrent neural network (referred to as a predictor) is stacked with bidirectional rnn (referred to as an estimator). this predictor-estimator qe system was the bestperforming one in wmt 2017. 3 it was further extended in deepquest architecture @xcite . these systems required extensive pretraining, which makes them dependent on large parallel corpora and computationally expensive. to overcome this problem, cross-lingual embeddings @xcite were used to reduce the burden of deep neural network architecture. transquest used these cross-lingual embeddings and was the best-performing sentence-level qe model at wmt 2020 qe shared task @xcite . for the sentence-level qe task, the model was finetuned on multilingual pre-trained representations. for the word-level qe task, the authors used direct assessment (da) quality scores from the mlqe-pe dataset. motivated by this work, we finetune our word-level model to yield sentence-level qe. overall, the tremendous progress in the field of quality estimation is achieved thanks to the annual focus of the shared task organized by wmt and thanks to the annotated data released in these tasks, leading to the development of various opensource systems such as quest @xcite , quest++ @xcite , deepquest @xcite , openkiwi @xcite and tran-squest @xcite . ## task description the task consists of building a quality estimation system that (1) predicts the quality score for an input pair of the source text and mt hypothesis, and (2) provides word-level evidence for its predictions. ## dataset description the dataset for the shared task consists of training, development, and test sets. the training and development sets are estonian-english (et-en) and romanian-english (ro-en) partitions of the mlqe-pe dataset @xcite . the test set consists of sentence-level quality scores and word-level error annotations for these two language pairs. the goal of the shared task is to estimate the word quality in unsupervised settings (no training data for word-level qe whatsoever). however, participating systems can also be labeled as "unconstrained" and use word-level qe training data. additionally, there are zero-shot test sets for two language pairs, i.e., german-chinese (de-zh) and russian-german (ru-de), where no sentence-level or word-level annotations were available at training time. ## methodology in the proposed system, we use a pre-trained xlm-r model @xcite to obtain representations of input sentences in continuous space. xml-r is trained on large-scale multilingual com-moncrawl datasets. we have two separate models for the word-level and sentence-level quality estimation. ## experiments and results in all our experiments, we use xlm-r large model foot_4 using hugging face transformers @xcite . we run all our experiments on nvidia rtx 3090. to find the best parameters, we run a grid search. the optimal hyper-parameters are 2e -5 learning rate, three epochs, and batch size of 16. ## conclusion the paper describes our submission to the shared task, eval4nlp, co-located with emnlp. first, we built a word-level quality estimation model.