QUAK : A Synthetic Quality Estimation Dataset for K orean- E nglish Neural Machine Translation (2022)

Abstract: With the recent advance in neural machine translation demonstrating its importance, research on quality estimation (QE) has been steadily progressing. QE aims to automatically predict the quality of machine translation (MT) output without reference sentences. Despite its high utility in the real world, there remain several limitations concerning manual QE data creation: inevitably incurred non-trivial costs due to the need for translation experts, and issues with data scaling and language expansion. To tackle these limitations, we present QUAK, a Korean-English synthetic QE dataset generated in a fully automatic manner. This consists of three sub-QUAK datasets QUAK-M, QUAK-P, and QUAK-H, produced through three strategies that are relatively free from language constraints. Since each strategy requires no human effort, which facilitates scalability, we scale our data up to 1.58M for QUAK-P, H and 6.58M for QUAK-M. As an experiment, we quantitatively analyze word-level QE results in various ways while performing statistical analysis. Moreover, we show that datasets scaled in an efficient way also contribute to performance improvements by observing meaningful performance gains in QUAK-M, P when adding data up to 1.58M.

## introduction quality estimation (qe) is the task of predicting the translation quality as a continuous value or discrete tags by referring to a source sentence and its machine translation (mt) output @xcite @xcite . since quality annotations on mt output are applied in various ways according to the granularity levels (word, sentence, document, etc.), qe research has been constantly developing in recent years @xcite @xcite . mt output tags ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok bad ok bad ok bad ok bad ok ok ok source 중국 당국이 부인하지 않는 것으로 볼 때 가능성이 높다 . source tags ok ok ok ok ok ok ok bad bad ok alignments 0-3 1-4 2-7 3-5 3-6 4-8 5-8 6-0 7-13 8-11 8-12 9-14 edits (1) insertion (' '→ it) (2) substitution (@xmath0: an example of quak dataset. for the correct translation, one insertion and three substitutions are required for the mt output. although not included in this example, if there is a missing word, a bad tag is attached to the location of the corresponding gap token. we indicate the alignment information (alignments) in the form of {source index}-{aligned mt output index}. owing to this importance, datasets for training qe systems are being released continuously. however, we highlight three limitations for the existing qe dataset. (1) first, non-trivial human labor and time cost are required when constructing data. source sentences, mt output, and quality annotations are dataset prerequisites for qe learning, among which translation experts proficient in a language pair are essential in the labeling process. employing experts is far more difficult especially in low-resource languages. to mitigate the above limitations, we introduce quak 1 , a large-scale korean-english synthetic qe dataset. this is built as an automated process by taking the @xcite approach and aims to train word-level qe. namely, the data generation process does not demand human post-editing, allowing data to be built at scale than manual methods. in addition, language extension is relatively free as it is language-agnostic within a language pair in which google translation is possible and a corresponding corpus exists. therefore, we adopt korean-english, one of the morphologically rich languages rarely addressed in the qe field. for constructing quak, a monolingual or parallel corpus and an mt model are required. quak is divided into three sub-quak datasets according to data sources: (1) quak-monolingual (quak-m) leveraging a monolingual corpus of the target language, (2) quak-parallel (quak-p) leveraging a parallel corpus, and (3) quak-hybrid (quak-h) jointly leveraging monolingual and parallel corpus. the final quak training data size in quak-p and quak-h is 1.58m and 6.58m in quak-m, which is about 225 times and 940 times larger than the 7k size of the wmt official dataset @xcite . considering that quak is synthetic data, we scrutinize the dataset with statistics and a quantitative analysis to provide reliability and quality assurance. in the quantitative analysis, in particular, we first compare the word-level qe model finetuning performance based on multiple multilingual pre-trained language models (mplms) using only 100k pieces from each sub-quak. thereafter, we use the best performing model to incrementally scale the data size and track performance fluctuations. as a result of the experiment, the xlm-roberta (xlm-r) @xcite large model is the most competitive, showing a difference of up to 0.12 mcc compared to other mplms such as multilingual bart (mbart) (liu et al., 1 our quak dataset is publicly available at https:// bit.ly/3dqe2ke . 2020), xlm @xcite . furthermore, scaling the data to 1.58m tends to improve overall model performance. based on the mt output-side, quak-m obtains performance gain of maximum 0.042, quak-p 0.037, and quak-h 0.029 mcc. our contributions are as follows: ## related work as qe research has increasingly been introduced recently, human-labeled qe datasets are also being released @xcite @xcite . however, data construction processes have several limitations in terms of time cost, data size, and available language pairs. many studies have been conducted continuously to handle these limitations. to name a few, @xcite propose a synthetic data construction method that utilizes a parallel corpus to alleviate the cost of human labor and time cost. in such study, translation errors committed through a language model or an nmt system are injected to parallel sentences. a similar method of generating synthetic data through parallel corpus has also been leveraged in automatic post-editing research @xcite . to address data size restrictions caused by time cost in human annotations, attempts have been made using data augmentation @xcite et al., 2020 to tackle constraints about available language pairs, cross-lingual zero-shot qe approaches are constantly studied @xcite . following this trend, wmt21 includes zero-shot to their main interests, evaluating the competence of a qe model on unseen languages @xcite . this study addresses all of the respective factors and presents quak. ## quak quak is a qe dataset for evaluating the quality of korean-english mt output, and includes three sub-quaks according to the data selection. each sub-quak comprises (1) a source sentence, (2) its mt output, and (3) ok/bad quality annotation. source sentence and its mt output are utilized as model input, allowing the model to classify the quality of these sentences into ok/bad tags on a token basis. quality annotations separately exists for source sentence and mt output. when the model predicts the translation quality, these are used as a groundtruth for evaluation. with primary consideration of efficiency and effectiveness, we construct data in a fully-automatic manner, exploited method by @xcite . we select existing monolingual or parallel corpora for our data sources (detailed in section 3.1) and use them to each data production process (detailed in section 3.2). ## conclusion we expose three drawbacks in terms of manual qe data construction: human labor and time cost, resulting in limited amount of data and limited language pairs. taking this into account, we present quak, a synthetic korean-english qe dataset for word-level qe. we automatically generated three sub-quaks with three strategies and quantitatively analyzed the trained qe models using them. first, quak-p is generated based on parallel corpus and induced the best performance among three sub-quaks. along with quak-p, an increase in the data size of quak-m had a positive effect on performance gain. however, in further expansion using wikipedia, the improvement in the low ter range was poor, so the overall performance fell. the qe model trained with quak-h did not show a steady performance gain. this dataset was built in a fully automated manner, eliminating human intervention while increasing reusability and scalability. the quak dataset generation process is language-agnostic if there is an mt model and corresponding corpus (monolingual or parallel).