KU X Upstage’s Submission for the WMT 22 Quality Estimation: Critical Error Detection Shared Task (2022)

Abstract: This paper presents KU X Upstage’s submission to the quality estimation (QE): critical error detection (CED) shared task in WMT22. We leverage the XLM-RoBERTa large model without utilizing any additional parallel data. To the best of our knowledge, we apply prompt-based fine-tuning to the QE task for the first time. To maximize the model’s language understanding capability, we reformulate the CED task to be similar to the masked language model objective, which is a pre-training strategy of the language model. We design intuitive templates and label words, and include auxiliary descriptions such as demonstration or Google Translate results in the input sequence. We further improve the performance through the template ensemble, and as a result of the shared task, our approach achieve the best performance for both English-German and Portuguese-English language pairs in an unconstrained setting.

## introduction this paper presents our submission to the critical error detection (ced) shared task among the quality estimation (qe) tasks of wmt22 @xcite . ced is a task of detecting cases where translation errors in source sentences or translation results distort meaning in terms of race, gender, safety, law, finance, etc. @xcite @xcite . critical translation errors in the shared task appear in the form of mistranslation, hallucination, and deletion in source sentences or translation results, and errors can be classified into five categories: additions, deletions, named entities, meaning, and numbers. even if machine translation (mt) systems produce fluent translations, the fact that they cannot be free from fatal semantic errors emphasizes the importance of preventing social repercussions from the errors. forbidding socially bad influences and losses from these meaning deviations is the purpose of the ced task @xcite . participating systems distinguish only critical errors, not correct translations or simple translation errors. in contrast to last year, submissions should be provided with continuous scores rather than binary labels. the official script calculates scores with automatically assigned classes based on the threshold value of the index corresponding to the number of errors. similar to last year, we participated in unconstrained english-german (en-de) and portuguese-english (pt-en) utilizing released training datasets foot_0 . to perform the ced task, we exploit the xlm-roberta large model @xcite as utilized in the baseline without additional parallel data. in addition, we adopt prompt-based fine-tuning to mitigate catastrophic forgetting during fine-tuning by maximizing the linguistic capability obtained through pre-training. in prompt-based fine-tuning, the downstream task is reformulated into a clozestyle, which is consistent with the masked language modeling objective. the word for the masked part is predicted by the model based on the task-specific template @xcite . recent studies have demonstrated the remarkable effects of promptbased learning in the natural language processing field @xcite @xcite @xcite , and we apply this new paradigm to the qe task. we manually generate templates each containing a source sentence, its translation result, and a description with a mask token for the ced task. furthermore, we generate label words @xcite to map the words to be filled in the masked part and labels. exploring appropriate templates in promptbased fine-tuning is important because the performance ranges widely depending on the template used. therefore, we design multiple hard prompts through prompt engineering, and these are configured into three types of templates according to additional information: plain template, template with demonstration, and template with google translate. through answer engineering, we map contrastive words for each ok and bad tag in diverse combinations. to obtain the final score, we extract probability for words mapped to bad. we further improve performance by ensembling values from templates. our approach outperforms the baseline models in en-de and pt-en by a substantial margin and achieves first place. experimental results demonstrate that simply setting up the training method without modifying the model or augmenting the data with additional parallel corpora significantly affects the performance. ## conclusion we applied prompt-based learning to the ced task by forming a learning objective for the task similar to that in pre-training. this method outperformed the baseline performance while preserving the model parameters and data settings. we performed manual prompt engineering and answer engineering to explore intuitive hard prompts. in addition, because finding optimal prompts is difficult, we ensembled predictions from diverse templates to address the performance variation and achieve additional performance boost. our method is simple but powerful, and we hope that this method will be actively introduced to qe tasks in future studies.