Tailoring Domain Adaptation for Machine Translation Quality Estimation (2023)

Abstract: While quality estimation (QE) can play an important role in the translation process, its effectiveness relies on the availability and quality of training data. For QE in particular, high-quality labeled data is often lacking due to the high-cost and effort associated with labeling such data. Aside from the data scarcity challenge, QE models should also be generalizabile, i.e., they should be able to handle data from different domains, both generic and specific. To alleviate these two main issues — data scarcity and domain mismatch — this paper combines domain adaptation and data augmentation within a robust QE system. Our method is to first train a generic QE model and then fine-tune it on a specific domain while retaining generic knowledge. Our results show a significant improvement for all the language pairs investigated, better cross-lingual inference, and a superior performance in zero-shot learning scenarios as compared to state-of-the-art baselines.

## introduction predicting the quality of machine translation (mt) output is crucial in translation workflows. informing translation professionals about the quality of an mt system allows them to quickly assess the overall usefulness of the generated translations and gauge the amount of post-editing that will be required @xcite . quality estimation (qe) is an approach that aims to reduce the human effort required to analyze the quality of an mt system by assessing the quality of its output without the need for reference translations. qe can be applied on word-, sentence-or document-levels. the goal of sentence-level qe, which is the focus of our work, is to predict a quality label based on a source sentences and its mt equivalents. this label, (i.e., the quality estimate), can be expressed in various ways such as ter/hter @xcite , bleu @xcite or any metric of interest to the user. training a sentence-level qe system typically requires aligned data of the form: source sentence (src), target sentence (trg), and quality gold label (lbl). however, most quality labels are by-products of mt and post-editinga rather difficult and expensive process -limiting the size of the available qe data @xcite . the wmt qe shared task @xcite has been offered a platform to compare different qe systems and to share qe data. despite efforts from initiatives like the qe shared task to publicly release qe datasets, such resources remain scarce across language pairs and, by extension, also have a limited coverage across domains @xcite . this can pose a challenge for all qe models, especially recent ones that utilize large pre-trained language models (llms) @xcite , since fine-tuning pre-trained models with small datasets has been demonstrated to be quite unstable @xcite . furthermore, qe models trained on specific data do not generalize well to other domains that are outside of the training domain @xcite . domain mismatches lead to significant decreases in the performance of qe models @xcite . to improve the generalizability of qe models, it is important to establish the right balance between domain-specific and generic training data. to date, only a few attempts have been made to address this challenge @xcite @xcite . thus, the majority of qe models have difficulty with accurately estimating quality across different domains, whether they are generic or specific @xcite . in this work, we propose to tackle both the data scarcity and the domain mismatch challenge that llm-based qe models face. we propose a methodology whereby a small amount of domainspecific data is used to boost the overall qe prediction performance. this approach is inspired by work on domain adaptation (da) in the field of mt, where a large generic model is initially trained and then fine-tuned with domain-specific data @xcite . to assess the validity of the proposed approach in qe, we conducted experiments using small and large, authentic and synthetic data in bilingual, cross-lingual, and zero-shot settings. we experimented with publicly available language pairs from english (en) into german (de), chinese (zh), italian (it), czech (cz), and japanese (ja) and from romanian (ro) and russian (ru) into english (en). we used the common test sets from the wmt2021 qe shared tasks foot_0 . our experiments show a statistically significant improvement in the performance of qe models. our findings also indicate that not only our implementation leads to better multi-/cross-lingual qe models (where multi-/cross-lingual data is provided) but also zero-shot qe (where no data for the evaluated language pairs was provided at training). the main contributions of our research are: to the best of our knowledge, this is the first qe methodology to use da and dag. furthermore, it is easily reusable and adaptable: (i) while we used xlm-r in our experiments, one can easily replace it with any preferred llm as long as the input-output criteria are met; (ii) we built our tool around hugging face (hf) implementations of llms, meaning one can employ a certain generic model and apply it to any qe task by simply fine-tuning it on (newly-collected) qe data. ## domain adaptation for specialized qe in this section, we outline our methodology for training llm-based qe models for a specific domain with limited available in-domain data. this involves: (i) a set of training steps that we found to be particularly effective, and (ii) dag techniques to improve the qe models' specificity. additionally, we provide details on two different training modes we implemented (with or without tags). ## results to assess the performance of our approach we evaluate output from the trained qe models in comparison to the reference quality metric (hter/ter) on the test sets described in data groups 1 and 3. we use pearson's coefficient (ρ ∈ -1 : 1, which we rescale to -100 to 100 for clarity) to correlate our predictions with the test set. we use the bleu score as a metric to evaluate the translation quality of our mt models. ## conclusion and future work this paper addresses two key challenges related to quality estimation (qe) of machine translation (mt): (i) the scarcity of available qe data and (ii) the difficulties in estimating translations across diverse domains. the primary aim of this study is to enhance the performance of qe models by addressing these challenges. to do so, we propose a solution that utilizes domain adaptation (da) techniques adopted from mt. we adapt the "mixed fine-tuning + fine-tuning" approach @xcite and extend it with data augmentation as an alternative to the traditional oversampling technique. we adopt a three-step training methodology: (i) we fine-tune xlm-r, a language model, with a large generic qe dataset, which enables the model to generalize; (ii) we fine-tune the model with a mix of out-of-domain (ood) and indomain (id) data derived from two data augmentation (dag) approaches; and (iii) we fine-tune the model with a small amount of domain-specific data, which leads to a more specific model. we evaluated models' performance with and without domain tags appended to the sentences. our experiments show significant improvements across all language pairs under consideration, indicating that our proposed solution has a beneficial impact in addressing the aforementioned challenges. our study also demonstrates the effectiveness of both proposed dag approaches and shows that using domain tags improves the performance of the models. additionally, we find that our model outperforms the baseline in the context of zeroshot learning and in cross-lingual inference. moving forward, there are several directions for future work based on our findings. first, it would be interesting to investigate the performance of our pipeline on low-resource language pairs, where there is limited id data available. this is particularly relevant given the smaller coverage of qe datasets compared to parallel data in mt. second, we only used one type of ood data in our experiments (en-it); it would be useful to explore other ood data over different language pairs for qe. third, it would be valuable to study the performance of other llms than xlm-r. fourth, since the choice of languages employed in the pipeline was based on availability, we would suggest exploring a more regulated approach for selecting the languages to be used in the proposed pipeline. specifically, the optimal transfer languages can be selected based on their data-specific features, such as dataset size, word overlap, and subword overlap, or dataset-independent factors, such as genetic (see appendix a.5) and syntactic distance @xcite .