D a LC : Domain Adaptation Learning Curve Prediction for Neural Machine Translation (2022)

Abstract: Domain Adaptation (DA) of Neural Machine Translation (NMT) model often relies on a pre-trained general NMT model which is adapted to the new domain on a sample of in-domain parallel data. Without parallel data, there is no way to estimate the potential benefit of DA, nor the amount of parallel samples it would require. It is however a desirable functionality that could help MT practitioners to make an informed decision before investing resources in dataset creation. We propose a Domain adaptation Learning Curve prediction (DaLC) model that predicts prospective DA performance based on in-domain monolingual samples in the source language. Our model relies on the NMT encoder representations combined with various instance and corpus-level features. We demonstrate that instance-level is better able to distinguish between different domains compared to corpus-level frameworks proposed in previous studies Finally, we perform in-depth analyses of the results highlighting the limitations of our approach, and provide directions for future research.

## introduction the classical domain adaptation scenario @xcite usually relies on an existing nmt model trained on large datasets originated from various sources. this general model is adapted to the new domain, with a small sample of in-domain parallel data, through finetuning or other da methods. without any parallel in-domain data, we cannot estimate the quality of general nmt model on the domain of interest, nor can we anticipate what will be the benefits of the da and how much parallel in-domain data is required. in this work, we address the problem that translation service providers may regularly face when * corresponding author receiving a request for a new domain translation. in such case, a new domain is often defined by its source language samples, and the translation provider needs to invest into in-domain parallel dataset creation in order to be able to perform evaluation and domain adaptation of its general model. current state of the art research in nmt domain adaptation rarely provides any insights on the amount of data required to perform domain adaptation depending on input domain characteristics. this is however a desirable feature that would allow to (i) estimate the data creation cost (time and money-wise) for the client requesting an adaptation to completely new domain; (ii) make an informed decision on how to distribute fixed data creation budget when there is a need to handle multiple da simultaneously (as we demonstrate further sharing this budget equally across all domains may not be optimal). the goal of this work is to gain better insights on domain adaptation dynamics, and provide practical guidelines for such a real-life scenario. several studies address the problem of learning curve estimation models @xcite @xcite of mt or nlp models without actually training those. this is done by training a prediction model which takes corpuslevel representation x as an input and predicts the score y for this corpus as an output. in the case of mt learning curves prediction, x would correspond to the parallel data sample (used to train mt model), and y is the bleu score achieved by an mt model trained on x. this means that each training point creation requires training a new mt model, which may become very costly if we want to create a training set of reasonable size. in this work, we propose a novel framework to perform the learning of the prediction model at instance-level. ## related work the problem of learning curve prediction can be related to a number of different existing problems in natural language processing (nlp) and machine learning. in this section, we briefly review few works that have tried to predict learning curves (also known as scaling laws) or domain shift for different nlp tasks. we also overview some works on active learning for nmt since those features can also be relevant to our task. ## analysis of real domain adaptation learning curves in this section, we analyse several real learning curves for nmt domain adaptation to show how learning curves do behave across domains and to motivate our work. in order to perform this analysis, we first present the nmt baseline model (i.e., general model) we utilize, as well as datasets used for domain adaptation. we also discuss the nmt evaluation metrics that we can rely on when training the learning curve predictor model 1 . nmt model. we consider two different nmt systems: english-german and german-english systems trained on wmt20 dataset @xcite . we provide technical details about architecture and datasets used to train those nmt systems in the appendix a. ## approach in this section, we first formalize the da learning curve predictor. then, we describe the representations and our model for the problem. ## analysis and discussion the results reported in previous section suggest that overall dalc prediction error depends a lot on the nature of the test domain (as seen on figures 3 , 7 and in this section we try to analyse this phenomenon and explain such behaviour. ## conclusion in this work, we formulate a problem of domain adaptation learning curves prediction for nmt as instance-level learning framework. we demonstrate that it is possible to learn reasonable learning curve prediction model with a very small amount of nmt model instances via instance-level learning rather than corpus-level learning that most previous works rely on. we propose a dalc model relying on nmt encoder representations, combined with various instance and corpus-level features. we show that such model is able to achieve good results with small amount of pretrained model instances. we perform in-depth analysis of the results for the domains where the predictor was less successful and conclude that the capacity of the predictor relying on the source-side sample only can be limited for some domains. further analysis of characteristics of such domains could lead to better domain adaptation strategies. finally, we believe it will be interesting to deepen the connections between active learning framework and learning curves prediction frameworks which could mutually help one another.