Domain adapted machine translation: What does catastrophic forgetting forget and why? (2024)

Abstract: Neural Machine Translation (NMT) models can be specialized by domain adaptation, often involving fine-tuning on a dataset of interest. This process risks catastrophic forgetting: rapid loss of generic translation quality. Forgetting has been widely observed, with many mitigation methods proposed. However, the causes of forgetting and the relationship between forgetting and adaptation data are underexplored.This paper takes a novel approach to understanding catastrophic forgetting during NMT adaptation by investigating the impact of the data. We provide a first investigation of what is forgotten, and why. We examine the relationship between forgetting and the in-domain data, and show that the amount and type of forgetting is linked to that data’s target vocabulary coverage. Our findings pave the way toward better informed NMT domain adaptation.

## introduction the specialization of neural machine translation (nmt) models for high performance in a specific domain, such as legal or healthcare, is of strong interest to academia @xcite and industry @xcite . fine-tuning, sometimes known as transfer learning, is a wellestablished domain adaptation method that continues training a pre-trained nmt model on some new dataset from the domain of interest @xcite . however, fine-tuning on domainshifted data can result in catastrophic forgetting @xcite . this apparently simple statement has been widely observed in nmt, but rarely examined. catastrophic forgetting in nmt is described variously as 'degradation of general-domain performance' @xcite or '[forgetting] previous domain knowledge' @xcite ), * work completed while at rws typically referencing lower scores in some quality metric. however, prior work on forgetting in nmt focuses on mitigation, leaving two important gaps. firstly, prior work does not determine what is forgotten in concrete terms. lower scores in a reference-based quality metric can indicate either poor translation or simply a vocabulary shift towards the new domain. this is especially true for string-based metrics like bleu @xcite . prior work does not distinguish between quality drop and vocabulary shift, and further does not address whether vocabulary shift 'forgetting' is beneficial or detrimental to translation of the generic and new domains. secondly, prior work almost universally treats forgetting and its mitigation as independent of the adaptation dataset. in fact, the contents of the adaptation dataset will impact forgetting -consider the amount of forgetting expected if fine-tuning on a 1000-sentence sample of the pre-training dataset, versus 1000 copies of the same sentence pair. understanding the relationship between adaptation data and forgetting is crucial for predicting how well domain adaptation will work, whether adapted performance is likely to generalise, or whether forgetting mitigation approaches are necessary. the contributions of this paper lie in addressing these gaps. specifically: ## understanding generic data mix-in in the previous section, we found that adaptation dataset vocabulary coverage has a strong negative correlation with forgetting. a natural question follows: what would be the effect of ensuring all of these adaptation datasets had 100% vocabulary coverage? to answer, we propose and perform minimal mix-in, a targeted variant of mixed finetuning @xcite . we focus on target coverage and the quality degradation metrics bleu and comet, both for brevity and as these had the strongest relationship with vocabulary coverage. mixed fine-tuning aims to mitigate forgetting by mixing examples from the generic training set into the adaptation set. for minimal mix-in, we add generic examples to the adaptation set if they include a target token that is not in the adaptation dataset so far. aside from the novelty of targeted mix-in data, our goal is to examine the effect of improving the vocabulary coverage with minimal other change to the adaptation data and no change at all to the model architecture, adaptation or inference procedure. excepting kor and bsd, minimal mix-in produces an adaptation dataset where fewer than 10% of examples are generic. to benchmark the forgetting mitigation possible with a similar non-minimal data augmentation, we follow a popular mixed fine-tuning recipe found in the literature @xcite which uses a 1:1 ratio of randomly sampled generic segments to in-domain segments. we refer to this as random 1:1. ## conclusions this paper investigates what is forgotten during nmt domain adaptation, and why. we show that vocabulary shift during adaptation is not necessarily beneficial, and that detrimental shift can be orthogonal to quality metrics. we find forgetting correlates with in-domain vocabulary coverage, allowing better prediction of how adaptation will behave on a particular dataset. our findings emphasise that nmt adaptation research should not be dataset agnostic: in-domain data characteristics are critical to how adaptation can succeed or fail.