What Knowledge Is Needed? Towards Explainable Memory for k NN - MT Domain Adaptation (2023)

Abstract: kNN-MT presents a new paradigm for domain adaptation by building an external datastore, which usually saves all target language token occurrences in the parallel corpus. As a result, the constructed datastore is usually large and possibly redundant. In this paper, we investigate the interpretability issue of this approach: what knowledge does the NMT model need? We propose the notion of local correctness (LAC) as a new angle, which describes the potential translation correctness for a single entry and for a given neighborhood. Empirical study shows that our investigation successfully finds the conditions where the NMT model could easily fail and need related knowledge. Experiments on six diverse target domains and two language-pairs show that pruning according to local correctness brings a light and more explainable memory for kNN-MT domain adaptation.

## introduction domain adaptation in neural machine translation (nmt) aims at adapting pre-trained nmt models to a target domain @xcite @xcite @xcite . fine-tuning @xcite has been the de facto standard for adaptation. however, fine-tuning suffers from the catastrophic forgetting problem @xcite . recently, @xcite propose knn-mt, showing a new paradigm for domain adaptation. knn-mt first explicitly extracts translation knowledge in the target domain training data into a key-value datastore with a pre-trained nmt model. for each datastore entry, the key is a continuous representation and the value is a symbolic token. the datastore is then used to assist the nmt model during translation. the knn-mt framework circumvents the necessity to disturb the parameters of the pre-trained nmt model and enables quick adaptation by switching datastores. to understand the relationship between the datastore and the nmt model, this paper conducts investigations on the interpretability issue: what knowledge does the nmt model need? intuitively, the pre-trained nmt model only needs knowledge that remedies its weaknesses. thus, we propose to explore this issue from the point of local correctness (section 3). our local correctness includes two aspects, the correctness of translating a given entry (entry correctness) and, more importantly, the correctness of performing translation in a given neighborhood in the representation space (neighborhood correctness). for the entry correctness, we check whether the nmt could make correct translation for the entry itself and accordingly split the datastore entries into two categories, namely known and unknown. based on entry correctness, we examine neighborhood correctness to more comprehensively evaluate the nmt model's underlying capability. specifically, we propose a knowledge margin metric to evaluate the maximum size of the neighborhood where the nmt could make correct translation. intuitively, the nmt model may fail when the knowledge margin is small. to verify our interpretation, we devise a datastore pruning algorithm plac (pruning with local correctness), which simply removes entries with a higher knowledge margin value (section 4). these entries are less useful for adaptation, because the nmt model translates well in their neighborhood. we conduct experiments on six diverse target domains in two language pairs (section 6). compared with existing pruning baselines @xcite , plac prunes more entries (up to 45%) in four opus domains' datastore without hurting translation performance. through ablation study, we reveal that simply relying on entry correctness is not enough, showing that the novel metric knowledge margin for the neighborhood correctness could be the key to build a light and more explainable memory for knn-mt domain adaptation. ## background for nmt domain adaptation, knn-mt constructs a datastore d based on the given target domain bilingual corpus c and use it to provide helpful target domain translation knowledge for the pretrained nmt model m. in this section, we briefly introduce knn-mt and its advanced variant, adaptive knn-mt @xcite . ## building explainable memory based on local correctness because local correctness are good indicators for translation failures. it could also be interpreted as the importance of datastore entries. to verify this interpretation, we propose a pruning algorithm, i.e. pruning with local correctness (plac), to cut off entries with a high knowledge margin (algorithm 1). there are two steps in the algorithm. in the first step, each entry (h, y) in the datastore d is checked for their local correctness. if knowledge margin of (h, y) is greater than or equal to the threshold k p , the entry is collected as the pruning candidates foot_6 . in the second step, these pruning candidates are randomly selected and get removed from d until the required pruning ratio is reached. since our method does not need to train any additional neural networks, it can be easily implemented. the pruned source sentence (x): wie ist cy@@ an@@ ok@@ it anzu@@ wenden ? previous translation (@xmath0: how to use cy@@ an@@ ok@@ no. type retrieved keys: source (x) retrieved keys: target (@xmath1. you will have cy@@ an@@ ok@@ it 3 unknown wo@@ für wird cy@@ an@@ ok@@ it ange@@ wendet ? what is cy@@ an@@ ok@@ it 4 unknown wel@@ ches risiko ist mit cy@@ an@@ ok@@ it verbundenn ? what is the risk associated with cy@@ an@@ ok@@ it 5 unknown die folgenden neben@@ wirkungen wurden in verbindung mit der anwendung von cy@@ an@@ ok@@ it berichtet . the following un@@ desi@@ rable effects have been reported in association with cy@@ an@@ ok@@ it 6 unknown warum wurde cy@@ an@@ ok@@ it zugelassen ? why has cy@@ an@@ ok@@ it 7 unknown beson@@ dere vorsicht bei der anwendung von cy@@ an@@ ok@@ it ist erforderlich take special care with cy@@ an@@ ok@@ it 8 unknown wie wirkt cy@@ an@@ ok@@ it ? how does cy@@ an@@ ok@@ it nmt's prediction (y ′ t ): ite correct target token (yt): it ## experiment setup this section introduces general experiment setup for evaluating pruning effect. more implementation details can be found in appendix c. ## conclusion it is interesting to explore how a neural model and a symbolic model works together. in this paper, we propose to analyze the local correctness of the neural model's predictions to identify the conditions where the neural model may fail. by introducing a knowledge margin metric to measure the local correctness, we find that the nmt model often fails when the knowledge margin is small. these results provide support for building a more explainable machine translation system. based on analyses, we can safely prune the datastore with the proposed plac method. empirically, the datastore could be successfully pruned up to 45% while retaining translation performance. this results validate our earlier findings about the local correctness and translation failures. our method is general to different knn-mt variants and easy to implement. future directions maybe using local correctness to explore more interpretability issue of nmt domain adaptation, e.g. catastrophic forgetting.