Using Confidential Data for Domain Adaptation of Neural Machine Translation (2021)

Abstract: We study the problem of domain adaptation in Neural Machine Translation (NMT) when domain-specific data cannot be shared due to confidentiality or copyright issues. As a first step, we propose to fragment data into phrase pairs and use a random sample to fine-tune a generic NMT model instead of the full sentences. Despite the loss of long segments for the sake of confidentiality protection, we find that NMT quality can considerably benefit from this adaptation, and that further gains can be obtained with a simple tagging technique.

## introduction the availability of in-domain data remains essential to ensure the quality of neural machine translation (nmt), especially in technical domains @xcite . however, obtaining such data is often challenging, and in many real-world scenarios this is further aggravated by data confidentiality or copyright concerns. in fact, when data content is sensitive, the owner may simply deny providing its translation memories to the translation company it is hiring @xcite . this can lead to considerably worse mt quality, higher post-editing efforts, and subsequently higher translation costs for the data owners themselves. when the complete data cannot be shared in its original form, releasing fragmented data can be considered as a compromise. the most wellknown example of releasing fragmented data is google n-gram @xcite . n-gram tables consisting of sequences of n words and their counts in a given corpus were routinely used to train count-based language models @xcite before the advent of neural methods. however, n-grams are not optimal for training state-of-the-art nlp models such as sequence-to-sequence lstm @xcite or transformers @xcite . in fact, one of the main strengths of these models is the ability of handling arbitrarily long contexts, which would be hindered by the use of fragmented data. in this paper, we take a pragmatic approach and ask: if the data owner can only release fragmented data due to confidentiality issues, can this still benefit downstream nmt quality in any way? motivated by the brittleness of nmt in out-ofdomain settings @xcite and the increasing availability of large pre-trained models @xcite , we focus on the task of adapting a strong-performing general-domain nmt system to various technical domains. we show that fine-tuning on phrase pairs can be a viable solution to exploit confidential data, but the scale of improvements varies strongly across target domains. ## background to our knowledge, the use of confidential data in mt has not received much attention recently. cancedda (2012) proposed an encryption-based (onetime pad) method for phrase-based statistical machine translation (pb-smt). however, pb-smt is nowadays clearly outperformed by nmt @xcite , which function completely differently and therefore require new solutions to preserve data confidentiality. in the broader context of nlp, secure multiparty computation @xcite and homomorphic encryption @xcite have been used to provide strong privacy guarantees. since these cryptographic methods incur high performance penalties (see @xcite for an overview of their performance in deep learning), more recent proposals have focused on the careful use of simpler cryptographic primitives while training a model over encrypted text due to confidentiality reasons. for instance, texthide @xcite allows to perform natural language understanding tasks while requiring the participants to complete an encryption step in a federated setting. the aforementioned studies mostly focus on preventing explicit/implicit leakage of partial information while training the models. by contrast, we explore the possibility of using fragmented data to improve state-of-the-art nmt applications. scenario as illustrated in figure 1 , we consider a common case where a translation company (tc) provides professional services based on a pipeline of nmt and human post-editing. tc wants to improve the quality of its nmt models by training or adapting them on the clients' previously translated data. due to confidentiality concerns, the clients only provide their data in a fragmented form as a compromise. if this kind of data can be used to improve the nmt model, both the clients and the company will benefit by abating human postediting costs. thus, we want to study the possibility of sharing fragmented data for improving utility while preserving the confidentiality of data. ## approach releasing fragmented data in the form of n-grams has a long tradition in nlp @xcite . however, fixed-size n-gram extraction is not directly applicable to parallel data because it breaks translation equivalence with the target side. as a solution, we propose to use phrase pairs @xcite as a text fragmentation method. ## experimental setup we evaluate our approach on german-english in the domains of medicine descriptions, software manuals, and eu legislation. to simulate a realistic production setup, we start from a strong nmt system pre-trained on large amounts (28m sentences) of publicly available data. baseline nmt we use the transformer-based system @xcite pre-trained by facebook for the wmt'19 news translation task @xcite foot_0 and released as part of the fairseq toolkit @xcite . this model was ranked first in the wmt'19 news competition @xcite with a bleu score of 40.8. ## results we evaluate the quality of nmt models by bleu @xcite computed with sacre-bleu @xcite . the phrase-adapted models are compared to the non-adapted baseline @xcite , and to fine-tuning on the original (non fragmented) dataset in order to determine the maximum possible gains. results are reported in our main finding is that phrase pairs can indeed be used to fine-tune a nmt model without any changes to the architecture or the need of specific fine-tuning algorithms. the bleu gains over the non-adapted baseline vary between +7.0 on emea and +1.0 on jrc. this is relevant for our scenario because even translation companies without significant in-house nmt expertise could easily apply our solution to their workflow. our approach is also applicable in cases where tc uses nmt as an outsourced (cloud-based) service, by sending the provider phrase pairs instead of full sentences for model adaptation. ## conclusions we have studied the problem of domain adaptation of nmt models when domain-specific data cannot be shared due to confidentiality or copyright concerns. inspired by a common nlp practice of sharing confidential data in the form of n-grams @xcite , we propose to use phrase extraction @xcite , shuffling and subsampling as a data fragmentation technique for translation data. our experiments on three different domains show that this type of data can be used to fine-tune nmt models leading to considerable improvements on top of a strong baseline and further gains when a simple phrase tagging technique is used. we also find that the magnitude of these gains varies largely across domains, which we tentatively attribute to the different length profiles of our datasets (e.g. legal domain has much longer sentences than the other domains). while our results show that text fragmentation is indeed compatible with modern machine translation systems adaptation, more work needs to be done before our method can be applied on actual sensitive data. to this end, we plan to determine metrics for the quantification of confidentiality protection (or violation) when an adversary tries to reconstruct the original documents. our starting point for this direction would be @xcite , who presented a technique for this purpose only in the context of (monolingual) n-grams.