Translation Transformers Rediscover Inherent Data Domains (2021)

Abstract: Many works proposed methods to improve the performance of Neural Machine Translation (NMT) models in a domain/multi-domain adaptation scenario. However, an understanding of how NMT baselines represent text domain information internally is still lacking. Here we analyze the sentence representations learned by NMT Transformers and show that these explicitly include the information on text domains, even after only seeing the input sentences without domains labels. Furthermore, we show that this internal information is enough to cluster sentences by their underlying domains without supervision. We show that NMT models produce clusters better aligned to the actual domains compared to pre-trained language models (LMs). Notably, when computed on document-level, NMT cluster-to-domain correspondence nears 100%. We use these findings together with an approach to NMT domain adaptation using automatically extracted domains. Whereas previous work relied on external LMs for text clustering, we propose re-using the NMT model as a source of unsupervised clusters. We perform an extensive experimental study comparing two approaches across two data scenarios, three language pairs, and both sentence-level and document-level clustering, showing equal or significantly superior performance compared to LMs.

## introduction neural machine translation (nmt, @xcite heavily depends on training data and the text domains covered in it. full-scale nmt transfomer models @xcite are usually trained on a mix of corpora from several domains @xcite . however, the field lacks an understanding of how these nmt models represent the training data domains in their inner vector spaces. ## analysis in this section, we perform an analysis of inherent domain representations in translation transformers. we reveal how well the domain-specific information in text representations is preserved in nmt models. we focus on "out-of-the-box" nmt systems without any changes and explore the extent to which we can use their internal representations to match the original text domains using principal component analysis (pca) and k-means clustering. we also measure the effect of using broader document-level representations. additionally, we compare nmt representations to the ones extracted from a pre-trained language model, for which aharoni and goldberg (2020) revealed a high degree of domain-specific information. ## practical application our analysis in section 3 revealed that nmt models represent domains in their embedding space separately, similarly to what pre-trained language models do @xcite . we demonstrated that simple clustering on nmt representations allows recovering original data domains to a large degree. this section proposes to utilize this finding to improve an existing framework of automatic domain generation for nmt. in this framework, related work first clusters the training data using representations from an external encoder, and then the base-line nmt model is adapted (fine-tuned) on each cluster separately. we propose to re-use the nmt baseline itself as the encoder in this framework. representations extracted from translation transformers are specific to the task of translation. we hypothesise that it might result in clusters most suitable for downstream translation tasks like finetuning to specific domains/clusters. moreover, an advantage of our scenario is that we cluster the same data (with our nmt model) that we use for nmt model training. it is a frequent multi-domain nmt setup, where multiple target domains are available in training. in the pre-trained language model setup, the data will be more outof-domain, despite the model's generality. ## experiments in this section we perform an extensive experimental study comparing performance of the existing automatic domains framework (section 4.1) with our proposed version (section 4.2). we experiment with both sentence-level and document-level representations as a basis for k-means algorithm on three language pairs and two data scenarios. we first train baseline transformer nmt models on concatenated data from all domains (same baseline as in section 3) and then cluster the training, development, and test data using either this same baseline or xlm-r. next, we fine-tune foot_1 our baseline models to the different obtained data partitions (clusters) and compare the translation quality of resulting fine-tuned (adapted) models. ## conclusion in this work, we made a two-fold contribution. the first is to the field of nmt interpretation and analysis. we have shown that a baseline transformer nmt encoder preserves enough domain-specific information to distinguish between oracle domains in a mixed corpus without supervision. we showed an evolution of this property across the transformer layer using pca and k-means clustering on the level of sentences and documents. comparison to xlm-r based clusters demonstrated that both sentence-level and document-level nmt clusters show higher cluster purity (similarity to original text domains). next, we utilized our analysis insights to improve an existing practical cluster-based multidomain nmt approach @xcite . in a setting with preset domains (i.e., available corpus/domain labels), tuning to nmt clusters on average matches or surpasses xlm-r clusters. additionally, nmt cluster-based tuning mostly matches the translation quality when tuning to original corpus labels, with some exceptions that we also analyze and explain. finally, in the case of a heterogeneous corpus (paracrawl), the performance of fine-tuned nmt models depends on the number of clusters, language pairs, and other parameters. we see significant improvement for en-et and en-cs translation when comparing xlm-r and nmt-based clusters (on both sentence and document levels). for de-en, the domain tuning results depend on the nmt model's capacity for learning each language pair's translation. a.2 language model xlm-r base our model of choice from the family of bert-like models is the base version of the xlm-r @xcite . it is a single multilingual model covering about 100 languages, which is very useful when dealing with machine translation systems, where for different language pairs we may not have a separate monolingual bert for each source language. we choose xlm-r as opposed to the multilingual bert @xcite since it is a more recent and better performing @xcite model. we choose the base version because it is most compatible to our nmt baseline in terms of capacity.