Categorizing Semantic Representations for Neural Machine Translation (2022)

Abstract: Modern neural machine translation (NMT) models have achieved competitive performance in standard benchmarks. However, they have recently been shown to suffer limitation in compositional generalization, failing to effectively learn the translation of atoms (e.g., words) and their semantic composition (e.g., modification) from seen compounds (e.g., phrases), and thus suffering from significantly weakened translation performance on unseen compounds during inference. We address this issue by introducing categorization to the source contextualized representations. The main idea is to enhance generalization by reducing sparsity and overfitting, which is achieved by finding prototypes of token representations over the training set and integrating their embeddings into the source encoding. Experiments on a dedicated MT dataset (i.e., CoGnition) show that our method reduces compositional generalization error rates by 24% error reduction. In addition, our conceptually simple method gives consistently better results than the Transformer baseline on a range of general MT datasets.

## introduction neural machine translation (nmt) has achieved competitive performance on benchmark datasets such as wmt @xcite @xcite . however, the generalizaiton to low-resource domains @xcite @xcite and robustness to slight input perturbations @xcite are relatively low for nmt models. in addition, recent studies show that nmt systems are vulnerable to compositional generalization (lake and @xcite @xcite @xcite , namely the ability to understand and produce a potentially infinite (formally exponential to the input size) number of novel combinations of known atoms @xcite @xcite @xcite . take cognition @xcite , a dedicated mt dataset, for example (figure 1 ). despite that certain instances of translation atoms (e.g., small, large, car, and chair) and their semantic compositions (e.g., small chair and large car) are frequent in training data, unseen compositions of the same atoms (e.g., large chair) during testing can suffer from large translation error rates. compositionality is also a fundamental issue in language understanding and motivated for translation @xcite , which has been suggested as being essential for robust translation @xcite and efficient low-resource learning @xcite . the current dominant method to nmt employs a sequence-to-sequence architecture @xcite , where an encoder is used to find representations of each input token that thoroughly integrates its sequence-level context in-formation, and a decoder refers to such contextualized representations for generating a translation sequence. a key reason of failure on compositional generalization is that the correspondence between pairs of token sequences is modeled as a whole. specifically, nmt models are trained end-to-end over large parallel data without disentangling the representation of individual words or phrases from that of whole token sequences. at the sequence level, the source input sample space is highly sparse mainly due to semantic composition, and small changes to a sentence can lead to out-ofdistribution issues @xcite @xcite . intuitively, one way to solve this problem is to decouple token-level information from the source sequence by injecting token-level translation distribution (e.g., p (petit|small)) into the source representation. given the fact that the source-side contextualized representations encode rich tokenlevel translation information @xcite , we categorize sparse token-level contextualized source representations into a few representative prototypes over training instances, and make use of them to enrich source encoding. in this way, when encoding a sequence, the model observes less sparse prototypes of each token, thus alleviating excessively memorizing the sequencelevel information. we propose a two-stage framework to train prototype-based transformer models (proto-transformer). in the first stage, we warm up an initial transformer model which can generate reasonable representations. in the second stage, for each token, we run the trained model to extract all contextualized representations over the training corpus. then, we perform clustering (e.g., k-means) to obtain the prototype representations for each token. take figure 2 as an example, for the token "toy", we collect all the contextualized representations and cluster them into 3 prototypes. finally, we extend the base model by fusing the prototype information back into the encoding process through a prototype-attention module, and continue to train the whole model until convergence. experimental results on cognition show that our method significantly improves novel composition translation by over 24% error reduction, demonstrating the effectiveness for tackling the compositional generalization problem. to further verify the effectiveness on more datasets, we conduct experi-ments on 10 commonly used mt benchmarks and our method gives consistent bleu improvement. we also present empirical analysis for prototypes and quantitative analysis on compositional generalizaiton. the comparison between the one-pass and the two-pass training procedure shows that the one-pass method is both faster and more accurate than the two-pass one, demonstrating that more generalizable prototypes extracted from early training phrase are more beneficial to compositional generalization. additionally, quantitative analysis demonstrates that our proposed model is better at handling longer compounds and more difficult composition patterns. the code is publicly available at https://github.com/aries-lm/catmt4cg.git . ## related work compositional generalization recent work @xcite has demonstrated weak compositionality of neural models using dedicated datasets. various methods haven been proposed to solve the issue of compositional generalization such as encoding more inductive bias @xcite @xcite @xcite , meta-learning (lake, 2019; @xcite , and data augmentation @xcite . recently, @xcite and @xcite show that the transformer architecture can perform better on compositional generalization with some modifications. although these methods have demonstrated better generalization or interpretability, most of them are limited small vocabulary and limited samples semantic parsing datasets. in the context of machine translation, lake and baroni (2018) construct a small dataset where the training data contains a word daxy along with its parallel sentences of a single pattern (e.g., i am daxy, je suis daxist) while the test set contains novel patterns (e.g., he is daxy). however, the experiment is limited in that the test set only consists of 8 samples. different from existing work, @xcite propose a large dataset (cognition) and construct a large-scale test set that contains newly constructed constituents as novel compounds, so that generalization ability can be evaluated directly based on compound translation error rate. we proposed a method enhancing compositional generalization on the dedicated dataset of @xcite , while at the same time gives improvements to the machine she chose a cute style. taylor chose to get flowers. taylor chose the color she wanted. ## conclusion we investigated a conceptually simple method for enhancing compositional generalizaiton of nmt models, proposing a two-stage training framework to fuse prototype representations into the encoding process of transformer. experiments on cognition show the effectiveness of our method on compositional generalization, and extensive results over 9 translation tasks verify the generality of our method. to our knowledge, we are the first to propose tokenlevel categorization for nmt, achieving promising performance on both a large-scale compositional generalization dataset and general datasets.