Real-World Compositional Generalization with Disentangled Sequence-to-Sequence Learning (2023)

Abstract: Compositional generalization is a basic mechanism in human language learning, which current neural networks struggle with. A recently proposed Disentangled sequence-to-sequence model (Dangle) shows promising generalization capability by learning specialized encodings for each decoding step. We introduce two key modifications to this model which encourage more disentangled representations and improve its compute and memory efficiency, allowing us to tackle compositional generalization in a more realistic setting. Specifically, instead of adaptively re-encoding source keys and values at each time step, we disentangle their representations and only re-encode keys periodically, at some interval. Our new architecture leads to better generalization performance across existing tasks and datasets, and a new machine translation benchmark which we create by detecting naturally occurring compositional patterns in relation to a training set. We show this methodology better emulates real-world requirements than artificial challenges.

## introduction the transformer architecture @xcite and variants thereof have become ubiquitous in natural language processing. despite widespread adoption, there is mounting evidence that transformers as sequence transduction models struggle with compositional generalization @xcite @xcite . it is basically the ability to produce and understand a potentially infinite number of novel linguistic expressions by systematically combining known atomic components @xcite . attempts to overcome this limitation have explored various ways to explicitly inject compositional bias through data augmentation @xcite @xcite or new training objectives @xcite @xcite . the majority of existing approaches have been designed with semantic parsing in mind, and as a result adopt domain-and task-specific grammars or rules which do not extend to other tasks (e.g., machine translation). in this work we aim to improve generalization via general architectural modifications which are applicable to a wide range of tasks. our starting point are @xcite who unveil that one of the reasons hindering compositional generalization in transformers relates to their representations being entangled. they introduce dangle, a sequence-to-sequence model, which learns more disentangled representations by adaptively re-encoding (at each time step) the source input. for each decoding step, dangle learns specialized source encodings by conditioning on the newly decoded target which leads to better compositional generalization compared to vanilla transformers where source encodings are shared throughout decoding. although promising, their results are based on synthetic datasets, leaving open the question of whether dangle is effective in real-world settings involving both complex natural language and compositional generalization. we present two key modifications to dangle which encourage learning more disentangled representations more efficiently. the need to perform re-encoding at each time step substantially affects dangle's training time and memory footprint. it becomes prohibitively expensive on datasets with long target sequences, e.g., programs with 400+ tokens in datasets like smcalflow @xcite . to alleviate this problem, instead of adaptively re-encoding at each time step, we only reencode periodically, at some interval. our decoder is no different from a vanilla transformer decoder except that it just re-encodes once in a while in order to update its history information. our second modification concerns disentangling the representations of source keys and values, based on which the encoder in dangle (and also in transformers) passes source information to the decoder. instead of computing keys and values using shared source encodings, we disassociate their representations: we encode source values once and re-encode keys periodically. we evaluate the proposed model on existing benchmarks @xcite and a new dataset which we create to better emulate a real-world setting. we develop a new methodology for detecting examples representative of compositional generalization in naturally occurring text. given a training and test set: (a) we discard examples from the test set that contain out-of-vocabulary (oov) or rare words (in relation to training) to exclude novel atoms which are out of scope for compositional generalization; (b) we then measure how compositional a certain test example is with respect to the training corpus; we introduce a metric which allows us to identify a candidate pool of highly compositional examples; (c) using uncertainty estimation, we further select examples from the pool that are both compositional in terms of surface form and challenging in terms of generalization difficulty. following these three steps, we create a machine translation benchmark using the iwslt 2014 german-english dataset as our training corpus and the wmt 2014 german-english shared task as our test corpus. experimental results demonstrate that our new architecture achieves better generalization performance across tasks and datasets and is adept at handling real-world challenges. machine translation experiments on a diverse corpus of 1.3m wmt examples show it is particularly effective for long-tail compositional patterns. ## background: the dangle model we first describe dangle, the disentangled transformer model introduced in @xcite focusing on their encoder-decoder architecture which they show delivers better performance on complex tasks like machine translation. let @xmath0, x 2 , ..., x n ] denote a source sequence; let f encoder and f decoder denote a transformer encoder and decoder, respectively. x is first encoded into a sequence of n contextualized representations: this step not only generates a new token y t+1 , but also updates the internal target memory m t by concatenating m t-1 with the newly calculated hidden states corresponding to y t . dangle differs from vanilla transformers in that it concatenates the source input with the previously decoded target to construct target-dependent input for adaptive decoding: the adaptive encoder consists of two components. c t is first fed to k 1 transformer encoder layers to fuse the target information: finally, the adaptive source encodings h t together with the target context [y 1 , y 2 , ..., y t ] are fed to a transformer decoder to predict y t+1 : in a departure from vanilla transformers, dangle does not reuse the target memory from previous steps, but instead re-computes all target-side hidden states based on new source encodings h t . similarly to transformers, dangle accesses source information at each decoding step via encoder-decoder attention layers where the same encodings h t are used to compute both keys k t and values v t : ## the r-dangle model in this section, we describe the proposed model, which we call r-dangle as a shorthand for realworld disentangled transformer. ## a real-world compositional generalization challenge models of compositional generalization are as good as the benchmarks they are evaluated on. a few existing benchmarks are made of artificially synthesized examples using a grammar or rules to systematically control for different types of generalization (lake and baroni, 2018a; @xcite @xcite . unfortunately, synthetic datasets lack the complexity of real natural language and may lead to simplistic modeling solutions that do not generalize to real world settings @xcite . other benchmarks @xcite focus on naturally occurring examples but create train-test splits based on the properties of their formal meaning representations (e.g., logical forms). however, formal annotations of meaning are not readily available for tasks other than semantic parsing. since compositional generalization is a general problem, it is desirable to define it on the basis of natural language alone rather than by means of semantic parsing and the availability of formal annotations. it is fair to assume that a sota model deployed in the wild, e.g., a transformer-based machine translation system, will be constantly presented with new test examples. many of them could be similar to seen training instances or compositionally different but in a way that does not pose serious generalization challenges. an ideal benchmark for evaluating compositional generalization should therefore consist of phenomena that are of practical interest while challenging for sota models. to this end, we create react, a new realworld dataset for compositional generalization in machine translation. our key idea is to obtain a generalization test set by detecting compositional patterns in relation to an existing training set from a large and diverse pool of candidates. specifically, we use the iwslt 2014 @xmath1. this procedure identifies naturally occurring compositional patterns which we hope better represent practical generalization requirements than artificially constructed challenges. in the following, we describe how we identify examples that demand compositional generalization. while we create our new benchmark with machine translation in mind, our methodology is general and applicable to other settings such as semantic parsing. for instance, we could take a relatively small set of annotated user queries as our training set and create a generalization challenge from a large pool of unlabeled user queries. filtering out-of-vocabulary atoms compositional generalization involves generalizing to new compositions of known atoms. the wmt corpus includes many new semantic and syntactic atoms that are not attested in iwslt. a large number of these are out-of-vocabulary (oov) words which are by definition unknown and out of scope for compositional generalization. we thus discard wmt examples with words occurring less than 3 times in the iwslt training set which gives us approximately a pool of 1.3m examples. for simplicity, we do not consider any other types of new atoms such as unseen word senses or syntactic patterns. measuring compositionality how to define the notion of compositional generalization is a central question in creating a benchmark. previous definitions have mostly centered around linguistic notions such as constituent or context-free gram-mars @xcite @xcite . these notions are appropriate for synthetic examples or logical forms as their underlying hierarchical structures are well-defined and can be obtained with ease. since we do not wish to synthesize artificial examples but rather detect them in real-world utterances, relying on the notion of constituent might be problematic. sentences in the wild are often noisy and ungrammatical and it is far from trivial to analyze their syntactic structure so as to reliably identify new compositions of known constituents. we overcome this problem by devising a metric based on n-gram matching which assesses how compositional a certain example is with respect to a training corpus. specifically, we first create a lookup dictionary of atomic units by extracting all n-grams that occur more than 3 times in the training corpus. given a candidate sentence, we search the dictionary for the minimum number of n-grams that can be composed to form the sentence. for example, for sentence "x 1 x 2 x 3 x 4 x 5 " and dictionary (x 1 , x 2 , x 3 x 4 , x 5 , x 1 x 2 , x 3 x 4 x 5 , ), the minimum set of such n-grams is (x 1 x 2 , x 3 x 4 x 5 ). a sentence's compositional degree with respect to the training corpus is defined as the ratio of the minimum number of n-grams to its length (e.g., 2/5 = 0.4 for the above example). we select the top 60,000 non-overlapping examples with the highest compositional degree as our candidate pool. as we discuss in section 6, compositional degree further allows us to examine at a finer level of granularity how model performance changes as test examples become increasingly compositional. estimating uncertainty examples with the same compositional degree could pose more or less difficulty to neural sequence models (see last two utterances in we follow the uncertainty estimation framework introduced in @xcite for sequence prediction tasks. specifically, we train 10 transformer models with different random initializations on iwslt (our training corpus), and run inference over the candidate pool created in the previous stage; for each example in this pool, we measure the disagreement between ensemble models using reverse mutual information, a novel measure @xcite which quantifies knowledge uncertainty, i.e., a model's uncertainty in its prediction due to lack of understanding of the data rather than any intrinsic uncertainty associated with the task (e.g., a word could have multiple correct translations). we use the token-level approximation of knowledge uncertainty. we empirically find that the most uncertain examples are extremely noisy and barely legible (e.g., they include abbreviations, typos, and nonstandard spelling). we therefore throw away the top 2,000 uncertain examples and randomly sample 3,000 instances from the next 18,000 most uncertain examples in an attempt to create a generalization test set with diverse language patterns and different levels of uncertainty. analysis we analyze the compositional nature of react by comparing it to several popular benchmarks. specifically, for all datasets, we count the number of novel test set n-grams that have not been seen in the training. we extract n-grams over words and parts of speech (pos); word-based n-grams represent more superficial lexical composition while n-grams based on pos tags reflect more of syntactic composition. as shown in train test • and i can 't believe you 're here and that i 'm meeting you here at ted . ( prp rb in nn . ) • you see , this is what india is today . the ground reality is based on ( dt nn nn vbz vbn in ) a cyclical world view . ## experimental setup datasets we evaluated r-dangle on two machine translation datasets and one semantic parsing benchmark which we selected to maximally reflect natural language variations and real-world generalization challenges. these include: (a) react, the machine translation benchmark developed in this paper; we used the iwslt 2014 @xmath2; (b) cognition @xcite is a semi-natural machine translation benchmark focusing on english-chinese sentence pairs; source sentences were taken from the story cloze test and rocstories corpora @xcite , and target sentences were constructed by post-editing the output of a machine translation engine; (c) smcalflow-cs @xcite is a semantic parsing dataset for task-oriented dialogue, featuring real-world human-generated utterances about calendar management; following previous work @xcite , we report experiments on the compositional skills split, considering a few-shot learning scenario (with 6, 16, and 32 training examples). see appendix a for more details on these datasets. models on machine translation, our experiments evaluated two variants of r-dangle depending on whether keys and values are shared (r-dangle shr ) or separate (r-dangle sep ). we implemented all machine translation models with fairseq @xcite . we compared r-dangle against a vanilla transformer @xcite and the original dangle model @xcite which used the popular fairseq configuration transformer_iwslt_de_en. we also implemented bigger variants of these models using 12 encoder layers and 12 decoder layers which empirically led to better performance. r-dangle shr and r-dangle sep also use a 12-layer decoder. we tuned the number of layers of the adaptive components (k 1 = 2 and k 2 = 10) on the development set. for r-dangle sep , we adopted a 10-layer value encoder and a 10-layer key encoder (k 1 = 2 and k 2 = 8), with the top 8 layers in the two encoders being shared. this configuration produced 12 differently parametrized transformer encoder layers, maintaining identical model size to comparison systems. previous work @xcite has shown the advantage of pre-trained models on the smcalflow-cs dataset. for our semantic parsing experiments, we therefore built r-dangle on top of bart-large @xcite . we only report results with r-dangle shr as the r-dangle sep architecture is not compatible with bart. we again set k 1 = 2 and k 2 = 10. we provide more detail on model configurations in appendix b. variants on react and cognition, across different re-encoding intervals. r-dangle sep is consistently better than r-dangle shr which confirms that representing keys and values separately is beneficial. we also observe that smaller intervals lead to better performance (we discuss this further later). ## related work the realization that neural sequence-to-sequence models struggle with compositional generalization has led to numerous research efforts aiming to precisely define this problem and explore possible solutions to it. a line of research focuses on benchmarks which capture different aspects of compositional generalization. @xcite repurpose existing semantic parsing benchmarks for compositional generalization by creating more challenging splits based on logical form patterns. in scan (lake and baroni, 2018b) compositional generalization is represented by unseen combinations of seen actions (e.g., jump lturn). @xcite define compositional generalization as generalizing to examples with maximum compound divergence (e.g., combinations of entities and relations) while guaranteeing similar atom distribution to the training set. @xcite design five linguistic types of compositional generalization such as generalizing phrase nesting to unseen depths. in react, our definition of compositional generalization is dependent on the data distribution of the candidate corpus, which determines what compositional patterns are of practical interest and how frequently they occur. another line of work focuses on modeling solutions, mostly ways to explicitly instil compositional bias into neural models. this can be achieved by adopting a more conventional grammar-based approach @xcite or incorporating a lexicon or lexicon-style alignments into sequence models @xcite . other work employs heuristics, grammars, and generative models to synthesize examples for data augmentation @xcite @xcite @xcite or modifies standard training objectives with new supervision signals like attention supervision or meta-learning @xcite @xcite . our work builds on dangle @xcite , a disentangled sequence-to-sequence model, which tries to tackle compositional generalization with architectural innovations. while dangle is conceptually general, our proposal is tailored to the transformer and features two key modifications to encourage more disentangled representations and better computational efficiency. ## conclusions in this paper we focused on two issues related to compositional generalization. firstly, we improve upon dangle, an existing sequence-to-sequence architecture that generalizes to unseen compositions by learning specialized encodings for each decoding step. we show that re-encoding keys periodically, at some interval, improves both efficiency and accuracy. secondly, we propose a methodology for identifying compositional patterns in real-world data and create a new dataset which better represents practical generalization requirements. experimental results show that our modifications improve generalization across tasks, metrics, and datasets and our new benchmark provides a challenging testbed for evaluating new modeling efforts.