Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization (2023)

Abstract: Recent studies have shown that sequence-to-sequence (seq2seq) models struggle with compositional generalization (CG), i.e., the ability to systematically generalize to unseen compositions of seen components. There is mounting evidence that one of the reasons hindering CG is the representation of the encoder uppermost layer is entangled, i.e., the syntactic and semantic representations of sequences are entangled. However, we consider that the previously identified representation entanglement problem is not comprehensive enough. Additionally, we hypothesize that the source keys and values representations passing into different decoder layers are also entangled. Starting from this intuition, we propose CompoSition ( Compo se S yntactic and Semant i c Representa tion s), an extension to seq2seq models which learns to compose representations of different encoder layers dynamically for different tasks, since recent studies reveal that the bottom layers of the Transformer encoder contain more syntactic information and the top ones contain more semantic information. Specifically, we introduce a composed layer between the encoder and decoder to compose different encoder layers’ representations to generate specific keys and values passing into different decoder layers. CompoSition achieves competitive results on two comprehensive and realistic benchmarks, which empirically demonstrates the effectiveness of our proposal. Codes are available at https://github.com/thinkaboutzero/COMPOSITION .

## introduction a crucial property of human language learning is its compositional generalization (cg) -the algebraic ability to understand and produce a potentially infinite number of novel combinations from known components @xcite @xcite show the workflow of how humans exhibit cg. suppose interpreters know the translation: @xcite for "lost the dog" and @xcite for "he liked" (semantic information). when they first encounter "lost the dog he liked", they can correctly translate [丢失了他喜欢的 狗] instead of @xcite depending on pattern 2.3 (syntactic information). 2017). for example, if a person knows "the doctor watches a movie" [医生看电影] foot_0 and "the lawyer" @xcite , then it is natural for the person to know the translation of "the lawyer watches a movie" is [律师看电影] even though they have never seen it before. such nature is beneficial for generalizing to new compositions of previously observed elements, which is often required in real-world scenarios. despite astonishing successes across a broad range of natural language understanding and generation tasks @xcite @xcite , neural network models, in particular the very popular sequence-tosequence (seq2seq) architecture, are argued difficult to capture the compositional structure of human language @xcite @xcite . a key reason for failure on cg is different semantic factors (e.g., lexical meaning and syntactic patterns) required by cg are entangled, which was proved explicitly or implicitly to exist in the representation of the encoder uppermost layer (encoder entanglement problem) by previous studies @xcite @xcite @xcite @xcite @xcite @xcite . in other words, the syntactic and semantic representations of sequences are entangled. in order to alleviate the encoder entanglement problem, one line of research on cg mainly concentrate on improving the encoder representation or separating the learning of syntax and semantics which adopt similar approaches to humans' strategies for cg (see figure 1 ). specifically, several works either produce two separate syntactic and semantic representations, and then compose them @xcite @xcite or design external modules, and then employ a multi-stage generation process @xcite @xcite @xcite . moreover, some studies explore bag-of-words pre-training @xcite , newly decoded target context @xcite or prototypes of token representations over the training set @xcite to improve the encoder representation. furthermore, we hypothesize that the source keys and values representations passing into different decoder layers are also entangled (keys, values entanglement problem), not just the representation of the encoder uppermost layer. we will further illustrate it in section 5.1. therefore, one natural question can be raised: how to alleviate keys, values entanglement problem. as a remedy, we examine cg from a new perspective to solve it, i.e., utilizing different encoder layers' information. we conduct preliminary analysis provided in appendix a, and conclude that the bottom layers of the transformer encoder contain more syntactic information and the top ones contain more semantic information. inspired by this, we collect representations outputted by each encoder layer instead of separating the learning of syntax and semantics. so one intuitive solution to solve keys, values entanglement problem is to learn different and specific combinations of syntactic and semantic information (i.e., representations outputted by each encoder layer) for keys and values of different decoder layers. we argue that an effective composition is to provide different combinations for different tasks and a specific combination for a particular task. for example, the model can learn preference of layers in different levels of the encoder for different tasks (i.e., for a task, the information at encoder layer 0 may be more important, however, for b task, the information at encoder layer 5 may be more important). additionally, the model can select which encoder layer of information is most suitable for itself (that is, which encoder layer of information is the most important) for a particular task. inspired by that, we propose the composed layer (learnable scalars or vectors) to generate different specific source keys and values passing into different decoder layers for different particular tasks, since we argue that the learned scalars or vectors (i.e., different dynamic composition modes) by the model itself during training process can be dynamically adjusted for different particular tasks, and provide a way to learn preference of layers in different levels of the encoder for a particular task. putting everything together, we propose composition (compose syntactic and semantic representations), an extension to seq2seq models that learns to compose the syntactic and semantic representations of sequences dynamically for different tasks. composition is simple yet effective, and mostly applicable to any seq2seq models without any dataset or task-specific modification. experimental results on cfq @xcite (semantic parsing) and cognition @xcite (machine translation, mt) empirically show that our method can improve generalization performance, outperforming competitive baselines and other techniques. notably, composition achieves 19.2% and 50.2% (about 32%, 20% relative improvements) for instance-level and aggregate-level error reduction rates on cognition. extensive analyses demonstrate that composing the syntactic and semantic representations of sequences dynamically for different tasks leads to better generalization results. ## methodology we adopt the transformer architecture @xcite to clarify our method, however, our proposed method is mostly applicable to any seq2seq models. in the following, we first introduce the transformer baseline (section 3.1), and then our proposed composition (section 3.2). ## experiments we mainly evaluate composition on two comprehensive and realistic benchmarks for measuring cg, including cfq @xcite and cognition @xcite . ## analysis in this section, we conduct in-depth analyses of composition to provide a comprehensive understanding of the individual contributions of each component. for all experiments, we train a com-position (6-6 encoder and decoder layers) instead of other experimental settings on the cognition dataset, unless otherwise specified. ## conclusion in this paper, we examine cg from a new perspective, i.e., utilizing different encoder layers' information. specifically, we propose an extension to seq2seq models which composes different encoder layers' representations dynamically to generate specific keys and values passing into different decoder layers. experiments on cognition and cfq have shown the effectiveness of our proposal on cg without any dataset or task-specific modification. to our knowledge, we are the first to point out a new representation entanglement problem and investigate how to utilize information of different encoder layers benefits cg, achieving promising results on two realistic benchmarks. we hope the work and perspective presented in this paper can inspire future related work on cg.