Improving Compositional Generalization in Classification Tasks via Structure Annotations (2021)

Abstract: Compositional generalization is the ability to generalize systematically to a new data distribution by combining known components. Although humans seem to have a great ability to generalize compositionally, state-of-the-art neural models struggle to do so. In this work, we study compositional generalization in classification tasks and present two main contributions. First, we study ways to convert a natural language sequence-to-sequence dataset to a classification dataset that also requires compositional generalization. Second, we show that providing structural hints (specifically, providing parse trees and entity links as attention masks for a Transformer model) helps compositional generalization.

## introduction compositional generalization is the ability of a system to systematically generalize to a new data distribution by combining known components or primitives. for example, assume a system has learned the meaning of "jump" and that "jump twice" means that the action "jump" has to be repeated two times. upon learning the meaning of the action "jax", it should be able to infer what "jax twice" means. although modern neural architectures are pushing the state of the art in many complex natural language tasks, these models still struggle with compositional generalization @xcite . in order to advance research in this important direction, in this paper we present two main contributions 1 . first, we present a binary classification dataset which is hard in a compositional way. this allows for studying the compositional generalization ability of a larger range of models than sequence generation tasks, since the task only requires an encoder, and not a decoder. specifically, 1 http://goo.gle/compositional-classification we present a methodology to convert an existing semantic parsing dataset, cfq @xcite , into a binary classification dataset that is also compositionally hard. our second and main contribution is showing that a transformer-based model can better generalize compositionally if we provide hints on the structure of the input. specifically, we do so by modifying the attention mask used by the model. this is an interesting result, as (except for two additions, which we elaborate on in section 4) attention masks do not "add" any attention capabilities to the model. instead, it seems that it is the removal of certain attention pairs that makes the difference. this suggests that vanilla transformer is having a hard time suppressing non-compositional attention. ## background this section overviews existing work on compositional generalization and then some background on the transformer models used in this paper. please see section b in the appendix for detailed review. ## the cfq classification dataset the compositional freebase questions (cfq) dataset @xcite is an nlu dataset to measure the compositional capability of a learner. it is designed around the task of translating a natural language question into a sparql query. the dataset has been automatically generated by a grammar and contains 239,357 sentence/query pairs. an example is shown in figure 3a . as shown in the original work of @xcite in order to properly measure the compositional generalization ability of a model, the train and test sets should be split with similar distributions of tokens (atoms), but different distributions of their compositions (the compounds). in the cfq dataset, to ensure this, two divergences, namely atom divergence and compound divergence, between the train and dev/test set are measured while constructing the splits. as a result, carefully selected splits called maximum compound divergence (mcd) splits are hard for standard neural networks (they perform well in the train set, but poorly in the test set), while the random splits are easier. we convert the cfq dataset into a dataset with a binary classification task. in this new dataset, the input is a question and a sparql query, and the task is to determine whether these two sequences have the same meaning or not. two considerations must be made to ensure the resulting dataset requires compositional generalization: negative example strategies: positive instances of the binary classification task can be obtained directly from the original dataset, but to obtain negatives, we use either of two strategies: model negatives are important, as otherwise, the task becomes too easy and would likely not require compositional generalization. see figure 1 for examples of random/model negative instances. ## compositional generalization via structure annotation our hypothesis is that part of the difficulty in compositional generalization is to parse the structure of the input. to test this, we evaluate the performance of models when we provide annotations for two structural elements of the inputs: parse trees of both the natural language sentences and sparql queries, and entity cross links (linking entity mentions from the natural language side to the corresponding mentions in the sparql query). the parse trees of the questions are already given in the original cfq dataset as constituency-based parse trees. since the trees include intermediate nodes indicating syntactic structures, we append tokens representing them at the end of each question. we created a simple parser to generate dependencybased parse trees for the sparql queries. we join the roots of the two trees to make a single global tree with @xmath0. we represent the structure of the inputs by masking attention ("hard mask") or with relative attention @xcite ) labels ("soft mask"). additionally, we allow attention pairs in the masks connecting the entities appearing both in the question and the queries. we call these links entity cross links, and they are found by simple string match (e.g. "m0"). notice that while relative attention labels and the additional tokens to represent the constituency parse tree of the natural language add capabilities to the model, the "hard mask" structure annotations described above (which result in the larger performance gains) do not add any attention capabilities to the model. instead, they simply remove non-structure attention edges. figure 3b shows the parse trees, and figure 3c and 3d show the masks for an example. ## results and discussion we used the etc @xcite transformer model implementation as it allows us to provide the hard and soft masks described above in an easy way. in all experiments, we report auc in the dev set as the evaluation metric (we did not evaluate on the test set). please see section a in the appendix for training details. ## conclusions the main contribution of this paper is to show that providing structure annotations in the form of attention masks significantly helps transformer models generalize compositionally. this is interesting for two main reasons: first, it shows that neural network models do have the innate ability to generalize compositionally to some extent, but need some guidance to do so (e.g., by providing attention masks as in our work). this reinforces previous work showing that lstms also can, in principle, generalize compositionally, but they just do so with very low probability @xcite . the second reason is that structure annotations, which we provided manually, could be generated by another model in future work. we also presented a procedure for generating classification datasets that require some degree of compositional generalization starting from sequence generation datasets.