Meta-Learning to Compositionally Generalize (2021)

Abstract: Natural language is compositional; the meaning of a sentence is a function of the meaning of its parts. This property allows humans to create and interpret novel sentences, generalizing robustly outside their prior experience. Neural networks have been shown to struggle with this kind of generalization, in particular performing poorly on tasks designed to assess compositional generalization (i.e. where training and testing distributions differ in ways that would be trivial for a compositional strategy to resolve). Their poor performance on these tasks may in part be due to the nature of supervised learning which assumes training and testing data to be drawn from the same distribution. We implement a meta-learning augmented version of supervised learning whose objective directly optimizes for out-of-distribution generalization. We construct pairs of tasks for meta-learning by sub-sampling existing training data. Each pair of tasks is constructed to contain relevant examples, as determined by a similarity metric, in an effort to inhibit models from memorizing their input. Experimental results on the COGS and SCAN datasets show that our similarity-driven meta-learning can improve generalization performance.

## introduction compositionality is the property of human language that allows for the meaning of a sentence to be constructed from the meaning of its parts and the way in which they are combined @xcite . by decomposing phrases into known parts we can generalize to novel sentences despite never having encountered them before. in practice this allows us to produce and interpret a functionally limitless number of sentences given finite means @xcite . ## methods we introduce the meta-learning augmented approach to supervised learning from @xcite ; @xcite that explicitly optimizes for outof-distribution generalization. central to this approach is the generation of tasks for meta-learning by sub-sampling training data. we introduce three kinds of similarity metrics used to guide the construction of these tasks. ## related work compositional generalization a large body of work on compositional generalization provide models with strong compositional bias, such as specialized neural architectures @xcite case training generalization accuracy distribution primitive @xmath0. 0.5 1 baseline tree-maml primitive @xmath1. 0.4 0.6 0.8 1 baseline tree-maml primitive @xmath2. 0 0.2 0.4 baseline tree-maml primitive @xmath3. 0 0.5 1 baseline tree-maml meta-learning for nlp meta-learning methods @xcite @xcite ) that are widely used for few-shot learning, have been adapted for nlp applications like machine translation @xcite and relation classification @xcite . in this work, we extend the conventional maml @xcite algorithm, which was initially proposed for few-shot learning, as a tool to inject inductive bias, inspired by @xcite ; @xcite . for compositional generalization, @xcite proposes a meta-learning procedure to train a memory-augmented neural model. however, its meta-learning algorithm is specialized for the scan dataset @xcite and not suitable to more realistic datasets. ## conclusion our work highlights the importance of training objectives that select for robust generalization strategies. the meta-learning augmented approach to supervised learning used here allows for the specification of different constraints on learning through the design of the meta-tasks. our similarity-driven task design improved on baseline performance on two different compositional generalization datasets, by inhibiting the model's ability to memorize large sections of its input. importantly though the overall approach used here is model agnostic, with portions of it (str-maml, lev-maml, and uni-maml) proving dataset agnostic as well requiring only that the input be a natural language sentence. our methods are simple to implement compared with other approaches to improving compositional generalization, and we look forward to their use in combination with other techniques to further improve models' compositional ability. bailin wang, richard shin, xiaodong liu, oleksandr polozov, and matthew richardson. 2020b. rat-sql: relation-aware schema encoding and linking for text-to-sql parsers. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 7567-7578, online. association for computational linguistics. kaizhong zhang and dennis shasha. 1989. simple fast algorithms for the editing distance between trees and related problems. siam journal on computing, 18(6):1245-1262.