Revisit Systematic Generalization via Meaningful Learning (2022)

Abstract: Humans can systematically generalize to novel compositions of existing concepts. Recent studies argue that neural networks appear inherently ineffective in such cognitive capacity, leading to a pessimistic view and a lack of attention to optimistic results. We revisit this controversial topic from the perspective of meaningful learning, an exceptional capability of humans to learn novel concepts by connecting them with known ones. We reassess the compositional skills of sequence-to-sequence models conditioned on the semantic links between new and old concepts. Our observations suggest that models can successfully one-shot generalize to novel concepts and compositions through semantic linking, either inductively or deductively. We demonstrate that prior knowledge plays a key role as well. In addition to synthetic tests, we further conduct proof-of-concept experiments in machine translation and semantic parsing, showing the benefits of meaningful learning in applications. We hope our positive findings will encourage excavating modern neural networks’ potential in systematic generalization through more advanced learning schemes.

## introduction as a crucial characteristic of human cognition, systematic generalization reflects people's talents to learn infinite combinations of finite concepts @xcite . whether connectionist networks can express language and thoughts systematically has been controversial for many years @xcite @xcite @xcite @xcite . to date, the systematic compositionality in neural networks remains an appealing research topic. evidence on multiple explicitly proposed language-based generalization challenges suggests that models lack such cognitive capacity @xcite @xcite @xcite @xcite . tremendous efforts are made to tackle these challenges through architectural modifications @xcite @xcite @xcite , meta-learning (lake, 2019; @xcite , grammar @xcite , neuro-symbolic models @xcite @xcite , data augmentation @xcite @xcite @xcite , and loss design @xcite . despite their astounding accomplishments, standard sequence-to-sequence (seq2seq) models @xcite appear to have relatively weak inductive biases, failing to capture underlying hierarchical structure. in contrast, the successful one-shot generalization in the turn-left experiment on the simplified commai navigation (scan) task reveals the potential of seq2seq recurrent networks in controlled figure 2 : adapted examples from geography. in the left one, intuitively, knowing how the new concept (e.g., "usa") relates to the other existing ones (e.g., "us") can boost the learning and memory of this knowledge as a whole. in the right one, bidirectional arrows denote symmetric relations. "mississippi" and "california" are two specific states, and thus both are hyponyms of "state". in turn, "state" is a hypernym of them. due to a common hypernym, "mississippi" and "california" become a co-hyponym for each other. {"heavily populated", "congested", "populus"} is a group of synonyms as sharing similar semantics. finally, "u.s.", as a kind of abbreviation, is a lexical variant of "united states". in this work, as a response to the call, we question whether neural networks are indeed deficient or just conventional learning protocols unable to exploit their full potential @xcite . we revisit the systematic generalization of seq2seq models from a meaningful learning perspective @xcite @xcite . given the idea that humans are used to memorizing concepts in a relational manner, we hypothesize that the success of the turn-left experiment results from the semantic relationships between old concepts and new ones. for example, in figure 1 , a model can understand the meaning of "turn left and walk and jump left" from "walk left and jump left" via the semantic link between two concepts (in bold) since both denote to the same action "lturn walk". to validate our hypothesis, we reproduce the oneshot compositional generalization by semantic linking that exposes semantic relationships through either inductive learning or deductive learning @xcite @xcite . on the one hand, by introducing new concepts sharing the same context, we hope the model can capture the underlying semantic connections inductively. on the other hand, by involving a rule-like concept dictionary without specific context information, we hope the model can utilize the general cross-lingual supervised signals as anchor points so as to launch the semantic linking deductively. in experiments, we treat concepts in the initial data set as primitives and generate variant samples and rules accordingly. next, we mix them up and construct a seq2seq task after a random split. we repeatedly train and evaluate models but slowly decrease the number of times they see each variant until one-shot learning. we observe there is hardly a performance drop in scan for three representative model structures. this evidences that, with semantic linking, even canonical neural networks can generalize systematically to new concepts and compositions. such observation holds consistently across two more semantic parsing (sp) datasets. the followed sensitivity analysis shows that prior knowledge also takes essential parts. lastly, as a proof-of-concept, we demonstrate how meaningful learning already benefits models in standard machine translation (mt) and sp. overall, our contributions foot_0 are as follows: ## meaningful learning in educational psychology, meaningful learning refers to learning new concepts by relating them to old ones @xcite . in figure 2 , intuitively, the utilization of meaningful learning can encourage learners to understand information continuously built on concepts the learners already understand @xcite . following this, we intend to examine models' systematic compositionality by exploring semantic linking that establishes semantic relations between primitives (old concepts) and their variants (new concepts). we propose to spoon-feed semantic knowledge to models for semantic linking in two ways, that is, inductive learning and deductive learning @xcite @xcite . in this section, we discuss the process of semantic linking and take "jump" from scan as an example primitive to illustrate the learning scheme. ## systematic generalization the following section specifies the setup and outcome of the experiments. we first employ scan as the initial testbed to reproduce the one-shot generalization conditioned on the semantic linking. then, we examine neural networks' potential to achieve this on scan and two real-world tasks of sp, followed by a sensitivity analysis. ## from scan to real data thus far, we have argued the feasibility of systematic generalization activated by semantic linking. we move on to discuss how it already benefits machines in solving real problems. many recent papers propose to improve systematic generalization by techniques such as data augmentation @xcite and metalearning (lake, 2019; @xcite . the success is reasonable given our findings. replacing fragments in real training samples with others that share similar contexts is supported by our inductive learning. we have demonstrated that similar context information can help establish the semantic links between new concepts and old ones, thus enabling models to generalize compositionally. by considering concepts as pointers in the memory, meta-learning equips models with memory loading to make connections between new and old concepts as semantic linking. the utility of similar unsupervised techniques @xcite in both compositional generalization and real tasks can be attributed to inductive learning as well. besides, our sensitivity analysis in section 3.5 shows that adding seemingly independent primitive samples or rules can also improve the generalization, which has been further validated recently @xcite . in addition to inductive-based methods, some works @xcite @xcite , incorporating bilingual dictionaries in low-resource mt, can fall in the field of deductive-based ones. as a proof-of-concept, we reproduce the word-to-word augmentation, or called deductive learning in this work, by training models on not only the base training set but also concept rules. intuitively, we wonder to which extent deductive semantic linking can promote models' performance in mt (iwslt'14 and iwslt'15) and sp (geography and advising). we report the evaluation results in ## conclusion we revisit systematic generalization from a meaningful learning perspective. according to the theory, we conduct semantic linking to expose semantic relations between new and old concepts via either inductive learning or deductive learning. experimental results on scan, geo, and adv support that seq2seq neural networks, as a class of modern machine learning methods, can behave systematically after semantic linking. testing with various difficulties indicates that both semantic linking and prior knowledge are two essential factors in such generalization, in agreement with what humans do in meaningful learning. finally, we group recent methods in either the inductivebased or deductive-based category, followed by a proof-of-concept, to highlight the already-existing advantages of meaningful learning in applications such as machine translation and semantic parsing. we want to underline that, to the best of our knowledge, this work is the first one exploring the optimistic results observed by lake and baroni (2018). our positive findings oppose the recent prevailing view that neural networks appear inherently ineffective in such cognitive capacity, thus confirming the mixed picture. by rationalizing recent findings from a meaningful learning perspective, we hope to encourage followers to interpret the exceptional generalization ability through the connection between neural nets and human cognition.