F ROM SCAN TO R EAL D ATA : S YSTEMATIC G ENERALIZATION VIA M EANINGFUL L EARNING Anonymous authors
Paper under double-blind review
A BSTRACT
Humans can systematically generalize to novel compositions of existing concepts.
There have been extensive conjectures into the extent to which neural networks can do the same.
Recent arguments supported by evidence on the SCAN dataset claim that neural networks are inherently ineffective in such cognitive capacity.
In this paper, we revisit systematic generalization from the perspective of meaningful learning, an exceptional capability of humans to learn new concepts by connecting them with other previously known knowledge.
We propose to reassess models’ compositional skills conditioned on the semantic connections between new and old concepts. In experiments, following the meaningful learning principle, we augment a training dataset in either an inductive or deductive manner to exposure such semantic links to models.
Our observations on SCAN, as well as two real-world datasets on semantic parsing, suggest that modern sequenceto-sequence models, including RNNs, CNNs, and Transformers, can successfully one-shot generalize to novel concepts and compositions through semantic linking.
We further demonstrate that both prior knowledge and semantic linking play a key role in achieving systematic generalization and that inductive learning generally works better than deductive learning.
Lastly, we provide an explanation for data augmentation techniques by concluding them into either inductive-based or deductive-based meaningful learning.
We hope our ﬁndings will encourage excavating existing neural networks’ potential in systematic generalization through more advanced learning schemes.
I NTRODUCTION
As a crucial characteristic of human cognition, systematic generalization reﬂects people’s ability to learn inﬁnite combinations of ﬁnite concepts (Chomsky, 1957; Montague et al., 1970).
However, weak systematic compositionality has been considered as a primary obstacle to the expression of language and thought in connectionist networks for a long time (Fodor & Pylyshyn, 1988; Hadley, 1994; Marcus, 1998; Fodor & Lepore, 2002; Frank et al., 2009; Brakel & Frank, 2009; Marcus, 2018).
Whether models can generalize systematically is still an appealing research topic until now.
Recent works state that modern neural networks have not mastered these language-based generalization challenges in multiple explicitly proposed datasets (Lake & Baroni, 2017; Bastings et al., 2018;
Keysers et al., 2019; Hupkes et al., 2020; Kim & Linzen, 2020).
These studies conclude that models lack such cognitive capacity, which calls for a more systematic study.
Apart from the proposal of benchmarks, existing research mainly focuses on novel architectural designs (Chen et al., 2020) data augmentation (Andreas, 2020; Aky¨urek et al., 2021) and meta-learning (Lake, 2019; Conklin et al., 2021) to enable systematic generalization.
In this work, however, we question that whether neural networks are indeed deﬁcient or just conventional learning protocols unable to exploit their full potential (Csord´as et al., 2021).
Inspired by meaningful learning from the ﬁeld of educational psychology (Mayer, 2002), we revisit systematic generalization to see whether neural networks still fail after semantic linking .
To exposure semantic links between new concepts and existing ones, we augment prior knowledge through either inductive learning or deductive learning as what humans do in meaningful verbal learning (Ausubel, 1963).
To be speciﬁc, inductive learning is a bottom-up approach from the more speciﬁc to the mode general.
By introducing new concepts sharing the same context with existing ones in speciﬁc samples, we hope the model can capture the underlying semantic connections and thus generalize to novel compositions of new concepts.
On the contrary, deductive learning is a top-down approach from the more general to the more speciﬁc.
By involving a rule-like concept dictionary without speciﬁc context information, we hope the model can utilize the general cross-lingual supervised signals as anchor points so as to launch the semantic linking.
In experiments, inductive and deductive learning stand for training models on extra samples with or without context, respectively, as two standard data augmentation techniques (Arthur et al., 2016; Wei & Zou, 2019; Nag et al., 2020).
We mainly focus on three semantic relationships, namely, lexical variant , co-hyponym , and synonym .
Starting from SCAN, our experiments conﬁrm that, with semantic linking, even canonical neural networks can generalize systematically to new concepts and compositions.
Moreover, this holds consistent across two more semantic parsing datasets.
As an ablation study, we further examine such one-shot compositional generalization and ﬁnd that both prior knowledge and semantic linking take essential parts.
Lastly, we extend from toy sets to real data and explain how models’ meaningful learning capability beneﬁts them in solving real problems such as machine translation and semantic parsing with the assistance of data augmentation techniques.
Overall, our contributions are as follows: ( 1 ) We formally revisit systematic generalization by introducing semantic linking from a meaningful learning perspective.
( 2 ) We show how to conduct semantic linking by two common data augmentation approaches, either inductively or deductively. ( 3 ) We observe that modern neural networks can achieve systematic generalization with semantic linking, and both prior knowledge and semantic linking play a key role, which is in line with meaningful learning theory.
( 4 ) We extend from SCAN to real data and demonstrate that many recent data augmentation techniques belong to either inductive or deductive learning.
M EANINGFUL L EARNING
Learning new concepts by relating them to the existing ones is deﬁned as a process of meaningful learning in educational psychology (Ausubel, 1963; Mayer, 2002).
The utilization of meaningful learning can encourage learners to understand information continuously built on concepts the learners already understand (Okebukola & Jegede, 1988).
Following the same idea, we intend to examine models’ systematic compositionality by exploring semantic linking that establishes semantic relations between primitives P (old concepts) and their variants V :=
{ V p | ∀ p ∈ P } (new concepts).
To spoon-feed semantic knowledge to models for semantic linking, we augment the training data by either inductive learning or deductive learning (Hammerly, 1975; Shaffer, 1989; Thornbury, 1999) as humans learning vocabulary.
In this section, we discuss the process of semantic linking and take “jump” from SCAN as an example primitive to illustrate the learning scheme in Figure 1.
2. S EMANTIC L INKS
We aim to revisit systematic generalization by exposing semantic links such as lexical variants, cohyponyms, and synonyms.
Lexical Variant refers to an alternative expression form for the same concept.
Co-hyponym is a linguistic term to designate a semantic relation between two group members belonging to the same broader class, where each member is a hyponym and the class is a hypernym (Lyons & John, 1995).
Synonym stands for a word, morpheme, or phrase that shares exactly or nearly the same semantics with another one.
We provide an example and a detailed description in
Appendix.
2. I NDUCTIVE L EARNING
Inductive learning is a bottom-up approach from the more speciﬁc to the more general.
In grammar teaching, inductive learning is a rule-discovery approach that starts with the presentation of speciﬁc examples from which a general rule can be inferred (Thornbury, 1999).
Inspired by that, we propose to conduct semantic linking by introducing variant samples sharing the same context with their primitives during training.
The assumption is that models can observe the interchange of primitives and their variants surrounded by the same context in the hope of coming up with a general hypothesis that there is a semantic linking between primitives and their variants (Harris, 1954).
Formally, we describe inductive learning as follows.
For a sequence-to-sequence task T : X → Y , we have a source sequence x ∈ X and its target sequence y ∈ Y .
We prepare prompts set Z := { z = f prompt ( x ) | x ∈ X } , where f prompt ( · ) replaces the primitive in x with a slot mark [ z p ] .
Then, we generate X IL :
= { x IL =
f fill ( z , v ) | z ∈ Z , v ∈ V } by ﬁlling [ z p ] with variants in V p .
There is no change from the target side, so we get Y IL by copying y as y IL for each x IL correspondingly.
Finally, we train models on ( (cid:2) XX IL (cid:3) , (cid:2) YY IL (cid:3) ) to operate semantic linking inductively.
In practice, as shown in Figure 1, we ﬁrst get a prompt “ [Z jump ] twice ” given a primitive sample “ jump twice ”.
After that, we generate variant samples by replacing the slot mask
“ [Z jump ] ” with variants such as “ jump 0 ”.
Finally, training models on generated variant samples like “ jump 0 twice ” combined with prior knowledge, we aim to establish the semantic relationships between primitives and their variants inductively.
This process can be also treated as a kind of replacement augmentation (Wei & Zou, 2019).
2. D EDUCTIVE L EARNING
Deductive Learning, on the opposite of inductive learning, is a top-down approach from the more general to the more speciﬁc.
As a rule-driven approach, teaching in a deductive manner often begins with presenting a general rule and is followed by speciﬁc examples in practice where the rule is applied (Thornbury, 1999).
To align with this deﬁnition, we intend to do semantic linking deductively by combining a bilingual dictionary that maps primitives and their variants to the same in the target domain.
This additional dictionary, hence, mixes the original training task with word translation (Mikolov et al., 2013b).
Without any speciﬁc context, we hope the model can utilize the general cross-lingual supervised signals as anchor points so as to launch the semantic linking.
Formally, we describe deductive learning as follows.
We ﬁrst treat P as the source dataset X DL P directly and then prepare the corresponding target dataset Y DL P by either decomposing samples from Y manually or feeding X DL P to a trained external model.
Similarly, we can consider V as another source dataset
X DL V and prepare its target dataset Y DL V by copying the corresponding y DL P as y DL V for all x DL V as variants of each x DL P .
After all, we get X DL as (cid:2)
X DL P X DL V (cid:3) and Y DL as (cid:2)
Y DL P Y DL V (cid:3) .
The mapping from
X DL to Y DL is a dictionary to translate primitives and their variants to the same targets without any speciﬁc context information.
We name ( x DL , y DL ) as a concept rule , ( x DL P , y DL P ) as a primitive rule , and ( x DL V , y DL V ) as a variant rule since they are more rule-like without contexts.
We train models on ( (cid:2)
XX DL (cid:3) , (cid:2) YY DL (cid:3) ) to operate semantic linking deductively.
In practice, as presented in
Figure 1, we directly make use of primitive “ jump ” and its variants such as “ jump 0 ” as the source sequences with action “JUMP” as their same target sequences.
By exposing both the primitive rule “ jump ” → “JUMP” and variants rules like “ jump 0 ” → “JUMP” during training, we intend to build the semantic connections between primitives and their variants deductively.
This manner is similar to the bilingual dictionary augmentation (Arthur et al., 2016; Nag et al., 2020).
S YSTEMATIC G ENERALIZATION
Although previous studies argue that neural networks fail to match humans in systematic generalization (Lake & Baroni, 2017; Keysers et al., 2019), we revisit such algebraic compositionality conditioned on the semantic linking to see whether the conclusion will change.
The following section moves on to specify the process and outcome of experiments.
We ﬁrst intend to make use of
SCAN as the initial testbed to observe the presence of systematic generalization with the assistance of semantic relations.
Then, we verify neural networks’ potential to achieve the systematic generalization activated by semantic linking on SCAN, as well as two real-world tasks of semantic parsing.
Following ablation studies further examine models’ compositional capability
3. D ATASETS
There is evidence suggesting that SCAN may be far from enough to fully capture the kind of generalization, where even a simple model can behave as if it owns comparable skills (Bastings et al., 2018; Keysers et al., 2019).
Thus, starting from SCAN, we introduce GEO and ADV generated respectively from real semantic parsing datasets: Geography and Advising.
2 Modiﬁcation on datasets is speciﬁed in each experiment for the goal of examining machines’ systematic generalization across conditions.
SCAN is one of the benchmarks to investigate neural networks’ compositional generalization (Lake & Baroni, 2017).
It includes 20910 pairs of commands in English to their instructed action sequences.
We deﬁne P SCAN :
= { “ jump ” , “ look ” , “ run ” , “ walk ” } to be in line with previous works.
We focus on lexical variants and create V SCAN by adding a sufﬁx that consists of an underline and a unique number for each primitive.
We control | V SCAN | by setting the upper limit of this number.
An example variant of “ jump ” is “ jump 0 ” and both mean the same action “JUMP”.
Geography is a common semantic parsing dataset (Zelle & Mooney, 1996; Srinivasan et al., 2017).
It is also named as geo880 since it contains 880 examples of queries about US geography in natural language paired with corresponding query expressions.
It is later formatted to SQL language with variables in the target sequences (Finegan-Dollak et al., 2018).
GEO is the dataset generated based on Geography, where we regard 4 of 9 annotated variables as hypernyms and keep them as they are in SQL sequences.
The other variables are restored by entities from the source sequence accordingly.
As a result, the overall data size is 618 after processing and we can make use of the “is-a” hypernymy relations between entities and variables for semantic linking.
To be speciﬁc, we deﬁne P GEO := { “ new york city ” , “ mississippi rivier ” , “ dc ” , “ dover ” } with V GEO consisting of entities as co-hyponyms sharing the same variable group with primitives.
An example variant of “ new york city ” is “ houston city ” and both are in the same variable group “CITY NAME”.
Advising , as our second semantic parsing dataset, includes 4570 questions about course information in natural language paired with queries in SQL (Finegan-Dollak et al., 2018).
Similar to GEO,
ADV is generated on the basis of Advising with 4 of 26 variables as hypernyms.
Precisely, we deﬁne P ADV :
= { “ a history of american ﬁlm ” , “ aaron magid ” , “ aaptis ” , “ 100 ” } and V ADV as cohyponyms of primitives sharing the same variables.
For instance, “ advanced at ai techniques ” is a co-hyponym of “ a history of american ﬁlm ” sharing the same variable “TOPIC”.
3. M ODELS AND E XPERIMENTAL S ETUP
What follows is an account of network conﬁgurations and experimental settings.
Without speciﬁc instruction, they are shared throughout experiments.
Models.
After testing a range of their adapted versions, we employ three dominant model candidates with an encoder-decoder framework (Sutskever et al., 2014), that is, RNN, CNN, and TFM.
In terms of RNN, we reproduce bi-directional recurrent networks (Schuster & Paliwal, 1997) with long shortterm memory units (Hochreiter & Schmidhuber, 1997) and an attention mechanism (Bahdanau et al., 2015).
We follow the convolutional seq2seq architecture presented by Gehring et al.
(2017) with regard to CNN and the attention-based structure proposed by Vaswani et al.
(2017) in the case of
TFM.
More details are provided in Appendix.
Training.
We apply the mini-batch strategy to sample 128 sequence pairs for each training step.
We use Adam optimizer (Kingma & Ba, 2015) with an (cid:96) 2 gradient clipping of 5 .
(Pascanu et al., 2013) and a learning rate of 1 e − 4 to minimize a cross-entropy loss.
We freeze the maximum training epoch at 320 for CNN and 640 for RNN and TFM.
In contrast to early stopping (Prechelt, 1998), we prefer a ﬁxed training regime sufﬁcient enough for models to fully converge in practice with a focus on the systematic generalization observation instead of superior structure exploration.
To prevent uncontrolled interference, we train all models from scratch instead of ﬁne-tuning (Devlin et al., 2019).
Evaluation.
Token accuracy and sequence accuracy serve as two primary metrics in the following experiments.
The former is a soft metric that allows partial errors in a sequence, while the latter is tricky and strictly does not.
The reported results, along with standard deviation, are the mean of ﬁve runs.
3. E XPERIMENT : M EANINGFUL L EARNING
This experiment probes the models’ compositional generalization via meaningful learning in SCAN.
We compare performances across various conditions starting from the conventional training pipeline as a baseline.
Usually, new concepts appear as out-of-vocabulary (OOV).
A typical training pipeline often involves replacement (Wei & Zou, 2019) to handle new concepts, especially for those sharing same or close semantic with existing concepts.
Thanks to their incredible algebraic compositionality, humans can effectively capture the underlying semantic connections between new concepts and old ones and generalize the prior knowledge to novel combinations by meaningful learning, given only a few demonstrations.
To investigate the extent to which models can do the same, we gradually reduce the number of training samples generated by replacement augmentation until there is only one for each variant.
Although the ﬁnal one-shot learning (Vinyals et al., 2016) is challenging, we hope to observe the presence of models’ meaningful learning by measuring the performance loss due to a decreasing number of training samples per variant.
Experimental setup.
Following replacement augmentation, we assign placeholders at the positions of primitives in source sequences and later put back their 10 variants but keep the identical target sequences.
Consequently, we have a total of 329,190 samples when | V SCAN | is 40 and randomly split them into a training set (80%) and a test set (20%).
The training set is further processed to remove samples having multiple variants.
Thus, we ensure that the number of variants’ occurrences iswhile training at the one-shot condition.
Eventually, the training dataset contains 235,002 samples.
Models directly trained on this full dataset serve as baselines.
Then, to format a gradual transition from baselines to meaningful learning, we train the same models on various datasets conditioned on a decreasing number of augmented samples for each variant until the one-shot learning setting.
Besides, we use the variant rule “ jump 0 ” → “JUMP” as the only training sample for “ jump 0 ” as a case of our deductive learning and consider the rest as our inductive learning.
Results.
Surprisingly, as elaborated in Figure 2, RNN has no signiﬁcant performance drop when the training size is reduced from 235,002 (100%) to 16736 (1).
It still achieves 99.92% test sequence accuracy when there is only one training sample for each variant.
The same happens for CNN and
TFM.
Despite a slight ﬂuctuation, they keep the results almost consistent regardless of whether the number of training variant samples is full or 1.
The single sample works as a whole augmented dataset and enables models to generalize to novel compositions of learned variants.
We want to underline that the sample can be either just a variant rule or a variant sample derived from a primitive prompt.
As we deﬁned, developing semantic linking through the former is deductive learning, and that through the latter is inductive learning.
By utilizing such semantic relations between primitives and their variants, models show they can perform one-shot generalization systematically via meaningful learning after semantic linking.
3. S EMANTIC L INKING I NJECTION
Having observed the success after semantic linking, one question that needs to be asked, however, is how it works.
Therefore, the following two experiments evaluate models’ systematic generalization, [Table] particularly for prior knowledge and semantic linking.
A sliding scale of difﬁculty is carefully designed by weakening these two factors according to the policy that the greater the difﬁculty, the more compositional skills are required.
We further validate our ﬁndings on GEO and ADV.
We use the same evaluation protocol across different datasets in this section.
Given dataset ( X , Y ) as prior knowledge regarding primitives, we generate the test set by replacement augmentation.
Speciﬁcally, we replace the primitives in source sequences with their variants to generate novel compositions.
So far, variants exist as OOV since they are not in the training set.
Then, we incorporate additional either ( X IL , Y IL ) or ( X DL , Y DL ) to the base training set ( X , Y ) so as to introduce variants in training and establish semantic linking inductively or deductively.
As in the previous experiment, we ensure that each variant only has a single sample and appears only once.
After training on ( (cid:2) XX IL (cid:3) , (cid:2) YY IL (cid:3) ) or ( (cid:2) XX DL (cid:3) , (cid:2) YY DL (cid:3) ) , models are evaluated on the same test set prepared before by replacement augmentation.
For convenience, we keep the same settings as in the previous experiment, where | V SCAN | is 40 and 10 variants for each primitive.
We use the full variants set for GEO, for example, 39 variants for “ new york city ”, while we randomly samplevariants for each primitive in ADV so that we cover all the variants with an appropriate test size.
3.4. E XPERIMENT : S EMANTIC L INKING I NJECTION VIA I NDUCTIVE L EARNING
Experimental setup.
We increase the difﬁculty of compositional learning by excluding primitive samples from the training set.
We want to stress that, with a higher level of difﬁculty, models have to generalize not only to new concepts but also to their new compositions.
• Standard : Models are trained on ( (cid:2) XX IL (cid:3) , (cid:2) YY IL (cid:3) ) without any adjustments.
• Difﬁcult : We remove primitive samples sharing the same context with their variant samples.
For example, we remove “ jump twice ” due to “ jump 0 twice ”, and thus models have to generalize to “ jump 0 twice ” without seeing “ jump twice ”.
• Challenging: We also exclude primitive training samples of the same length as their variant samples.
For instance, models have to reproduce the same generalization to “ jump 0 twice ” without seeing primitive samples of length 2, including “ jump twice ”, “ jump right ”, “ jump left ”, “ jump thrice ”, and many others.
Results.
In SCAN, what stands out in Table 2 is an excellent one-shot generalization for all three networks.
The participation of ( X IL , Y IL ) induces a near-perfect generalization.
Even the worst results obtained by TFM in Challenging are around 98 .
76% and 96 .
38% in terms of token accuracy and sequence accuracy separately.
The outcomes conﬁrm that networks can inductively learn the semantic relation from context after semantic linking.
After that, models of different architectures can successfully achieve systematic generalization to novel compositions of variants during the test.
What is noticeable is a slight drop in both metrics as the difﬁculty upgrades.
The disappearance of the training samples in Difﬁcult and Challenging settings can lead to a performance drop.
This is well in line with the widely accepted belief in meaningful learning theory, as well as our expectation, that prior knowledge is one of the keys related to humans’ remarkable generalization.
Therefore, we conclude that both semantic linking and background knowledge exert powerful effects upon the potential of models to generalize systematically.
The trends above on SCAN can also be found on GEO and ADV, while more apparent changes in metrics again verify our ﬁndings that prior knowledge is essential.
Either excluding primitive samples containing the same context or those of the same sequence length as their variant samples can produce a steep fall in the generalization, which is not so sharp on SCAN.
On GEO, CNN can lose an absolute sequence accuracy of almost 18 .
26% from Standard to Difﬁcult, and that for TFM drops 7 . 66% .
This upholds our argument that generalization via meaningful learning is inseparable from sufﬁcient prior knowledge.
The overall [Table] decline in performance can be attributed to the switch from toy sets to actual datasets since both
GEO and ADV own a much more complex encoding and decoding space than SCAN.
3.4. E XPERIMENT : S EMANTIC L INKING I NJECTION VIA D EDUCTIVE L EARNING
Experimental setup.
We increase the difﬁculty of compositional learning by excluding primitive rules from the training set as follows:
• Standard : Models are trained on ( (cid:2)
XX DL (cid:3) , (cid:2) YY DL (cid:3) ) without any adjustments.
• Difﬁcult : We remove primitive rules from the training set, and train models on ( (cid:2) XX DL V (cid:3) , (cid:2)
YY DL V (cid:3) ) .
Results.
In SCAN, incorporating deductive semantic linking, all three networks are able to attain satisfying compositional generalization as shown in Table 3.
CNN achieves the highest 99 .
96% in
Standard, while TFM takes the lowest 91 .
26% in Difﬁcult with regard to sequence accuracy.
However, even the lowest one is impressive as there is only one variant rule to introduce each variant during training.
We can see a consistent decline in accuracy across three different models if we undermine the semantic linking by removing primitive rules in Difﬁcult.
The most signiﬁcant sequence accuracy drop of 3 . 3% came from CNN when the difﬁculty upgrades.
We further validate our ﬁndings on GEO and ADV, and ﬁnd a similar trend.
There is a persistent performance loss because of the absence of primitive rules from the training set across models.
Concretely in GEO, the grade of
CNN declines from 32 .
33% in Standard to 23 .
58% in Difﬁcult in terms of sequence accuracy.
The causal role of semantic linking is also demonstrated by varying the difﬁculty.
Overall, the joining of concept rules assists in developing semantic links between primitives and variants during training, by which models can behave compositionally during the test.
Moreover, the different outcomes between Standard and Difﬁcult indicate that either concept rules or just variant rules can connect primitives with their variants semantically, though the former is better than the latter.
Again, the overall performance fall is the result of the more complicated task.
Another noteworthy ﬁnding is that neural networks can realize systematic generalization in either an inductive or a deductive way but perform better in the former setting.
By comparing such preference in Table 2 and Table 3, we ﬁnd that current black-box neural nets are more capable of exploring rules and patterns from speciﬁc samples with context information rather than understanding knowledge from general concept rules in our experiments.
This sheds light on why current machine learning is still highly data-driven and can hardly break through the bottleneck to realize advanced logic reasoning as human beings.
How to improve models’ generalization in deductive learning is an interesting future direction that we will work on.
3. A BLATION S TUDIES AND A NALYSIS
To explore other factors that may impact deductive learning, we conduct ablation studies with a varying | P SCAN | from { 1,2,3,4 } and | V SCAN | from { 1,5,10,15,20 } over RNN on SCAN.
The experimental setup is borrowed from Standard in Section 3.4.2.
Impact of | P SCAN | .
What attractive in Figure 3 (a) is that when the number of primitives grows, the generalization performance improves simultaneously in terms of both accuracy boosting and variance reduction.
It is counter-intuitive to see such improvement as we expect that primitive rules should work independently, and the number of primitives should not impact the systematic [Table] generalization a lot.
A potential reason is that semantic linking built by various “independent” primitive rules can proﬁt each other to trigger a more robust and stable systematic generalization.
For example, “ jump ” → “JUMP” and “ look ” → “LOOK” separate them from the samples with context information, such as “ jump right ” and “ look right ”.
Thus, we can regard “ right ” as a compositional rules shared among primitive samples and ﬁnally encourage models to generalize effectively.
Impact of | V SCAN | .
As presented in Figure 3 (b), RNN generalizes consistently well when the number of variants goes up.
Therefore, we report that the generalization among variants of the same primitive has a certain degree of independence within a reasonable range (e.g., ≤ 20 ).
F ROM SCAN TO R EAL D ATA
Thus far, we have argued the feasibility of systematic generalization activated by semantic linking, as well as other factors such as prior knowledge.
We move on now to discuss how such generalization already beneﬁt the eventual performance of machines in solving real problems.
Many recent papers propose to improve generalization on SCAN by data augmentation methods.
Meta-learning is reported to solve compositional problems by equipping models with memory loading (Lake, 2019).
The success is reasonable due to augmented data for the application of meta-learning.
By considering concepts as pointers in the memory, models are designed to make connections between new and old concepts as semantic linking.
Andreas (2020) suggests replacing fragments in real training samples with others that sharing similar contexts, which can also be supported by our inductive learning.
As in our ﬁndings, similar context information can help establish the semantic links between new concepts and old ones, thus enable models to generalize to novel combinations.
Besides, we have proved how replacement augmentation (Wei & Zou, 2019) works in Section 3.3.
We would like to stress that the utility of similar unsupervised techniques (Xie et al., 2019) in both compositional generalization and real tasks can be attributed to inductive learning as well since there is a need for systematic generalization in practice.
In addition to inductive-based ones, we notice many works incorporating bilingual dictionaries (Arthur et al., 2016; Nag et al., 2020), or called concept rules by us, in low-resource machine translation can fall in the ﬁeld of deductive-based methods.
As a proof-of-concept, we reproduce the
[Table]
[Table]
word-to-word augmentation, or called deductive learning, by training models on not only the base training set but also concept rules.
Intuitively [Table] we wonder to which extent deductive semantic linking can promote models’ performance in common machine translation (IWSLT’14 and IWSLT’150) and semantic parsing (Geography and Advising).
In machine translation, we construct bilingual dictionaries by feeding vocabulary to the Google Translation API.
The word translation can be regarded as concept rules if we consider synonyms of a primitive as their variants and such synonymous relationships as semantic links.
Consequently, we get 144,874 word samples as a training supplementary for En-De and De-En, and 110,099 for En-Fr and Fr-En.
In semantic parsing, we construct entity dictionaries by collecting entities (e.g., “ new york city ”).
They are translated to themselves since they do not change from the source natural language to the target SQL.
The entity mapping can be treated as concept rules in the view of semantic linking.
After that, we have a map of 103 entity translations for Geography and 1846 for Advising.
A detailed experimental setup can be found in
Appendix.
We report the evaluation results in Table 4 and Table 5, where the same models with deductive semantic linking can consistently obtain performance gains.
C ONCLUSION
Overall, we revisit systematic generalization from a meaningful learning perspective and introduce semantic linking to exposure semantic relations between new and old concepts to models during training.
To establish such semantic networks, we take advantage of two common data augmentation methods and name them as inductive and deductive learning to align with the meaningful learning theory.
The observed one-shot generalization on SCAN supports that neural networks as a class of modern machine learning methods can behave systematically after semantic linking. Extensive empirical results on SCAN, GEO, and ADV illustrate that prior knowledge and semantic linking are two essential factors in such generalization, in line with what humans do in meaningful learning.
Given such ﬁndings, we further group recent data augmentation methods in either the inductivebased or deductive-based category, followed by a proof-of-concept to highlight how semantic linking already beneﬁts models in solving real tasks such as machine translation and semantic parsing.
We hope this paper can encourage future works to excavate neural networks’ potential in systematic generalization through more advanced learning schemes.
E THICS S TATEMENT
As far as we know, we do not see any potential concerns such as negative societal impacts from our work.
R EPRODUCIBILITY S TATEMENT
Code, processed datasets, and experiment log of this work can be found in the attached supplementary materials and will be publicly available at GitHub.
The source of raw datasets and external tools are presented in footnotes.
Additional details regarding literature, model conﬁgurations, data statistics and experimental setup are offered in Appendix as a reference in the main paper
A A PPENDIX
In the following pages, we discuss our work in detail.
Our code and data can be found in the attached supplementary materials.
B S EMANTIC L INKS
Lexical Variant refers to an alternative expression form for the same concept, where the various forms may derive from foreign languages, abbreviations, and even mistakes.
A basic assumption is that all languages change over time due to non-linguistic factors.
Since the rise of sociolinguistics in the 1960s, studies on linguistic variability, a characteristic of language, are central to the language use and motivations for speakers to vary the pronunciation, word choice, or morphology of existing concepts (Labov, 1963).
Taking “ United States of America ” as an example, people have generally accepted the semantic connections among its lexical variants in history, including “ America ” and “ United States ”, as well as the initialisms “ U.S. ” and “ U.S.A ”.
Many efforts have been devoted on lexical variants representation (Nguyen & Grieve, 2020), detection (Barteld, 2017), normalization (Baldwin et al., 2015) to keep machines up with the trend of the times.
Co-hyponym is a linguistic term to designate a semantic relation between two group members belonging to the same broader class, where each member is a hyponym, also called subtype or subordinate, and the class is a hypernym (Lyons & John, 1995).
The “is-a” hypernymy relation between a generic hypernym and its speciﬁc hyponyms builds semantic connections among cohyponyms.
An example of such a hierarchical structure can be “ Mississippi ” and “ Massachusetts ” in the domain of “ state ”.
Speciﬁcally, “ Mississippi ” and “ Massachusetts ” are two hyponyms, and “ state ” is a hypernym.
Thus, “ Mississippi ” and “ Massachusetts ” are semantically connected to be co-hyponyms for each other.
Harvesting hypernymy relations (Wang & He, 2020) plays an essential role for downstream knowledge graph construction (Ji et al., 2021), out-vocabulary generalization (Dash et al., 2020), taxonomy expansion (Yu et al., 2020b), etc.
Synonym stands for a word, morpheme, or phrase that shares exactly or nearly the same semantics with another one.
Many tend to assume synonyms are utterances that occur in most contexts in common, so they are semantically closely related enough to be synonyms for each other (Rubenstein &
Goodenough, 1965; Harris, 1954).
The existence of the association to contexts is a basic assumption supporting the advance of recent masked language modeling (Devlin et al., 2019).
Given that, one of the deﬁnitions of a synonymous relation is a semantic link between two expressions if substitution of one for the other never hurts the true value of the context (Stanojevi´c et al., 2009).
For instance, the substitution of “ heavily populated ” for “ populous ” will seldom alter the truth of the sentence in Figure 4.
Such semantic similarity can be observed in continuous vector space from a trained representation as well (Mikolov et al., 2013a).
Synonym discovery (Yu et al., 2020a) has been a fundamental job to construct knowledge base and thus beneﬁts substantial researches.
congested , populus } , { area , region } are two groups of synonyms for sharing same or similar semantics.
Finally, U.S. , as a kind of abbreviation, is a lexical variant of United States .
C D ATA
IWSLT involves IWSLT’14 (Cettolo et al., 2014)
English-German (En-De) and German-English (De-En), IWSLT’15 (Cettolo et al., 2015)
English-French (En-Fr) and French-English (Fr-En) translation tasks.
The goal is to translate a sentence from one language to the other.
The IWSLT’
En-De and De-EN have 160,239 sequence pairs for training and 7,283 for validation.
We make use of IWSLT14.TED.dev { 2010, 2012 } and IWSLT14.TED.tst { 2010, 2011, 2012 } to measure translation performance, resulting in a total of 6,750 test samples.
In terms of IWSLT’15 En-Fr and
Fr-En, there are 205,572 sequence pairs for training.
We employ IWSLT15.TED.dev2010 and
IWSLT15.TED.tst { 2010, 2011, 2012, 2013 } as the validation set and IWSLT15.tst { 2014, 2015 } as the test set.
As a consequence, there are 5,519 samples for validation and 2,385 for evaluation.
For all four translation tasks, we apply BPE with 10K tokens to share.
D M ODELS
All models are developed with the encoder-decoder framework (Sutskever et al., 2014).
We reproduce RNN, CNN, and TFM by ourselves to have fewer parameters than original versions for the convenience of verifying systematic generalization.
The dropout rate is 0.5 for RNN, CNN, and TFM (Srivastava et al., 2014).
We implement LSTM, Transformer, and Dynamic Conv.
under the
[Table]
framework fairseq .
7 (Ott et al., 2019) and inherit its default model structures.
8 Without notes in tasks, hyperparameters are shared throughout the work.
We train all of our models on a single Nvidia Tesla V100.
RNN denotes bi-directional recurrent network (Schuster & Paliwal, 1997; Hochreiter & Schmidhuber, 1997) with long short-term memory units and an attention mechanism (Bahdanau et al., 2015).
Its encoder consists of two layers with a hidden size of 256 in each direction, and its decoder has one layer with a hidden size of 512.
The embedding size is 512 for both encoder and decoder.
There are a total of 5 .
29 M trainable parameters.
Teacher forcing with a rate of 0 .
5 serves to spur up the training process (Williams & Zipser, 1989).
CNN denotes the fully convolutional seq2seq network (Gehring et al., 2017).
The size of the position embedding layer is 128 for encoding and 256 for decoding, while that of the token embedding layer is 512 for both encoding and decoding.
There are 10 convolutional layers with 512 as the hidden size and 3 as the kernel size in both encoder and decoder to generate a total of 33 .
55 M trainable parameters.
TFM denotes transformers, an attention-based network (Vaswani et al., 2017).
As a tiny version,
TFM has 2 layers for each encoder and decoder with 8 attention heads and a dimension of 512.
The size of the feedforward layer is 2048.
We utilize the cyclic nature of sin and cos functions to represent token positions.
There are a total of 15 .
02 M trainable parameters.
LSTM is adapted from the recurrent network used by Luong et al.
(2015) for statistical machine translation.
The size of the embedding layer is 1000.
There are 4 layers in both encoder and decoder with a hidden size of 512 and a dropout rate of 0.2.
Transformer , the same as TFM, is adapted from the base version of transformers in the work of
Vaswani et al.
(2017), while TFM is a tiny version to test systematic generalization.
The dimension is 512 for the embedding layer, 1024 for the feedforward layer, and 512 for the attention layer.
There are 6 attention blocks in both encoder and decoder with 4 attention heads and 0.3 dropout probability.
Dynamic Conv. is adapted from the seq2seq convolutional network proposed by Wu et al.
(2019), where the hidden size of the embedding layer, encoder layer, and decoder layer is 512.
The number of attention heads is 4, and the dimension of the feedforward layer is 1024 for both encoder and encoder.
There are 6 layers in the encoder and 7 layers in the decoder.
The dropout rate is 0.1 for both attention and weight units.
E E XPERIMENTS S EMANTIC L INKING I NJECTION VIA I NDUCTIVE L EARNING
Semantic linking can be operated via inductive learning, where we replace the concept in the prompt with primitives and their variants.
The learning rate to train CNN in GEO is changed to 5 e − 4 .
[Table] [Table]
Prompts used in SCAN, GEO, and ADV are expressed in Table 8.
Detailed experimental results with respect to three levels can be found in Table 9, Table
[Table]
and Table
[Table] [Table]
[Table] [Table]
[Table]
S EMANTIC L INKING I NJECTION VIA D EDUCTIVE L EARNING
Semantic linking can be established via deductive learning, where we put concept rules without context information in the training set instead of speciﬁc sequence samples.
Example concept rules for SCAN, GEO
[Table]
and ADV are presented in Table 12.
Detailed experimental results with respect to two levels can be found in Table 13 and Table
[Table] [Table]
M ACHINE T RANSLATION
We show how semantic linking already beneﬁts models’ performance in machine translation.
The semantic links between primitives and their variants in machine translation is built upon the synonymous relations between tokens such as “ heavily populated ” and “ populous ”.
Given that synonymous connection is reversible as shown in Figure 4, a primitive can also be the other primitives’ variant.
Speciﬁcally
[Table]
we collect a dictionary of tokens for the source language and feed the token to the
Google Translation API to obtain a token map from the source language to the target one.
The same operation can be repeated from the target language to the source one.
Two dictionaries are combined into one with duplicates removed.
Consequently, we get 144,874 token-level samples as a
[Table]
training supplementary for IWSLT’14 En-De and De-En, and 110,099 for IWSLT’15 En-Fr and FrEn, which leads to a total of 305,113 training samples for IWSLT’14 En-De and De-En and 315, for IWSLT’15 En-Fr and Fr-En after such vocabulary augmentation.
Experimental Setup.
We evaluate our approach on IWSLT’14 Cettolo et al.
English-German (En-De) and German-English (De-En), IWSLT’15 Cettolo et al.
English-French (En-Fr) and
French-English (Fr-En) translation tasks.
We follow the standard evaluation protocol Ott et al.
(2019) that keeps the original training set and validation set but combines multiple previous test sets for ﬁnal evaluation 9 .
We apply BPE with 10K tokens for all tasks and report both BLEU Papineni et al.
and SacreBLEU Post (2018) scores for three baselines: LSTM Luong et al.
(2015), Transformer Vaswani et al.
(2017), and Dynamic Conv.
Wu et al.
(2019) in comparsion with same structures augmented by our method.
Results.
From Table 4, we observe a consistent improvement in both BLEU and SacreBLEU over all baselines when performed vocabulary augmentation, particularly up to 1 in SacreBlEU.
The additional synonym pairs not only construct the semantic linking between tokens in two languages explicitly, but also create a complicated semantic linking network implicitly because of synonyms within the single language and the transitivity nature of synonym relation.
Our experiments prove that semantic linking, which allows models to generalize systematically, is also beneﬁcial for improving machine translation performance.
S EMANTIC P ARSING
We consider a variable as a hypernym for its values and entities belonging to the same variable as co-hyponyms in semantic parsing.
Thus, we can treat all the entity values for all variables as primitives and the translations from primitives to their variables as primitive rules that later joins the base training set.
For a fair comparison, a token from this extra dataset will be marked as a unique unknown mark, “¡unk¿”, if it does not exist in the original base training set.
After that, we have a map of 103 entity translations for Geography and 1846 for Advising, resulting in a training size change from 701 to 804 for Geography and from 3814 to 5660 for Advising.
Experimental Setup.
We evaluate our method on two semantic parsing benchmarks, Geography, and Advising.
We train the same models as we analyzed before without more hyperparameter tuning, including RNN, CNN, and TFM.
There are some changes for CNN, where the learning rate is 5 e −in Geography, and the maximum sequence length for the decoder position embedding is 312 in
Advising.
We split 10% training samples as the validation set to ﬁnd the converged epoch and then add it back to the training set for the ﬁnal report.
Results.
As elaborated in Table 5, all three networks can achieve better performance in terms of both accuracy and variance.
Speciﬁcally, a 10 .
76% token accuracy and 9 .
95% sequence accuracy boosting are observed from RNN on Advising after such entity augmentation.
The results suggest that models can learn semantic linking or be more familiar with similar contexts from those primitive rules in a deductive way to enhance model systematic generalization and ﬁnally lead to better outcomes.