L EARNING TO G ENERALIZE C OMPOSITIONALLY BY T RANSFERRING A CROSS S EMANTIC P ARSING T ASKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural network models often generalize poorly to mismatched domains or distributions.
In NLP, this issue arises in particular when models are expected to generalize compositionally, that is, to novel combinations of familiar words and constructions.
We investigate learning representations that facilitate transfer learning from one compositional task to another: the representation and the task-speciﬁc layers of the models are strategically trained differently on a pre-ﬁnetuning task such that they generalize well on mismatched splits that require compositionality.
We apply this method to semantic parsing, using three very different datasets, COGS, GeoQuery and SCAN, used alternately as the pre-ﬁnetuning and target task.
Our method signiﬁcantly improves compositional generalization over baselines on the test set of the target task, which is held out during ﬁne-tuning.
Ablation studies characterize the utility of the major steps in the proposed algorithm and support our hypothesis
I NTRODUCTION
Recent work has spotlighted signiﬁcant shortcomings of neural network approaches to NLP in coping with compositional generalization (CG) (Lake & Baroni, 2018; Finegan-Dollak et al., 2018;
Keysers et al., 2020; Kim & Linzen, 2020; Shaw et al., 2021).
In those studies, the test set combines in unfamiliar ways linguistic elements that may themselves be familiar from the training set; for example, sentences in the test set may be longer than those observed in training, or may use familiar words in syntactic roles in which they did not occur in training.
The performance of popular neural architectures in contemporary NLP models such as Transformers drops considerably when they are expected to generalize compositionally.
Several approaches have been proposed to address this issue, including specialized architectures with compositional inductive biases (Furrer et al. 2020) and compositional data augmentation (Andreas, 2020), but as of yet the problem is far from solved.
In this paper, we study this challenge from the perspective of transferability of compositional generalization skills: is possible for a neural model to transfer compositional generalization skills acquired from one task to another task that also requires compositional generalization skills?
We ground our inquiries with the semantic parsing tasks on three very different datasets: G eo Q uery (Zelle, 1995), COGS (Kim & Linzen, 2020) and SCAN (Lake & Baroni, 2018).
For each task, we use existing compositional training/test splits or create new ones.
We propose a learning algorithm that can extract a compositional inductive bias from one task — a stage we refer to as pre-ﬁnetuning 1 — and transfers that bias to a target task, improving the models’ compositional generalization behavior on that task.
To extract the inductive bias so as to transfer, we introduce a new training algorithm DUEL .
In summary (cf.
Fig 1), DUEL is designed to be compatible with pre-trained neural encoder-decoder models, such as T5 (Raffel et al., 2020).
We view the encoder as learning a representation for the inputs and the decoder as “a task head” that is specialized to different tasks.
In pre-ﬁnetuing, framing the task’s two (compositional) splits as deriving representation from one split for zero-shot learning to the other split, DUEL trains the encoder and the decoder using different splits.
In contrast to using standard supervised learning over both splits as a whole for pre-ﬁnetuning, DUEL encourages the encoder to learn to represent the input texts in a way that facilitates greater compositional generalization, and that this transfers across domains and persists through ﬁne-tuning on the target task, as shown by our empirical studies.
The rest of the paper is organized as follows.
In §2, we detail our setup.
We describe our approach in §4, followed by empirical studies §5.
We discuss related work in §6 and conclude in §7.
P ROBLEM S ETUP
Our neural learner is given a pre-ﬁnetuning task and a target task.
For the pre-ﬁnetuning task, we have a training data distribution s and a test/eval data distribution ˜ s that may deviate from s .
Likewise, for the target task, we have q and ˜ q .
In the case of compositional generalization, the difference between s and ˜ s (or q versus ˜ q ) is deﬁned in compositional terms: the ways in which elements combine differ systematically between the two parts of each datasets (Keysers et al., 2020; Kim & Linzen, 2020;
Lake & Baroni, 2018; Hupkes et al., 2020).
Our goal is to train a neural model on ( s , ˜ s ) and then ﬁne-tune it on q so that the ﬁnal model performs well on ˜ q , namely, attaining strong compositional generalization performance on the target task.
Our assumption is that the difference between s and ˜ s is similar to the difference between q and ˜ q , such that an inductive bias acquired from ( s , ˜ s ) can be transferred to the target task.
A motivating example is that ˜ s contains longer texts than s and so does ˜ q than q : in this case, we would investigate whether learning to generalize from short to long texts is a transferrable skill.
T ASK , D ATA S ETS AND T HEIR C OMPOSITIONAL S PLITS
Data Splits In this work, we focus on semantic parsing, the task of translating a natural language text into a formal representation of its semantics.
Standard data splits for supervised learning partition a dataset randomly into a training set and testing set.
As such, the examples in both portions are drawn from the same distribution.
By contrast, compositional splits partition a dataset in a way that creates a distributional discrepancy between the training and the test set, such that strong performance on the test set requires compositional generalization from the training set.
Compositional splits are straightforward to create for datasets that are generated from grammars or templates, such as SCAN (Lake & Baroni, 2018) and COGS (Kim & Linzen, 2020), described in detail in the next section.
For instance, the templates used for the test set can include speciﬁc words in syntactic roles they did not appear in in the training set, or they can nest syntactic constituents more deeply than in the training set, creating differences in lengths of the input texts.
An alternative mechanism creates—based on an existing dataset for which the generative process is not available (for example, real-world user queries)—test sets that require compositional generalization by on splitting the dataset so as to obtain the maximum compound divergence (MCD) between the training and test examples (Keysers et al., 2020).
Here, compounds are complex expressions formed by composing atoms, and compound divergence is computed as
[Equation]
where F is the distribution over compounds, and C α ( P (cid:107) Q ) =
(cid:80) k p αk q 1 − α k .
The intuition behind this method is that given an existing dataset it is difﬁcult to limit the test examples to compounds that are completely absent from the training set, but by making the compound distributions as distinct as possible, we are primarily measuring the model’s performance on compounds are at least very infrequent in training.
We use the MCD split generated for SCAN from Keysers et al.
(2020) and the Target Maximum
Compound Divergence (TMCD) splits generated for GeoQuery from Shaw et al.
TMCD splits extend the MCD methodology to create compositional splits of non-synthetic datasets and deﬁne atoms and compounds based on the known syntactic structure of the output logical forms.
Datasets and Tasks Table 1 summarizes the 5 compositional splits we use in this paper.
We also compared with standard splits ( std ) where the training and test come from the same distribution.
[Table]
As an example for our setting in §2, taking COGS as our pre-ﬁnetuning task and G eo Q uery as our target task, we learn from the CG General split of COGS and transfer to the CG General split of
G eo Q uery.
In this example, COGS is the distribution s the distribution our our
COGS (Kim & Linzen, 2020) is a synthetic semantic parsing dataset generated from templates.
The inputs are English sentences and the outputs are corresponding logical forms inspired by λ -calculus, e.g ., A dog ate the cake → *cake ( x 4 ); dog ( x 1 ) AND eat.agent ( x 2 , x 1 ) AND eat.theme ( x 2 , x 4 ) .
COGS has two components: a training dataset, which we refer to as COGS BASE , and a compositional generalization (CG) dataset COGS CG .
Kim & Linzen (2020) show that the performance of neural models trained on COGS BASE degrades signiﬁcantly when they are applied to COGS CG .
COGS CG includes two kinds of CG challenges: lexical and structural.
In lexical CG , familiar words need to be interpreted in new syntactic positions; for example, in COGS BASE the word hedgehog may occur only in the subject position, in COGS CG it would need to be interpreted as as object.
By contrast, structural CG involves new combinations of familiar syntactic structures.
For example, in COGS BASE , prepositional phrases only modify objects ( Noah ate the cake on the plate ), whereas in COGS CG they modify subjects ( The cake on the plate burned ).
Likewise, in COGS BASE prepositional phrases can only be nested once ( Ava saw the ball in the bottle on the table ), but in COGS CG they are nested multiple times ( Ava saw the ball in the bottle on the table on the ﬂoor ), creating longer sentences.
COGS BASE has 24,155 examples and COGS CG has 21,000 examples, of which 18,000 are instances of lexical CG and 3,000 are instances of structural CG.
The G eo Q uery (Zelle, 1995; Tang & Mooney, 2001) is an annotated semantic parsing dataset contains 880 natural language questions about US geography (e.g., What states border Texas? ).
We use the same pre-processing as in Shaw et al.
(2021), replacing entity mentions with placeholders in the Functional Query Language (FunQL; Kate et al.
2005) output representations.
The input what states border m0 , for example, is associated with the output answer(intersection(state, next_to_2(m0))) .
Atoms and compounds are deﬁned over the FunQL expressions.
We adopt the length and TMCD splits from Shaw et al.
For the length split GEO len , the training set GEO SHORT consists of examples with shorter inputs than the test set GEO LONG .
We refer to the training and test set for the TMCD split as GEO TMCD 1 and GEO TMCD 2 .
Both splits divide G eo Q uery equally into two sets of 440 examples.
The SCAN dataset (Lake & Baroni, 2018) consists of over 20,000 natural language navigation commands and corresponding “action sequences”, e.g . jump twice → JUMP JUMP.
SCAN is not a semantic parsing dataset but is commonly used as a diagnostic dataset for evaluating compositional generalization.
We adopt the length split ( SCAN len ) and MCD split ( SCAN cd ) introduced by Keysers et al.
P ROPOSED M ETHOD
Intuition and Main Idea As mentioned in §2, our goal is to improve compositional generalization on our target task, where the training distribution is q and the evaluation distribution ˜ q .
The challenge is to leverage the information in s and ˜ s during the pre-ﬁnetuning stage such that the model, when ﬁne-tuned on q , generalizes better on ˜ q .
This contrasts with a standard supervised learning approach,
[Figure]
which would collapse together the examples from s and ˜ s during pre-ﬁnetuning, discarding the information about how examples are divided between s and ˜ s —information that, we hypothesize, can be useful for encouraging compositional generalization on the target task.
We assume a neural encoder-decoder architecture (Cho et al., 2014).
Our proposed method, DUEL , iteratively updates the parameters of the encoder on examples from ˜ s , while keeping the decoder ﬁxed, and then updates the decoder parameters on examples from s , while keeping the encoder ﬁxed.
The encoder updates are then retained prior to ﬁne-tuning on q , while the decoder updates are discarded (as the pre-ﬁnetuning and the target tasks are different).
In contrast to standard ﬁne-tuning,
DUEL encourages the encoder to learn to represent the input sequences in a way that facilitates compositional generalization, which we hypothesize will transfer across domains and persist through ﬁne-tuning.
Details For encoder-decoder architectures, the model’s parameters naturally partition into two sets, θ for the encoder and φ for the decoder, where θ parametrizes the representation f
( x ; θ ) of the input x and φ parametrizes the “task head” g ( · ) , which uses that representation to accomplish a task such as classiﬁcation or semantic parsing.
Taken together, the model’s output is p ( y | x ; θ, φ ) =
g ( f ( x ; θ ); φ ) .
Given a pre-ﬁnetuning task where both s and ˜ s are given, our goal is to drive the learner to learn the optimal representation and task head parameters ( θ ∗ , φ ∗ ) such that g ( f ( x ; θ ∗ ); φ ∗ ) performs the best on both distributions.
To achieve this, we design an iterative dueling game that the learner plays with the two distributions.
Suppose the learner’s parameters are currently ( θ, φ ) .
To perform optimally on ˜ s , we would like to use the task head g ( · ; φ ) without further training it, i.e ., perform well in an “zero-shot” setting; this setting directly anticipates how learning would proceed for the target task, where we will be training only on q , without exposure to ˜ q .
Since we are holding the task head g ( · ; φ ) ﬁxed, we need to optimize the representation f ( · ; θ ) such that it models ˜ s appropriately.
To achieve this desiderata, we update the representation parameters iteratively for at most T inner steps, according to
[Equation]
where B ˜ s is a batch of N samples from ˜ s and α is the step size.
Updates to φ follow the reverse logic.
The representation function that results from updating θ deﬁnes a prior on how s should be represented.
When updating the task head parameters φ , our goal is to have the task head perform optimally conditioned on this prior.
Concretely, we hold θ ﬁxed and apply iteratively at most T inner steps to φ :
[Equation]
where B s is a batch of samples from s .
Crucially, we alternate between these two types of updates, such that ˜ s informs the learner about the compositional inductive bias that should be incorporated into the representation f , and s teaches the learner how to use that representation to perform the task (here semantic parsing).
Algorithm 1:
Require: Data sets s , ˜ s ; Learning rate α ; Batch size N ; Outer loop iterations T outer ; Inner loop iterations T inner ; Inner loop early stopping criteria T patience ; Outer loop early stopping criteria
T min ;
Initialize model parameters θ, φ i ←while i < T outer do j ←while j < T inner do
Sample a batch B s from s
Compute loss: L ( θ, φ, B s )
=
1 N (cid:80) ( x
,y ) ∈ B s − log p ( y | x ; θ, φ )
Update parameters: φ ← φ − α · ∇ φ L ( θ, φ, B s )
j ← j +if AccuracyDecreases ( θ , φ , ˜ s , T patience )
then
Early stop: break end end k ←while k < T inner do
Sample a batch B ˜ s from ˜ s
Compute loss: L ( θ, φ, B ˜ s )
= 1 N (cid:80) ( x
,y ) ∈
B ˜ s − log p ( y | x ; θ, φ )
Update parameters: θ ← θ − α · ∇ θ L
( θ, φ, B ˜ s )
k ← k +if AccuracyDecreases ( θ , φ , s , T patience ) then
Early stop: break end endif k < T min then
Early stop outer loop: break end i ← i +endReturn: Model parameters θ , φ
We contrast our approach with the standard supervised learning approach, where the two distributions are merged together as s ∪ ˜ s , ignoring the compositional split:
[Equation]
DUEL is used for pre-ﬁnetuning.
When ﬁne-tuning the model on the target task q , we retain the representation component f ( · ; θ ) and re-initialize the task head g ( · ; φ ) .
Both θ and φ are then updated, as is standard, to optimize the loss function on q .
DUEL is illustrated schematically in the pre-ﬁnetuning panel of Fig.
The pseudocode is listed in
Algorithm 1.
The algorithm contains one outer loop for T outer rounds (with possible early stopping), which alternates between inner loops updating θ and φ .
The early-stopping monitoring function
AccuracyDecreases takes as arguments the current model parameters θ and φ , a data distribution to use for evaluation ( s or ˜ s ), and the maximum patience T patience for early-stopping.
This function returns true if accuracy on the dataset in question has not improved for T patience consecutive steps.
To terminate the outer loop (thus, the algorithm), we keep track how many steps the inner update for θ take.
If the number steps is smaller than a preset threshold T min , we conclude that the algorithm has converged to the desired representation since the difference in representing s and ˜ s is small, requiring little adaptation.
We do not use the same logic to limit how long it takes to optimize the task head parameters φ , conditioned on the representation, as the goal is to use the derived representation to arrive at a strong performance.
E XPERIMENTS 5. S ETTINGS
Datasets and Tasks
We use the datasets described in Table 1.
The target tasks always come from one of the compositional splits: COGS cg , GEO cd , SCAN cd , or GEO len .
For the pre-ﬁnetuning tasks, we consider two conﬁgurations: (1) standard splits for COGS , G eo Q uery and SCAN ; (2) compositional splits: COGS cg , GEO cd , SCAN cd , SCAN len .
Models We use two neural architectures, both encoder-decoder models based on the Transformer architecture (Vaswani et al., 2017).
The ﬁrst one, which we refer to as BERT 2 SEQ , uses the BERT encoder BERT SMALL (Devlin et al., 2019), with 4 Transformer layers, to learn the representation f ( x ; θ ) , followed by a Transformer-based sequence-to-sequence model with 2 encoder layers and 2 decoder layers, as the task head g ( · ; φ ) .
3 We initialize f ( · ) with the pre-trained BERT SMALL (Turc et al., 2019).
We also experimented with the much larger T5-Base model (Raffel et al., 2020).
We take the T encoder as f ( x ; θ ) and the T5 decoder as the task head g ( · ; φ ) ; the encoder and encoder have
Transformer layers each.
We initialize them with the pre-trained T5 model.
In both pre-ﬁnetuning and ﬁne-tuning, we generate task prompts by prepending a task tag to each input sentence (following Raffel et al. 2020).
We provide detailed examples of the task prompts for each task in Appendix B.
When using BERT 2 SEQ , our main results are obtained by initializing the task heads randomly.
For T5, we initialize the task heads with the pre-trained T5 decoder, for both pre-ﬁnetuning and ﬁne-tuning.
Baselines We compare DUEL to two baselines: (1) NONE , where we ﬁne-tune on the target task ( i.e ., q ) directly, without pre-ﬁnetuning; (2) MERGED , pre-ﬁnetuning on s ∪ ˜ s using eq.
(3) and then ﬁne-tuning on the target task.
As mentioned above, MERGED ignores the compositional splits and conducts pre-ﬁnetuning using the standard splits (cf.
Table 1).
5. DUEL LEARNS TRANSFERABLE COMPOSITIONAL GENERALIZATION INDUCTIVE BIAS
Main Results Table 2 shows the results of our experiments taking the CG General splits as the target splits.
The metric we report is exact match accuracy over output sequences.
For COGS , we report accuracy separately for the lexical and structural generalization cases, as our models’ performance differed dramatically between those two types of cases.
On SCAN , the NONE baseline (no pre-ﬁnetuning) matches the average accuracy 15.4% reported by Conklin et al.
Pre-ﬁnetuning substantially improves over the baselines, across the board.
In particular, pre-ﬁnetuning on SCAN cd leads to an accuracy of 57.8% on GEO TMCD 2 , surpassing — with a neural-only approach — the previous state-of-the-art of 56.6%, which was established by NQG-T5, a hybrid model that combines a grammar-based approach with T5 (Shaw et al., 2021). The detailed comparison to other existing approaches is provided in Appendix D.
Pre-ﬁnetuning on a compositional split is essential Do the improvements we observe in Tablespeciﬁcally reﬂect the contribution of pre-ﬁnetuning on compositional splits, or can they be attributed to pre-ﬁnetuning more generally?
Table 2 reports the performance of the MERGED baseline, i.e ., pre-ﬁnetuning on the merged dataset s ∪ ˜ s , without a compositional split.
Comparing to the NONE (no pre-ﬁnetuning) rows in Table 2, we observe that pre-ﬁnetuning improves performance on the target tasks even without using DUEL or the compositional split.
However, we see bigger gains from using DUEL on compositional splits ( DUEL rows with CG General category in Table 2), especially for the target tasks GEO cd and SCAN cd .
On COGS cg , the gain of using DUEL on compositional splits is less pronounced.
This reﬂects two facts: ﬁrst, that lexical generalization is not very challenging when the encoder uses a pre-trained model with strong lexical representations, such that performance is close to being saturate even without pre-ﬁnetuning; and second, that the structural generalization cases in COGS remain too difﬁcult for purely neural models, even with DUEL . [Table]
[Table] Within Table 2, it is clear that DUEL does not make a major difference from MERGED when restricted to standard splits .
This is reasonably expected as the DUEL procedure reduces to the MERGED when the data distribution s and ˜ s are exactly the same for the pre-ﬁnetuning task (cf. eq.(3) to eq.
In summary, the results so far suggest that DUEL is most effective when used in conjunction with pre-ﬁnetuning tasks with compositional splits.
5. W HEN WILL DUEL WORK BEST ?
We hypothesize DUEL works best when the pre-ﬁnetuning tasks and the target tasks share strong similarity in their compositional splits.
We gain further insights about how effective DUEL is by restricting our study to a special type of compositional generalization - the CG Length generalization.
Table 3 reports the results when the compsitional split in the target task GEO len is to generalize from shorter texts to longer ones.
Without pre-ﬁnetuning, the accuracy on GEO len is 38.6% with T5-Base, and 16.1% with BERT 2 SEQ .
The state-of-the-art results on this task is 52.2% from the NQG-T model (Shaw et al., 2021).
As before, pre-ﬁnetuning improves from that baseline.
SCAN len with DUEL improves the most for both neural models.
Intuitively, this is sensible as SCAN SHORT also consists of examples with shorter texts than those in the test set SCAN LONG : on average, the input lengths are 7.0 versus 8.2 on SCAN SHORT and SCAN LONG respectively, while the output lengths signiﬁcantly vary from 10.8 to 30.0.
In this case,
DUEL helps extract the inductive bias for generalizing on length difference.
Our experiments also show that COGS cg works better than SCAN cd .
The compositional splits in
COGS cg are multiple typed and not all about length difference.
Nonetheless, in the structural challenge portion of the COGS cg , the average text length is 61 in COGS CG , compared to 22 in COGS BASE .
On the other hand, SCAN MCD 1 and SCAN MCD 2 have average text lengths of 7.0 and 7.7, respectively.
The
SCAN std , which is the standard split, does not improve as it is not a compositional split. [Table]
Also, pre-ﬁnetuning on two tasks shows the best performance, which implies the potential of our method to learn a general compositional generalization strategy with multiple pre-ﬁnetuning tasks.
5. H OW MUCH CAN WE IMPROVE PERFORMANCE OF COGS BY GENERATING VERY SIMILAR PRE - FINETUNING TASKS ?
While DUEL performed well on the SCAN and G eo Q uery target tasks, performance on COGS was less impressive, especially on the structural generalization cases (Table 2).
It has recently been shown that the challenges presented by COGS can be addressed using symbolic methods, albeit ones that are informed by detailed knowledge of the generative process that was used to create the dataset (Liu et al., 2021).
Can we make additional progress on this dataset using a generic sequence-to-sequence neural model?
We hypothesized that the pre-ﬁnetuning splits we used before ﬁne-tuning on COGS were insufﬁciently similar to the split required for COGS (unlike, for example, the SCAN length split that correspond to the generalization behavior required for GEO len ).
To test this hypothesis, as a lower bound on DUEL ’s ability to extract inductive bias for compositional generalization, we designed a new experiment such that the pre-ﬁnetuning tasks will have the same compositional splits as the COGS .
Concretely, we created ten COGS variants with the same type of splits as COGS cg ; the only difference between the variants was that each of them used a different lexicon.
To create these variants, we ﬁrst used the SpaCy Python library to identify 3 part-of-speech classes: PROPN , NOUN and VERB .
For each proper noun, we selected 5 different alternatives.
For each noun and verb, we selected all the synonyms and antonyms from WordNet that had the same part-of-speech.
To create a single COGS variant, for each word in the COGS vocabulary, we built a 1-to-1 replacement mapping by randomly selecting one alternative, and generated new sentences by the mapping.
The following pair illustrates the mapping with an example (and more in Appendix C).:
The lexicons of the variants and the original COGS were completely disjoint in those part-of-speech classes; other words (in particular function words were the same across all variants.
We use
COGS _ VAR cg to denote the CG split of a single COGS variant.
For pre-ﬁnetuning, we used 1, 5 or 10 variants.
The results reported in Table 4 suggest that pre-ﬁnetuning with the same type of compositional generalization split only leads to a very minor improvement in accuracy, from 4.5% to 5.2%.
Note that the pre-ﬁnetuning tasks have an average accuracy of 79.3% on the structural generalization portions of those tasks.
Our method is thus surprisingly ineffective in this case, even when the pre-ﬁnetuning task is very similar in structure to the target task.
For future research, we plan to investigate the reasons of the limited improvement and approaches for extracting stronger inductive bias.
R ELATED W ORK
Transfer Learning A large body of work has attemped to leveraged multi-task learning to endow a model with an inductive bias that improves generalization on a main task of interest (Caruana, 1998; Bakker & Heskes, 2003; Raffel et al., 2020), with recent work in NLP sharing our focus on neural networks (Sogaard & Goldberg 2016; Hashimoto et al. 2016; Swayamdipta et al.
2018; for a review, see Ruder 2017).
Intermediate training of pre-trained sentence encoders on a task or a set of tasks that are related to the task of interest has been advocated, among others, by Phang et al.
(2018) and Aghajanyan et al.
Gururangan et al.
(2020) craft a training pipeline where a pre-trained language model is adapted to domain-speciﬁc and task-speciﬁc ones.
There are also empirical studies of transferrability among tasks (Vu et al., 2020).
Our work differs from most of the work in this area in that our goal is to learn to generalize compositionally from a training distribution to a test one; as such we provide the model with a strategic training and test set split drawn from a pre-ﬁnetuning task, which together illustrate the desired generalization behavior that we intend for the model to transfer to a target task.
This approach bears some resemblance to work on learning-to-learn or meta-learning (Thrun & Pratt, 1998; Gu et al., 2018), and a recent study on learning low-resource tasks (Chen et al., 2020a).
Compositional Generalization
A number of approaches have been explored to achieve compositional generalization in neural models for NLP.
Speciﬁcally, recent work has proposed new or modiﬁed model architectures (Li et al., 2019; Russin et al., 2019; Gordon et al., 2020; Liu et al., 2020; Nye et al., 2020; Chen et al., 2020b; Zheng & Lapata, 2020; Oren et al., 2020; Herzig & Berant, 2020;
Shaw et al., 2021;
Yin et al., 2021).
Furrer et al.
compared several of these architectures with general-purpose pre-trained models such as T5.
Other work has explored intermediate representations (Herzig et al., 2021) and compositional data augmentation procedures (Andreas, 2020;
Akyürek et al., 2020).
Our approach is most closely related to work that applies meta-learning to compositional generalization (Conklin et al., 2021;
Oren et al., 2021; Lake, 2019).
Conklin et al.
(2021) focus on using tasks that are similar to the target task to regularize the learning of neural models.
By systematically generating compositional splits on the synthetic dataset SCAN, Oren et al.
shows that the success of meta-learning depends on the speciﬁc splits that are presented to the learner (Oren et al., 2021).
Our work complements these studies by showing that it is possible to transfer an inductive bias learned on one task to a substantially different task.
C ONCLUSION
In this paper, we studied acquiring compositional generalization skills through a pre-ﬁnetuning task, where a compositional split is available, and transferring those skills to a target task where such a compositional split does not exist.
We proposed DUEL , a simple method for this purpose.
The underlying intuition of this method is that we would like to learn an input representation that incorporates a compositional inductive bias, and train it separately from a task head that performs a speciﬁc task.
We demonstrated the effectiveness of this method on 3 semantic parsing tasks.
In particular, using the COGS and SCAN synthetic tasks as pre-ﬁnetuning tasks, we were able to achieve high compositional generalization accuracy on two splits of the non-synthetic G eo Q uery dataset: the
Target Maximum Compound Divergence split GEO cd and the length split GEO len .
For GEO cd , our approach beats the state-of-the-art, established by a hybrid neurosymbolic model (Shaw et al., 2021); for GEO len , we substantially improved over the baselines.
Reproducibility Statement To ensure reproducibility, we describe the datasets and their train/test splits for our experiments in Section 3 and Section 5.1.
We explain our algorithm with pseudocode in
Section 4 Algorithm 1.
We also describe the model architecture and baselines in Section 5.1.
Further, we provide detailed hyper-parameters and hyper-parameter tuning methods in Appendix A.
We give examples of preprocessed inputs to T5 for each dataset in
Appendix B. We will release the code after the anonymity period at https://www.anonymous.com .
Appendix
In this supplementary material, we provide details omitted in the main text.
The content is organized as what follows:
• Section A. Implementation details and optimization hyper-parameters.
• Section B. Examples of T5 prompt for each of the task in our experiments.
• Section C. Examples of COGS _ VAR vs .
COGS .
• Section D. Additional experimental results and ablation studies of DUEL .
– Comparison with the existing works – A hypothesis on what learned by DUEL transfers between datasets – Ablations on the design of DUEL – The standard deviation of DUEL
A I MPLEMENTATION DETAILS A. H YPER - PARAMETERS IN FINE - TUNING
First, we list the hyper-parameters in ﬁne-tuning for each model and task.
For COGS , we ﬁne-tune for a maximum of 30k steps with a learning rate of 1 e − 4 .
Following Kim &
Linzen (2020), we evaluate every 500 iterations on the dev set of COGS and early-stop if the accuracy on the dev set has not improved for 5 consecutive step.
The batch size is 128 on T5-base and 512 on
BERT 2 SEQ .
For G eo Q uery, it has no dev set.
Therefore, we tune hyper-parameters on the standard split of the
G eo Q uery dataset.
We then use the same hyper-parameters for the GEO cd and GEO len splits.
On
T5-base, we ﬁne-tune for 5k steps with a batch size of 256 and a learning rate of 1 e − 4 .
On BERT 2 SEQ , we ﬁne-tune for 3k steps with a batch size of 512 and a learning rate of 2 e − 5 .
For SCAN , likewise, we tune hyper-parameters on the standard split of it.
We then use the same hyper-parameters for the SCAN cd and SCAN len splits.
On T5-base, we ﬁne-tune on for 3.5k steps with a batch size of 128 and a learning rate of 1 e − 4 .
On BERT 2 SEQ , we ﬁne-tune for 2.5k steps with a batch size of 512 and a learning rate of 1 e − 4 .
We use the Adam optimizer with weight decay.
We take the label smoothing (Müller et al., 2019) with weight factor 0.1 on BERT 2 SEQ but not T5-base.
Next, we provide the computation time for ﬁne-tuning on 1 TITAN Xp GPU.
For COGS , the average ﬁne-tuning time is about 24 hours on T5-base and 4 hours on BERT 2 SEQ .
For G eo Q uery, abouthours on T5-base and 0.4 hour on BERT 2 SEQ .
For SCAN , about 3 hours on T5-base and 0.5 hours on
BERT 2 SEQ .
A. V ALIDATION DATASET FOR HYPER - PARAMETER TUNING
As we train on the whole dataset with DUEL during pre-ﬁnetuning, we are not able to directly tune hyper-parameters for DUEL .
In order to validate the learned representation is good at OOD generalization from s to ˜ s without accessing the test data in ˜ s , we hold out a validation dataset for each task from ˜ s .
We then take the pre-ﬁnetuned model, re-initialize the task-head, ﬁne-tune it on s and test on the hold-out set of ˜ s .
We select the hyper-parameters of DUEL based on the test accuracy on the hold-out set.
For COGS , we create a validation dataset by holding a subset of 3000 examples from COGS CG .
Likewise, the validation datasets of G eo Q uery and SCAN hold out 20% of the data from GEO TMCDand SCAN MCD 2 , respectively.
A. H YPER - PARAMETERS IN PRE - FINETUNING
In pre-ﬁnetuning, we choose the maximum outer loop iterations T outer as 10, the maximum inner loop iterations T inner as 30k.
For efﬁciency, we evaluate every 500 iterations and set the early-stopping patience for the inner loop as 1.
We select a learning rate α from [ 1 e − 5 , 2 e − 5 , 1 e − 4 ] for all tasks and all models.
The batch size is different from each model.
For T5-base, we select a batch size of 32 from [8, 16, 32] for all tasks.
For BERT 2 SEQ , we select a batch size of 512 from [256, 512, 1024].
The early-stopping minimum iterations T min for the outer loop is different for each model and task.
For COGS , T min = 3000 on T5-base and T min = 2000 on BERT 2 SEQ .
For G eo Q uery, T min = 1000 on both T5-base and BERT 2 SEQ .
For SCAN , T min = 2000 on T5-base and T min = 1000 on BERT 2 SEQ .
We use the same optimizer and label smoothing method in ﬁne-tuning as in pre-ﬁnetuning.
Next, we provide the computation time for DUEL on 1 TITAN Xp GPU.
When pre-ﬁnetuning on
COGS , the average running time is about 60 hours on T5-base and 24 hours on BERT 2 SEQ .
For
G eo Q uery, about 24 hours on T5-base and 2 hours on BERT 2 SEQ .
And for SCAN , about 30 hours on
T5-base and 3 hours on BERT 2 SEQ .
B T5 PROMPT DETAILS
In this section, we provide examples of T5 prompt for each task in our experiment.
Our design of task prompt follows the standard way from Raffel et al.
B . COGS
Original input: Emma ate the ring beside a bed .
Prompt input: cogs:
Emma ate the ring beside a bed .
B
_ VAR
Original input: The tiddler appreciated that Kim measure that a cooky was slid .
Prompt input: cogs:
The tiddler appreciated that Kim measure that a cooky was slid .
B. GEOQUERY
Original input: what is the largest state capital in population
Prompt input: geo: what is the largest state capital in population
B
SCAN
Original input: run around right after look opposite left
Prompt input: scan: run around right after look opposite left
C M ORE EXAMPLES ON COGS _ VAR VS COGS
For each COGS example listed following, we give three examples from the corresponding COGS _ VAR .
COGS input sentence 1: The sailor dusted a boy .
COGS output program 1: *sailor(x_1); dust.agent(x_2, x_1) AND dust. theme(x_2, x_4) AND boy(x_4) [Table]
The crewman scattered a female_child .
The boater dotted a male_child .
The leghorn dotted a son .
[Table]
*crewman(x_1); scatter.agent(x_2, x_1) AND scatter.theme(x_2, x_4) AND female_child(x_4) *boater(x_1); dot.agent(x_2, x_1) AND dot.theme(x_2, x_4) AND male_child(x_4) *leghorn(x_1) ; dot.agent(x_2, x_1) AND dot.theme(x_2, x_4) AND son(x_4)
[Table]
The cookie was passed to Emma . [Table] [Table] .theme
[Table]
x_3, x_1)
[Table]
[Table]
The cooky was faded to Joan
The cooky was overstepped to Trudy
The cooky was eliminated to Kim
[Table]
*cooky(x_1); fade.theme(x_3, x_1) AND fade.recipient(x_3, Joan) *cooky(x_1); overstep.theme(x_3, x_1) AND overstep.recipient(x_3, Trudy) *cooky(x_1); eliminate.theme(x_3, x_1) AND eliminate.recipient(x_3, Kim) [Table] thought that a hippo cleaned
[Table]
think
.agent
(x_1,
Zoe AND think ccomp
x_
x
AND hippo(x_4) AND clean.agent(x_5, x_4)
[Table]
Larry cerebrated that a Hippo housecleaned .
Dana guessed that a hippopotamus picked.
Ronnie retrieved that a hippopotamus make_cleaned .
COGS output programs 3: cerebrate. Larry AND cerebrate.ccomp ) AND
Hippo houseclean guess hippopotamus pick.
retrieve agent
Ronnie)
.ccomp hippopotamus(x
) AND make_clean.
D A DDITIONAL EXPERIMENTAL RESULTS
In this section
[Table]
we provide the experimental results as well as the standard deviation of DUEL omitted in the main text.
[Table]
D. C OMPARISON WITH THE EXISTING WORKS
Several existing works leverage the speciﬁc designs of the datasets to solve SCAN (Andreas, 2020;
Li et al., 2019; Russin et al., 2019), most of the improvements do not transfer to non-synthetic tasks. We compare to two methods that solve SCAN but can also transfer to non-synthetic tasks, SpanBasedSP (Herzig & Berant, 2020) and NQG-T5 (Shaw et al., 2021). Also, we compare to a recently proposed model-agnostic meta-learning approach (Conklin et al., 2021)
[Table]
We use the original released code to run the experiments for SBSP (SpanBasedSP) and MAML.
For SBSP, we use the BERT-BASE model as in their work; For MAML, we replace their 2-layer
Transformer encoder and 2-layer Transformer decoder with the larger model T5-Base EncoderDecoder for a fair comparison.
DUEL outperforms MAML on SCAN cd and COGS cg , and outperforms SpanBasedSP substantially on
GEO cd .
As expected, it falls behind SpanBasedSP and NQG-T5 on SCAN cd . [Table] D. W HAT TRANSFERS BETWEEN DATASETS ? The ability to transfer across datasets is related to what exactly the model is learning by DUEL during the pre-ﬁnetuning stage.
Our hypothesis is that DUEL might restrict the encoder to learn to map structures compositionally, bits and pieces, instead of overﬁtting on arbitrary complex “structural noises”.
In the light of this hypothesis, the DUEL pre-ﬁnetuning regularizes the encoder to represent structural information in a more decomposable way, instead of “contextually interdependent”.
Suppose the input sentence is “jump right after walk right”.
Ideally, we would want the encoder to pick up the cue that “after” signiﬁes to represent “jump right” and “walk right” separately. However, for an end-to-end model, the representation could be biased to couple these two shorter phrases together to map to the desired output sequence. We have not come up with a deﬁnitive way to test the theory, instead, we added some indirect evidences to support the theory. We compare the input length of correctly predicted examples in GEO len between training with DUEL and MERGED in Table 6.
The results indicate that the improvement mostly comes from the longer inputs, which means the encoder is being regularized on the abstract notion of learning better from shorter structures and supports our hypothesis.
[Table]
D. A BLATIONS ON THE DESIGN OF DUEL
We evaluate on two ablations of DUEL : (1) DUEL ( F ), where we keep the encoder ﬁxed in ﬁne-tuning; (2) DUEL ( S ), where we switch the train and the test component in pre-ﬁnetuning.
Results are listed in
Table 7.
DUEL ( F ) shows a large performance drop comparing with DUEL , which implies it is beneﬁcial to ﬁnetune the encoder together with the decoder.
DUEL ( S ) shows minor degradation to DUEL when pre-ﬁnetuning on SCAN , likely due to its symmetric
MCD split.
Similarly, the COGS lexical generalization set has the same average input length as the train set and contains all the vocabs.
Thus, training set is largely symmetric with its lexical generalization set, which leads to minor change in DUEL ( S ).
However, DUEL ( S ) has a large gap with DUEL when pre-ﬁnetuning on G eo Q uery.
As the TMCD split only ensures all atoms appear in the train component GEO TMCD 1 , we ﬁnd it has about 30% of examples containing words that do not appear in the test component GEO TMCD 2 .
Thus, using GEO TMCD 2 to train the decoder and GEO TMCD 1 to train the encoder forces the encoder to model those “new” atoms.
This likely causes the encoder not learning enough inductive bias, thus incurring a larger drop in performance.
D. T HE STANDARD DEVIATION OF DUEL
In Tables 2 and 3 we report the mean of 3 runs for accuracy of the target tasks when pre-ﬁnetuning with DUEL .
The standard deviations for these runs are reported in Table 8 and 9, respectively.
[Table]
[Table]