L EARNING TO R ECOMBINE AND R ESAMPLE D ATA FOR C OMPOSITIONAL G ENERALIZATION
Akyürek akyurek@bu.edu Jacob Andreas
MIT CSAIL jda@mit.edu
A BSTRACT
Flexible neural sequence models outperform grammar- and automaton-based counterparts on a variety of tasks.
However, neural models perform poorly in settings requiring compositional generalization beyond the training data—particularly to rare or unseen subsequences.
Past work has found symbolic scaffolding (e g. grammars or automata) essential in these settings.
We describe R&R, a learned data augmentation scheme that enables a large category of compositional generalizations without appeal to latent symbolic structure.
R&R has two components: recombination of original training examples via a prototype-based generative model and resampling of generated examples to encourage extrapolation.
Training an ordinary neural sequence model on a dataset augmented with recombined and resampled examples signiﬁcantly improves generalization in two language processing problems—instruction following ( SCAN ) and morphological analysis ( SIGMORPHON 2018)—where R&R enables learning of new constructions and tenses from as few as eight initial examples
I NTRODUCTION
[Figure]
How can we build machine learning models with the ability to learn new concepts in context from little data?
Human language learners acquire new word meanings from a single exposure (Carey & Bartlett, 1978), and immediately incorporate words and the concepts they represent productively and compositionally into a larger linguistic or conceptual system (Berko, 1958; Piantadosi & Aslin, 2016).
Despite the remarkable success of neural network models on many learning problems in recent years—including one-shot learning of classiﬁers and policies (Santoro et al., 2016; Wang et al., 2016)—this kind of few-shot learning of composable concepts remains beyond the reach of standard neural models in both diagnostic and naturalistic settings (Lake & Baroni, 2018; Bahdanau et al., 2019a).
Consider the few-shot morphology learning problem shown in Fig.
1, in which a learner must predict various linguistic features (e.g. 3 rd person, S in G ular, PRES ent tense) from word forms, with only a small number of examples of the PAST tense in the training set.
Neural sequenceto-sequence models (e.g. Bahdanau et al., 2015) trained on this kind of imbalanced data fail to predict past-tense tags on held-out inputs of any kind (Section 5).
Previous attempts to address this and related shortcomings in neural models have focused on explicitly encouraging rule-like behavior, e.g. by modeling data with a symbolic system (Jia & Liang, 2016; Xiao et al., 2016;
Cai et al., 2017) or applying rule-based data augmentation (Andreas, 2020).
These procedures are either highly task-speciﬁc or make speciﬁc generative assumptions about sequence data, preventing them from generalizing well to less structured problems involving a mix of rule-like and exceptional behavior.
More fundamentally, they fail to answer the question of whether explicit rules are necessary for compositional inductive bias, and whether it is possible to achieve human-like compositional inductive generalization without appeal to an underlying symbolic generative process.
This paper describes a data augmentation procedure based on two weaker principles, recombination and resampling , and ﬁnds that it is sufﬁcient to induce many of the compositional generalizations studied in previous work.
Concretely: (1) We introduce a new multi-prototype generative model that synthesizes new input–output pairs by learning to copy fragments of reference training examples (recombination).
(2) We show that this model, sampled in low-density regions of the training data (resampling; Chawla et al., 2002), can be used to bootstrap an ordinary sequence-to-sequence model that achieves good performance in-distribution and generalizes compositionally beyond it (Fig. 1a).
Learned recombination and resampling can be combined into a procedure (abbreviated R&R) that promotes efﬁcient generalization in both challenging synthetic sequence modeling tasks (Lake & Baroni, 2018) and morphological analysis in multiple natural languages (Cotterell et al., 2018).
Neither recombination nor resampling require explicit representation of symbolic structure.
Together, they provide performance comparable to or better than state-of-the-art neuro-symbolic approaches.
Our results suggest that some failures of systematicity in neural models can be explained by simpler structural constraints on data distributions and corrected with weaker inductive bias than previously described.
We implemented our experiments in Knet (Yuret, 2016) using Julia (Bezanson et al., 2017), and code for all experiments is released in a public repository.
B ACKGROUND AND RELATED WORK
Compositional generalization Systematic compositionality—the capacity to identify rule-like regularities from limited data and generalize these rules to novel situations—is an essential feature of human reasoning (Fodor et al., 1988).
While details vary, a common feature of existing attempts to formalize systematicity in sequence modeling problems (e.g. Gordon et al., 2020) is the intuition that learners should make accurate predictions in situations featuring novel combinations of previously observed input or output subsequences.
For example, learners should generalize from actions seen in isolation to more complex commands involving those actions (Lake et al., 2019), and from relations of the form r(a,b) to r(b,a) (Keysers et al., 2020; Bahdanau et al., 2019b).
In machine learning, previous studies have found that standard neural architectures fail to generalize systematically even when they achieve high in-distribution accuracy in a variety of settings (Lake & Baroni, 2018;
Bastings et al., 2018; Johnson et al., 2017).
Data resampling and augmentation Learning to predict sequential outputs with rare or novel subsequences is related to the widely studied problem of class imbalance in classiﬁcation problems.
There, undersampling of the majority class or oversampling of the minority class has been found to improve the quality of predictions for rare phenomena (Japkowicz et al., 2000).
This can be combined with targeted data augmentation with synthetic examples of the minority class (Chawla et al., 2002).
Generically, given a training dataset D , learning with class resampling and data augmentation involves deﬁning an augmentation distribution ˜ p ( x, y | D ) and sample weighting function u ( x, y ) and maximizing a training objective of the form:
[Equation]
In addition to task-speciﬁc model architectures (Andreas et al., 2016; Russin et al., 2019), recent years have seen a renewed interest in data augmentation as a ﬂexible and model-agnostic tool for encouraging controlled generalization (Ratner et al., 2017).
Existing proposals for sequence models are mainly rule-based—in sequence modeling problems, specifying a synchronous context-free grammar (Jia & Liang, 2016) or string rewriting system (Andreas, 2020) to generate new examples.
Rule-based data augmentation schemes that recombine multiple training examples have been proposed for image classiﬁcation (Inoue, 2018) and machine translation (Fadaee et al., 2017).
While rulebased data augmentation is highly effective in structured problems featuring crisp correspondences between inputs and outputs, the effectiveness of such approaches involving more complicated, context-dependent relationships between inputs and outputs has not been well-studied.
Learned data augmentation What might compositional data augmentation look like without rules as a source of inductive bias?
As Fig.
1 suggests, an ideal data augmentation procedure ( ˜ p in Eq. 1) should automatically identify valid ways of transforming and combining examples, without pre-committing to a ﬁxed set of transformations.
2 A promising starting point is provided by prototype-based models, a number of which (Gu et al., 2018; Guu et al., 2018; Khandelwal et al., 2020) have been recently proposed for sequence modeling.
Such models generate a distribution p ( d ) via:
[Equation]
) where d (cid:48)
[Equation]
( D [Equation] for a dataset D and a learned sequence rewriting model p rewrite ( d | d (cid:48) ; θ ) .
(To avoid confusion, we will use the symbol d to denote a datum .
Because a data augmentation procedure must produce complete input–output examples, each d is an ( x, y ) pair for the conditional tasks evaluated in this paper.)
While recent variants implement p rewrite with neural networks, these models are closely related to classical kernel density estimators (Rosenblatt, 1956).
Existing work uses prototype-based models as replacements for standard sequence models.
We will show here that they are even better suited to use as data augmentation procedures—they can produce high-precision examples in the neighborhood of existing training data, then be used to bootstrap simpler predictors that extrapolate more effectively.
But our experiments will also show that existing prototype-based models give mixed results on challenging generalizations of the kind depicted in Fig.
1 when used for either direct prediction or data augmentation—performing well in some settings but barely above baseline in others.
Accordingly, R&R is built on two modeling innovations that transform prototype-based language models into an effective learned data augmentation scheme.
Section 3 describes an implementation of p rewrite that encourages greater sample diversity and well-formedness via a multi-prototype copying mechanism.
Section 4 describes heuristics for sampling prototypes d (cid:48) and model outputs d to focus data augmentation on the most informative examples.
Section 5 investigates the empirical performance of both components of the approach, ﬁnding that they together provide they a simple but surprisingly effective tool for enabling compositional generalization.
P ROTOTYPE - BASED SEQUENCE MODELS FOR DATA RECOMBINATION
We begin with a brief review of existing prototype-based sequence models.
Our presentation mostly follows the retrieve-and-edit approach of Guu et al.
(2018), but versions of the approach in this paper could also be built on retrieval-based models implemented with memory networks (Miller et al., 2016;
Gu et al., 2018) or transformers (Khandelwal et al., 2020; Guu et al., 2020).
The generative process described in Eq.
2 implies a marginal sequence probability:
[Equation]
Maximizing this quantity over the training set with respect to θ will encourage p rewrite to act as a model of valid data transformations : To be assigned high probability, every training example must be explained by at least one other example and a parametric rewriting operation.
(The trivial solution where p θ is the identity function, with p θ ( d | d (cid:48) =
d ) = 1 , can be ruled out manually in the design of p θ .)
When D is large, the sum in Eq.
3 is too large to enumerate exhaustively when computing the marginal likelihood.
Instead, we can optimize a lower bound by restricting the sum to an a in the wrong position. In contrast, recurrent neural networks can accurately model a n b n (Weiss et al., 2018; Gers & Schmidhuber, 2001). Of course, this language can also be generated using even more constrained procedures than GECA , but in general learned sequence models can capture a broader set of both formal regularities and exceptions compared to rule-based procedures. a neighborhood N ( d ) ⊂ D of training examples around each d :
[Equation]
The choice of N is discussed in more detail in Section 4.
Now observe that:
[Equation]
where the second step uses Jensen’s inequality.
If all |N ( d ) | are the same size, maximizing this lower bound on log-likelihood is equivalent to simply maximizing
[Equation]
over D —this is the ordinary conditional likelihood for a string transducer (Ristad & Yianilos, 1998) or sequence-to-sequence model (Sutskever et al., 2014) with examples d, d [-(cid:
-](
cid:48) ∈
N ( d ) .
We have motivated prototype-based models by arguing that p rewrite learns a model of transformations licensed by the training data.
However, when generalization involves complex compositions, we will show that neither a basic RNN implementation of p rewrite or a single prototype is enough; we must provide the learned rewriting model with a larger inventory of parts and encourage reuse of those parts as faithfully as possible.
This motivates the two improvements on the prototype-based modeling framework described in the remainder of this section: generalization to multiple prototypes (Section 3.1) and a new rewriting model (Section 3.2).
3. n - PROTOTYPE MODELS
To improve compositionality in prototype-based models, we equip them with the ability to condition on multiple examples simultaneously.
We extend the basic prototype-based language model to n prototypes, which we now refer to as a recombination model p recomb :
[Equation]
Rather than making p Ω ( d (cid:48) 1: n ) uniform on D n , we assume access to a set of compatible prototypes Ω ⊂
D n (Section 4) and let p Ω def = Unif (Ω) .
We update Eq.
6 to feature a corresponding multi-prototype neighborhood
N : D → Ω .
The only terms that have changed are the conditioning variable and the constant term, and it is again sufﬁcient to choose θ to optimize (cid:80) d (cid:48) [Equation] n ∈N ( d ) log p recomb ( d | d (cid:48) 1: n ) over
D .
However, as described next, effective implementation of p recomb requires fundamental changes to the process by which data are generated from prototypes.
3. R ECOMBINATION NETWORKS
Past work has found that latent-variable neural sequence models often ignore the latent variable and attempt to directly model sequence marginals (Bowman et al., 2016).
When an ordinary sequence-tosequence model with attention is used to implement p recomb , even in the one-prototype case, generated sentences often have little overlap with their prototypes (Weston et al., 2018).
We describe a speciﬁc model architecture for p recomb that does not function as a generic noise model, and in which outputs are primarily generated via explicit reuse of fragments of multiple prototypes, by facilitating copying from independent streams containing prototypes and previously generated input tokens.
[Equation]
As discussed in Section 5, the use of a continuous latent variable appears to make no difference in prediction performance for the tasks in this paper.
The remainder of our presentation focuses on the simpler model in Eq.
7. turn left and walk ▷
LTURN
WALK jump
▷
JUMP turn left and second prototype first prototype generation p write p copy, protop copy, out jump j u m p
j u m p
t
u r n ...
h out ▷ (a) (b) p copy, proto- ...
w a l k and l e ft
We take p recomb ( d | d (cid:48) 1: n ; θ ) to be a neural (multi-)sequence-to-sequence model (c.f.
Sutskever et al., 2014) which decomposes probability autoregressively: p recomb ( d | d (cid:48) 1: n ; θ ) =
(cid:81) t p ( d t | d <t , d (cid:48) 1: n ; θ ) .
As shown in Fig.
2, three LSTM encoders—two for the prototypes and one for the input preﬁx— compute sequences of token representations h proto and h out respectively.
Given the current decoder hidden state h tout , the model ﬁrst attends to both prototype and output tokens:
[Equation]
i < t (10)
[Equation]
To enable copying from each sequence, we project attention weights α out and α k proto onto the output vocabulary to produce a sparse vector of probabilities:
[Equation] [Equation]
Unlike rule-based data recombination procedures, however, p recomb is not required to copy from the prototypes, and can predict output tokens directly using values retrieved by the attention mechanism:
[Equation]
To produce a ﬁnal distribution over output tokens at time t , we combine predictions from each stream:
[Equation]
This copy mechanism is similar to the one proposed by Merity et al.
and See et al.
We compare 1- and 2-prototype models to an ordinary sequence model and baselines in Section 5.
S AMPLING SCHEMES
The models above provide generic procedures for generating well-formed combinations of training data, but do nothing to ensure that the generated samples are of a kind useful for compositional generalization.
While the training objective in Eq.
7 encourages the learned p ( d ) to lie close to the training data, an effective data augmentation procedure should intuitively provide novel examples of rare phenomena.
To generate augmented training data, we combine the generative models of
Section 3 with a simple sampling procedure that upweights useful examples.
4. R ESAMPLING AUGMENTED DATA
In classiﬁcation problems with imbalanced classes, a common strategy for improving accuracy on the rare class is to resample so that the rare class is better represented in training data (Japkowicz et al., 2000).
When constructing an augmented dataset using the models described above, we apply a simple rejection sampling scheme.
In Eq.
1, we set:
[Equation]
Here p ( d t ) is the marginal probability that the token d t appears in any example and (cid:15) is a hyperparameter.
The ﬁnal model is then trained using Eq. 1, retaining those augmented samples for which u ( d ) = 1 .
For extremely imbalanced problems, like the ones considered in Section 5, this weighting scheme effectively functions as a rare tag constraint : only examples containing rare words or tags are used to augment the original training data.
4. N EIGHBORHOODS AND PROTOTYPE PRIORS
How can we ensure that the data augmentation procedure generates any samples with positive weight in Eq.
The prototype-based models described in Section 3 offer an additional means of control over the generated data.
Aside from the implementation of p recomb , the main factors governing the behavior of the model are the choice of neighborhood function N ( d ) and, for n ≥ 2 , the set of prior compatible prototypes Ω .
Deﬁning these so that rare tags also preferentially appear in prototypes helps ensure that the generated samples contribute to generalization.
Let d 1 and d 2 be prototypes.
As a notational convenience, given two sequences d 1 , d 2 , let d
1 \ d 2 the set of tokens in d 1 but not d 2 , and d 1 ∆ d 2 denote the set of tokens not common to d 1 and d 2 .
1-prototype neighborhoods Guu et al.
(2018) deﬁne a one-prototype N based on a Jaccard distance threshold (Jaccard, 1901).
For experiments with one-prototype models we employ a similar strategy, choosing an initial neighborhood of candidates such that
[Equation]
(19) where lev is string edit distance (Levenshtein, 1966) and α , β and δ are hyperparameters (discussed in Appendix B).
2-prototype neighborhoods The n ≥ 2 prototype case requires a more complex neighborhood function—intuitively, for an input d , we want each ( d 1 , d 2 , . . . )
in the neighborhood to collectively contain enough information to reconstruct d .
Future work might treat the neighborhood function itself as latent, allowing the model to identify groups of prototypes that make d probable; here, as in existing one-prototype models, we provide heuristic implementations for the n = 2 case.
Long–short recombination : For each ( d 1 , d 2 ) ∈ N ( d ) , d 1 is chosen to be similar to d , and d 2 is chosen to be similar to the difference between d and d 1 .
(The neighborhood is so named because one of the prototypes will generally have fewer tokens than the other one.)
[Equation]
Here [ d \ d 1 ] is the sequence obtained by removing all tokens in d 1 from d .
Recall that we have deﬁned p Ω ( d 1: n ) def = Unif (Ω) for a set Ω of “compatible” prototypes.
For experiments using long–short combination, all prototypes are treated as compatible; that is, Ω = D × D .
Long–long recombination : N ( d ) contains pairs of prototypes that are individually similar to d and collectively contain all the tokens needed to reconstruct d :
[Equation]
For experiments using long–long recombination, we take Ω = { ( d 1 , d 2 ) ∈
D
× D : | d 1 ∆ d 2 | = 1 } .
D ATASETS & E XPERIMENTS
We evaluate R&R on two tests of compositional generalization: the SCAN instruction following task (Lake & Baroni, 2018) and a few-shot morphology learning task derived from the SIGMORPHONdataset (Cotterell et al., 2018).
Our experiments are designed to explore the effectiveness of learned data recombination procedures in controlled and natural settings.
Both tasks involve conditional sequence prediction: while preceding sections have discussed augmentation procedures that produce data points d
= ( x, y ) , learners are evaluated on their ability to predict an output y from an input x : actions y given instructions x , or morphological analyses y given words x .
For each task, we compare a baseline with no data augmentation, the rule-based GECA data augmentation procedure (Andreas, 2020), and a sequence of ablated versions of R&R that measure [Table] [Table] the importance of resampling and recombination.
The basic Learned Aug model trains an RNN to generate ( x, y ) pairs, then trains a conditional model on the original data and samples from the generative model.
Resampling ﬁlters these samples as described in Section 4.
Recomb-n models replace the RNN with a prototype-based model as described in Section 3.
Additional experiments (Table 1b) compare data augmentation to prediction of y via direct inference (Appendix E) in the prototype-based model and several other model variants.
5. SCAN
SCAN (Lake & Baroni, 2018) is a synthetic dataset featuring simple English commands paired with sequences of actions.
Our experiments aim to show that R&R performs well at one-shot concept learning and zero-shot generalization on controlled tasks where rule-based models succeed.
We experiment with two splits of the dataset, jump and around right .
In the jump split, which tests one-shot learning, the word jump appears in a single command in the training set but in more complex commands in the test set (e.g. look and jump twice ).
The around right split (Loula et al., 2018) tests zero-shot generalization by presenting learners with constructions like walk around left and walk right in the training set, but walk around right only in the test set.
Despite the apparent simplicity of the task, ordinary neural sequence-to-sequence models completely fail to make correct predictions on SCAN test set (Table 1).
As such it has been a major focus of research on compositional generalization in sequence-to-sequence models, and a number of heuristic procedures and specialized model architectures and training procedures have been developed to solve it (Russin et al., 2019; Gordon et al., 2020; Lake, 2019; Andreas, 2020).
Here we show that the generic prototype recombination procedure described above does so as well.
We use long–short recombination for the jump split and long–long recombination for the around right split .
We use a recombination network to generate 400 samples d = ( x, y ) and then train an ordinary LSTM with attention (Bahdanau et al., 2019b) on the original and augmented data to predict y from x .
Training hyperparameters are provided in Appendix D.
Table 1 shows the results of training these models on the SCAN dataset.
4 2-prototype recombination is essential for successful generalization on both splits.
Additional ablations (Table 1b) show that the continuous latent variable used by Guu et al.
(2018) does not affect performance, but that the copy mechanism described in Section 3.2 and the use of the recomb-2 model for data augmentation rather than direct inference are necessary for accurate prediction. [Table] [Table]
5. SIGMORPHON
The SIGMORPHON 2018 dataset consists of words paired with morphological analyses ( lemmas , or base forms [Table] and tags for linguistic features like tense and case, as depicted in Fig.
We use the data to construct a morphological analysis task (Akyürek et al., 2019) (predicting analyses from surface forms) to test models’ few-shot learning of new morphological paradigms.
In three languages of varying morphological complexity (Spanish, Swahili, and Turkish) we construct splits of the data featuring a training set of 1000 examples and three test sets of 100 examples.
One test set consists exclusively of words in the past tense, one in the future tense and one with other word forms (present tense verbs, nouns and adjectives).
The training set contains exactly eight past-tense and eight future-tense examples; all the rest are other word forms.
Experiments evaluate R&R’s ability to efﬁciently learn noisy morphological rules, long viewed a key challenge for connectionist approaches to language learning (Rumelhart & McClelland, 1986).
As approaches may be sensitive to the choice of the eight examples from which the model must generalize, we construct ﬁve different splits per language and use the Spanish past-tense data as a development set.
As above, we use long–long recombination with similarity criteria applied to y only.
We augment the training data withsamples from p recomb and again train an ordinary LSTM with attention for ﬁnal predictions.
Details are provided in Appendix B.
Table 2 shows aggregate results across languages.
We report the model’s F 1 score for predicting morphological analyses of words in the few-shot training condition (past and future) and the standard training condition (other word forms).
Here, learned data augmentation with both one- and twoprototype models consistently matches or outperforms GECA .
The improvement is sometimes dramatic: for few-shot prediction in Swahili, recomb-1 augmentation reduces the error rate by 40% relative to the baseline and 21% relative to GECA .
An additional baseline + resampling experiment upweights the existing rare samples rather than synthesizing new ones; results demonstrate that recombination, and not simply reweighting, is important for generalization.
Table 2 also includes a ﬁner-grained analysis of novel word forms : words in the evaluation set whose exact morphological analysis never appeared in the training set.
R&R again signiﬁcantly outperforms both the baseline and GECA -based data augmentation in the few-shot FUT + PAST condition and the ordinary OTHER condition, underscoring the effectiveness of this approach for “in-distribution” compositional generalization.
Finally, the gains provided by learned augmentation and GECA appear to be at least partially orthogonal: combining the GECA + resampling and recomb-1 + resampling models gives further improvements in Spanish and Turkish.
5. A NALYSIS
Why is R&R effective?
Samples from the best learned data augmentation models for SCAN and SIGMORPHON may be found in the Appendix G.3 .
We programaticaly analyzed 400 samples from recomb-2 models in SCAN and found that 40% of novel samples are exactly correct in the around right split and 74% in the jump split.
A manual analysis of 50 Turkish samples indicated that only 14% of the novel samples were exactly correct.
The augmentation procedure has a high error rate!
However, our analysis found that malformed samples either (1) feature malformed x s that will never appear in a test set (a phenomenon also observed by Andreas (2020) for outputs of GECA ), or (2) are mostly correct at the token level (inducing predictions with a high F 1 score).
Data augmentation thus contributes a mixture of irrelevant examples, label noise —which may exert a positive regularizing effect (Bishop, 1995)—and well-formed examples, a small number of which are sufﬁcient to induce generalization (Bastings et al., 2018).
Without resampling, SIGMORPHON models generate almost no examples of rare tags.
Why does R&R outperform direct inference?
A partial explanation is provided by the preceding analysis, which notes that the accuracy of the data augmentation procedure as a generative model is comparatively low.
Additionally, the data augmentation procedure selects only the highest-conﬁdence samples from the model, so the quality of predicted y s conditioned on random x s will in general be even lower.
A conditional model trained on augmented data is able to compensate for errors in augmentation or direct inference (Table 12 in the Appendix).
Why is Resampling without Recombination effective?
One surprising feature of Table 2 is performance of the learned aug (basic) + resampling model.
While less effective than the recombinationbased models, augmentation with samples from an ordinary RNN trained on ( x, y ) pairs improves performance for some test splits.
One possible explanation is that resampling effectively acts as a posterior constraint on the ﬁnal model’s predictive distribution, guiding it toward solutions in which rare tags are more probable than observed in the original training data.
Future work might model this constraint explicitly, e g. via posterior regularization (as in Li & Rush, 2020).
C ONCLUSIONS
We have described a method for improving compositional generalization in sequence-to-sequence models via data augmentation with learned prototype recombination models.
These are the ﬁrst results we are aware of demonstrating that generative models of data are effective as data augmentation schemes in sequence-to-sequence learning problems, even when the generative models are themselves unreliable as base predictors.
Our experiments demonstrate that it is possible to achieve compositional generalization on-par with complex symbolic models in clean, highly structured domains and outperform them in natural ones, using basic neural modeling tools and no symbolic scaffolding.
A M ODEL A RCHITECTURE A. P ROTOTYPE E NCODER
We use a single layer BiLSTM network to encode h kj proto as follows:
[Equation]
Morphology The hidden and embedding sizes are 1024.
No dropout is applied.
We project bidirectional embeddings to the hidden size with a linear projection.
We concatenate the backward and the forward hidden states.
SCAN We choose the hidden size as 512, and embedding size as 64.
We apply 0.5 dropout to the input.
We project hidden vectors in the attention mechanism.
A. D ECODER
The decoder is implemented by a single layer.
In addition to the hidden state and memory cell, we also carry out a feed vector through time:
[Equation]
The input to the LSTM decoder at time step t is the concatenation of the previous token’s representation, previous feed vector, and a latent z vector (in the VAE model).
[Equation]
Morphology We use a single-layer LSTM network with a hidden size of 1024, and an embedding size of 1024.
We initialize the decoder hidden states with the ﬁnal hidden states of the BiLSTM encoder.
feed is the same size as the hidden state.
No dropout is applied in the decoder.
Output calculations are provided in the original paper in equation Eq.
The query vector for the attentions is identically the hidden state:
[Equation]
Further details of the attention are provided in Appendix A.3.
SCAN The decoder is implemented by a single layer LSTM network with hidden size of 512, and embedding size of 64.
The embedding parameters are shared with the encoder.
Here the size of the feed vector is equal to embedding size, 64.
We have no self-attention for this decoder in the feed vector.
There is an attention projection with dimension is 128.
The details of the attention mechanism given in Appendix A.3.
Finally, we use transpose of the embedding matrix to project feed to the output space.
[Equation]
(27) output t contains unnormalized scores before the ﬁnal softmax layer.
We apply 0.7 dropout to h tout during both training and test.
The copy mechanism will be further described in Appendix A.3.
The input to the LSTM decoder is the same as Eq.
25 except the decoder embedding matrix, W d , shares parameters with encoder embedding matrix W e .
We applied 0.5 dropout to the embeddings d t − 1 .
The query vector for the attention is calculated by:
[Equation]
A. A TTENTION AND C OPYING
We use the attention mechanism described in Vaswani et al.
(2017) with slight modiﬁcations.
Morphology We use a linear transformation for key while retaining an embedding size of 1024, and leave query and value transformations as the identity.
We do not normalize by the square root of the attention dimension.
The query vector is described in the decoder Appendix A.2.
The copy mechanism for the morphology task is explained in the paper in detail.
SCAN We use the nonlinear tanh transformation for key , query and value .
That the attention scores are calculated separately for each prototype using different parameters as well as the normalization i.e. obtaining α ’s is performed separately for each prototype.
The copy mechanism for this task is slightly different and follows Gu et al.
We normalize prototype attention scores and output scores jointly.
Let ¯ α
i represent attention weights for each prototype sequence before normalization.
Then, we concatenate them to the output vector in Eq. 27.
[Equation]
We obtain a probability vector via a ﬁnal softmax layer:
[Equation]
That size of this probability vector is vocabulary size plus the total length all prototypes.
We then project this into the output space by:
[Equation]
where indices ﬁnds all corresponding scores in prob t for token w where there might be more than one element for a given w .
This is because one score can come from the output t region, and others from the prototype regions of prob t .
During training we applied 0.5 dropout to the indices from output t .
Thus, the model is encouraged to copy more.
B N EIGHBORHOODS AND S AMPLING
In the Eq. 20 and Eq.
we expressed the generic form of neighborhood sets.
Here we provide the implementation details.
SCAN In the jump split, we use long-short recombination with δ = 0 .
In around right we use long-long recombination with δ = 0 .
5 , and construct Ω so that the ﬁrst and second prototypes to differ by a single token.
We randomly pick k < 10 × 3 (10 different ﬁrst prototypes, and 3 different second prototypes for each of them) prototype pairs that satisfy these conditions.
For the recombexperiment, we use the same neighborhood setup except but consider only the k < 10 ﬁrst prototypes.
Sampling In the jump split, we used beam search with beam size 4 in the decoder.
We calculate the mean and standard deviation over the lengths of both among the ﬁrst d (cid:48) 1 and the second d (cid:48) 2 prototypes in the train set.
Then, during the sampling, we expect the ﬁrst and second prototypes whose length is shorter than their respective mean plus standard deviation.
This decision is based on the fact that the part of the Ω that the model is exposed to is determined by the empirical distribution, ˆΩ , that arises from training neighborhoods.
When sampling, we try to pick prototypes from a distribution that are close to properties of that empirical distribution.
In around right , we use temperature sampling with
T = 0 .
If a model cannot sample the expected number of both novel and unique samples within a reasonable time, we increase temperature T .
Morphology We use long-long recombination, as explained in the paper, with slight modiﬁcations which leverage the structure of the task.
We set Ω as:
[Equation]
For the recomb-1 model N ( d ) utilizes tag similarity, lemma similarity and is constructed using a score function:
[Equation]
Given d , we sort training examples by using score 1 as the comparison key and pick the four smallest neighbors (using a lexicographic sort) to form N ( d ) .
For the recomb-2 model, N ( d ) uses the same score function for the ﬁrst prototype as in the recombcase.
The second prototype is selected using:
[Equation]
Given x , and a scored ﬁrst prototype, we do one more sort over training examples by using score 2 as the comparison key.
Then we pick ﬁrst four neighbors for N ( d ) .
Sampling We use a mix strategy of temperature sampling with T = 0 .
5 and greedy sampling in which we use the former for d input and the latter for d output .
We sample 180 unique and novel examples.
C G ENERATIVE M ODEL T RAINING
Morphology All of the hyper parameters mentioned here are optimized by a grid search on the
Spanish validation set.
We train our models for 25 epochs 5 .
We use Adam optimizer with learning rate 0.0001.
The generative model is trained on morphological reinﬂection order ( d lemma d tags (cid:46)
d inﬂection ) from left to right, then the samples from the model are reordered for morphological analysis task ( d inﬂection (cid:46) d lemma d tags ).
SCAN We use different number of epochs for jump and around right splits where all models are trained for 8 epochs in the former and 3 epochs in the latter.
We use Adam optimizer with learning rate 0.002, and gradient norm clip with 1.0.
[Equation]
After generating novel samples, we either concatenate them to the training data (in morphology), or sample training batches from a mixture of the original training data and the augmented data.
Our conditional model is the same as the generative model used in morphology experiments, described in detail in the paper body, replacing d proto with x , and d with y .
Every conditional model’s size is the same as the corresponding generative model which was used for augmentation.
This is to ensure that the conditional model and the generative model have the same capacity.
We train conditional models for 150 epochs for SCAN and we used augmentation ratios of p aug = 0 .
01 and p aug = 0 .
2 in jump and around right , respectively.
For morphology, we train the conditional models for 100 epochs, and we use all generated examples for augmentation.
E D IRECT I NFERENCE
To adapt the prototype-based model for conditional prediction, we condition the neighborhood function on the input x rather than the full datum d , as in Hashimoto et al.
Candidate y s are then sampled from the generative model given the observed x while marginalizing over retrieved prototypes.
Finally, we re-rank these candidates via Eq.
and output the highest-scoring candidate.
F
VAE M ODEL
Prior p ( z ) : We use the same prior as Guu et al.
(2018) given in Eq. 31.
In this prior, z is deﬁned by a norm and direction vector.
The norm is sampled from the uniform distribution between zero and a maximum possible norm µ max = 10 . 0 , and the direction is sampled uniformly from the unit hypersphere.
This sampling procedure corresponds to a von Mises–Fisher distribution with concentration parameter zero.
[Equation]
where z
[Equation]
∼ U (0 , µ [Equation] [Equation]
[Equation]
For SCAN, the size of z is 32, and for morphology the size of z is 2.
Proposal Network q ( z | d, d 1: n ) :
Similarly to the prior, the posterior network decomposes z into its norm and direction vectors.
The norm vector is sampled from a uniform distribution at ( | µ | , min ( | µ | + (cid:15), µ max )) , and the direction is sampled from the von Mises–Fisher distribution vmF ( µ, κ ) where κ = 25 , (cid:15)
= 1 . 0 .
[Equation]
G A DDITIONAL R ESULTS G. M ORPHOLOGY R ESULTS
In the paper, Table 2 shows morphology results for (non-VAE) models with 8 hints (past- and futuretense examples in the training set).
Here, we provide additional results for different hint set sizes and model variants.
G.1. HINTS =
[Table] [Table]
G.1. HINTS =
Main 8-prototype F 1 results are provided in the body of the paper.
Here we provide exact match results and an extra set of comparisons to the VAE model.
[Table] [Table]
G.1.
HINTS =
[Table] [Table]
G. S IGNIFICANCE T ESTS
Tables 9, 10 and 11 sho the p -values for pairwise differences between the baseline and prototype-based models
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Table]
G. G ENERATED S AMPLES
All samples are randomly selected unless otherwise indicated.
G.3. SCAN
In Table
[Figure]
we present three test samples from the SCAN task along with the predictions by direct inference and the conditional model trained on the augmented data with recomb-2 .
Note that the augmentation procedure was able to create novel samples whose input ( x ) happens to be in the test set (Examples 1 and 3) while y may or may not be correct (Example 1)
[Figure]
[Table]
[Figure] [Table]
[Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
Below are a set of samples from the learned aug (basic) model for SCAN dataset’s jump and around right splits, in order:
IN: walk around left thrice after jump left left OUT: TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK
TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT
WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN
LEFT WALK
IN: run opposite right twice walk run left thrice OUT: TURN RIGHT TURN RIGHT RUN TURN RIGHT TURN
RIGHT WALK TURN LEFT TURN LEFT RUN TURN LEFT TURN LEFT RUN
Below are a set of samples from the recomb-1 model for SCAN dataset’s around right split.
Note that there were no samples with rare tags generated by the model for the jump split:
IN: run around right after walk around left OUT: TURN LEFT WALK TURN LEFT WALK TURN LEFT RUN TURN
LEFT RUN TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK
IN: look around right after jump around left OUT: TURN LEFT LOOK TURN LEFT JUMP TURN LEFT LOOK TURN
LEFT LOOK TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP
IN: look around right and jump around left OUT: TURN RIGHT LOOK TURN RIGHT LOOK TURN RIGHT LOOK TURN
LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP
IN: walk around right and turn right twice OUT: TURN RIGHT
WALK TURN RIGHT WALK TURN RIGHT WALK TURN
RIGHT WALK TURN RIGHT TURN RIGHT
Below are 4 samples from the recomb-2 model for each of SCAN dataset’s jump and around right splits, respectively:
IN: jump opposite left thrice after jump opposite left thrice OUT: TURN LEFT TURN LEFT JUMP TURN
LEFT TURN LEFT JUMP TURN LEFT TURN LEFT JUMP TURN LEFT TURN LEFT WALK TURN LEFT TURN LEFT WALK TURN
LEFT TURN LEFT WALK
IN: jump left thrice and jump left thrice OUT: TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT
JUMP TURN LEFT JUMP TURN LEFT JUMP
IN: jump opposite right and turn around left OUT: TURN RIGHT TURN
RIGHT JUMP
TURN LEFT TURN LEFT TURN LEFT TURN LEFT
IN: turn around left and jump around left OUT: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT
JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP
IN: look right twice after run around right OUT: TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT
RUN TURN RIGHT RUN TURN RIGHT
LOOK TURN RIGHT LOOK
IN: turn right twice after look around right OUT: TURN
G.3. M ORPHOLOGY
Below are a set of samples from the learned aug (basic) model in SIGMORPHON format.
şahmiçe şahmiçende N;LOC;SG;PSS2S karadan havaya füze karadan havaya füzel N;DAT;PL;PSS3P ernek erneklerine N;DAT;PL;PSS3P kiler kilerime N;DAT;SG;PSS1S mahlep mahlebimizi N;ACC;SG;PSS1P süzmek süzerler V;IND;3;PL;PRS;POS;DECL âlap âlaps N;LGSPEC1;3S;SG;PRS jöle jöleleri N;ACC;PL
envejecerse envejeciéndose
V.CVB;PRS
colaxar colaxa V;IND;PRS
SG pergedrer no pergedremos V;NEG;IMP;
PL mantear no mantees V;NEG;IMP;2;SG flaguear no flagueen V;NEG;IMP;
PL malacostar malacostaría V;COND;
SG
[Table]
Below are a set of samples from the learned aug (basic) + resampling model.
[Table]
Below are a set of samples from the recomb-1 + resampling model (the best performing model in
Table 2).
Here we additionally annotate samples with error categories.
[Table]
G. A TTENTION H EATMAP
Here we provide a visualization copy and attention mechanism in recomb-2 model for SCAN experiments
[Figure]
[Equation]
H T
[Equation]
H
T
[Equation]
H
T
[Equation]
[Equation]
[Equation]
G H
T
I _ T U R N _ R
[Equation]
G H
T
I _ T U R N _
R
I G
[Figure]
T
[Figure]
_ T U R N _
R
I G H
T
I _ T U R N _
R
I G H
T
I _ T U R N _
R
I G H
T
I _ T U R N _
R
I G H
T
I _ T U R N _
R
I G H
T
I _ T U R N _
L E F T
I _ J U M P
I _ T U
R N _
L E F T
I _ J U M P
[Figure]
t u r n a r o un d r i g h t t w i
c
e a n d j u m p l e f t t w i
c
e
I _ T U R N _
R
I G H
T
I _ T U R N _
R
I G H
T
I _ T U R N _
R
I G H
T
I _ T U R N _
R
I G H
T
I _ T U R N _
R
I G H
T
I _ T U R N _
R
I G H
T
I _ T U R N _
R
I G H
T
I _ T U R N _
R
I G H
T
I _ T U R N _
L E F T
I _ J U M P
I _ T U
R N _
L E F T
I _ J U M P
[Figure]
H C OMPUTE
We use a single 32GB NVIDIA V100 Volta GPU for each experiment.
For every experiment, the whole pipeline which consists of training of the generative model, sampling and training of the conditional model takes less than an hour.