W HAT THEY DO WHEN IN DOUBT : A STUDY OF INDUCTIVE BIASES IN SEQ 2 SEQ LEARNERS
Anonymous authors
Paper under double-blind review
A BSTRACT
Sequence-to-sequence (seq2seq) learners are widely used, but we still have only limited knowledge about what inductive biases shape the way they generalize.
We address that by investigating how popular seq2seq learners generalize in tasks that have high ambiguity in the training data.
We use four new tasks to study learners’ preferences for memorization, arithmetic, hierarchical, and compositional reasoning.
Further, we connect to Solomonoff’s theory of induction and propose to use description length as a principled and sensitive measure of inductive biases.
In our experimental study, we ﬁnd that LSTM-based learners can learn to perform counting, addition, and multiplication by a constant from a single training example .
Furthermore, Transformer and LSTM-based learners show a bias toward the hierarchical induction over the linear one, while CNN-based learners prefer the opposite.
The latter also show a bias toward a compositional generalization over memorization.
Finally, across all our experiments, description length proved to be a sensitive measure of inductive biases.
I NTRODUCTION
Sequence-to-sequence (seq2seq) learners (Sutskever et al., 2014) demonstrated remarkable performance in machine translation, story generation, and open-domain dialog (Sutskever et al., 2014;
Fan et al., 2018;
Adiwardana et al., 2020).
Yet, these models have been criticized for requiring a tremendous amount of data and being unable to generalize systematically (Dupoux, 2018; Loula et al., 2018; Lake & Baroni, 2017; Bastings et al., 2018).
In contrast, humans rely on their inductive biases to generalize from a limited amount of data (Chomsky, 1965; Lake et al., 2019).
Due to the centrality of humans’ biases in language learning, several works have studied inductive biases of seq2seq models and connected their poor generalization to the lack of the “right” biases (Lake &
Baroni, 2017; Lake et al., 2019).
In this work, we focus on studying inductive biases of seq2seq models.
We start from an observation that, generally, multiple explanations can be consistent with a limited training set, each leading to different predictions on unseen data.
A learner might prefer one type of explanations over another in a systematic way, as a result of its inductive biases (Ritter et al., 2017; Feinman & Lake, 2018).
To illustrate the setup we work in, consider a quiz-like question: if f (3) maps to 6 , what does f (4) map to?
The “training” example is consistent with the following answers: 6 ( f ( x ) ≡ 6) ;( f ( x ) = x + 3) ; 8 ( f ( x ) = 2 · x ) ; any number z , since we always can construct a function such that f (3) = 6 and f (4) =
z .
By analyzing the learner’s output on this new input, we can infer its biases.
This example demonstrates how biases of learners are studied through the lenses of the poverty of the stimulus principle (Chomsky, 1965; 1980): if nothing in the training data indicates that a learner should generalize in a certain way, but it does nonetheless, then this is due to the biases of the learner.
Inspired by the work of Zhang et al.
in the image domain, we take this principle to the extreme and study biases of seq2seq learners in the regime of very few training examples, often as little as one.
Under this setup, we propose four new synthetic tasks that probe seq2seq learners’ preferences to memorization-, arithmetic-, hierarchical- and compositional-based “reasoning”.
Next, we connect to the ideas of Solomonoff’s theory of induction (Solomonoff, 1964) and Minimal
Description Length (Rissanen, 1978; Grunwald, 2004) and propose to use description length, under a learner, as a principled measure of its inductive biases.
Our experimental study 1 shows that the standard seq2seq learners have strikingly different inductive biases.
We ﬁnd that LSTM-based learners can learn non-trivial counting-, multiplication-, and addition-based rules from as little as one example.
CNN-based seq2seq learners would prefer linear over hierarchical generalizations, while LSTM-based ones and Transformers would do just the opposite.
When investigating the compositional reasoning, description length proved to be a sensitive measure.
Equipped with it, we found that CNN-, and, to a lesser degree, LSTM-based learners prefer compositional generalization over memorization when provided with enough composite examples.
In turn, Transformers show a strong bias toward memorization.
S EARCHING FOR INDUCTIVE BIASES
To formalize the way we look for inductive biases of a learner M , we consider a training dataset of input/output pairs, T = { x i , y i } ni =1 , and a hold-out set of inputs, H = { x i } ki = n +1 .
W.l.o.g, we assume that there are two candidate “rules” that explain the training data, but do not coincide on the hold-out data: C 1 ( x i ) =
C 2 ( x i ) =
y i , 1 ≤ i ≤ n and ∃ i : C 1 ( x i ) (cid:54) =
C 2 ( x i ) , n + 1 ≤ i ≤ k .
To compare preferences of a learner M toward those two rules, we ﬁt the learner on the training data
T and then compare its predictions on the hold-out data H to the outputs of the rules.
We refer to this approach as “intuitive”.
Usually, the measures of similarity between the outputs are task-speciﬁc:
McCoy et al.
(2020) used accuracy of the ﬁrst term, Zhang et al.
(2019) used correlation and MSE, and Lake & Baroni (2017) used accuracy calculated on the entire output sequence.
We too start with an accuracy-based measure.
We deﬁne the fraction of perfect agreement (FPA) between a learner M and a candidate generalization rule C as the fraction of seeds that generalize perfectly in agreement with that rule on the hold-out set H .
Larger FPA of
M is w.r.t.
C , more biased
M is toward C .
However, FPA does not account for imperfect generalization nor allows direct comparison between two candidate rules when both are dominated by a third candidate rule.
Hence, below we propose a principled approach based on the description length.
Description Length and Inductive Biases At the core of the theory of induction (Solomonoff, 1964) is the question of continuation of a ﬁnite string that is very similar to our setup.
Indeed, we can easily re-formulate our motivating example as a string continuation problem: “ 3 → 6; 4 → ”.
The solution proposed by Solomonoff (1964) is to select the continuation that admits “the simplest explanation” of the entire string, i.e. that is produced by programs of the shortest length (description length).
Our intuition is that when a continuation is “simple” for a learner, then this learner is biased toward it.
We consider a learner M to be biased toward C 1 over C 2 if the training set and its extension according to
C 1 has a shorter description length (for M ) compared to that of C 2 .
Denoting description length of a dataset D under the learner
M as L M ( D ) , we hypothesise that if L M ( {C 1 ( x i ) } ki =1 ) <
L M ( {C 2 ( x i ) } ki =1 ) , then M is biased toward C 1 .
Calculating Description Length To ﬁnd the description length of data under a ﬁxed learner, we use the online (prequential) code (Rissanen, 1978; Grunwald, 2004; Blier & Ollivier, 2018).
The problem of calculating L M ( D ) , D = { x i , y i } ki =1 is considered as a problem of transferring outputs y i one-by-one, in a compressed form, between two parties, Alice (sender) and Bob (receiver).
Alice has the entire dataset { x i , y i } , while Bob only has inputs { x i } .
Before the transmission starts, both parties agreed on the initialization of the model
M , order of the inputs { x } , random seeds, and the details of the learning procedure.
Outputs { y i } are sequences of tokens from a vocabulary V .
W.l.o.g.
we ﬁx some order over { x } .
We assume that, given x , the learner M produces a probability distribution over the space of the outputs y , p M ( y | x ) .
The very ﬁrst output y 1 can be sent by using not more than c =
| y 1 | · log | V | nats, using a naïve encoding.
2 After that, both Alice and Bob update their learners using the example ( x 1 , y 1 ) , available to both of them, and get identical instances of M 1 .
Further transfer is done iteratively under the invariant that both Alice and Bob start every step t with exactly the same learners M t − 1 and ﬁnish with identical M t .
At step t Alice would use M t − 1 to
As we are interested in comparing candidate generalization rules, the value of the additive constant c is not important, as it is learner
In experiments, we subtract it from all measurements.
[Table] [Table]
latexit>
[Table]
<latexit sha
[Table]
base
[Table]
/+Si52qX6senNUq9aMijhJZJxtki/hkj9TJCTklDcLJHXkgz+TFuXcenVfnfdw64RQza+QHnI9P8/inCQ==</latexit> encode the next output y t .
This can be done using (cid:0) − log p M t − 1 ( y t | x t ) (cid:1) nats (MacKay, 2003).
Since
Bob has exactly the same model, he can decode the message to obtain y t and use the new pair ( x t , y t ) to update his model and get M t .
Alice also updates her model, and proceeds to sending the next y t
+ (if any), encoding it with the help of M t .
The cumulative number of nats transmitted is:
[Equation]
The obtained code length of Eq.
1 depends on the order in which y are transmitted and the procedure we use to update M .
To account for that, we average out the data order by training with multiple random seeds.
Further, for larger datasets, full re-training after adding a new example is impractical and, in such cases, examples can be transmitted in blocks.
If we measure the description length of the training data T shufﬂed with the hold-out data H , both datasets would have symmetric roles.
However, there is a certain asymmetry in the extrapolation problem: we are looking for an extrapolation from T , not vice-versa.
To break this symmetry, we always transmit outputs for the entire training data as the ﬁrst block.
While L M ( D ) is seemingly different from the “intuitive” measures introduced before, we can illustrate their connection as follows.
Consider a case where we ﬁrst transmit the training outputs as the ﬁrst block and all of the hold-out data outputs under C ,
C ( H ) , as the second block.
Then the description length is equal to cross-entropy of the trained learner on the hold-out data, recovering a process akin to the “intuitive” measuring of inductive biases.
With smaller blocks, the description length also catches whether a learner is capable of ﬁnding regularity in the data fast, with few data points; hence it also encodes the speed-of-learning ideas for measuring inductive biases (Chaabouni et al., 2019).
Finally, the description length has three more attractive properties when measuring inductive biases: (a) it is domain-independent, i.e. can be applied for instance in the image domain, (b) it allows comparisons across models that account for model complexity, and (c) it enables direct comparison between two candidate rules (as we will show in the Section 5).
T ASKS
We describe four tasks that we use to study inductive biases of seq2seq learners.
We select those tasks to cover different aspects of learners’ behavior.
Each task has a highly ambiguous training set which is consistent with inﬁnite number of generalizations.
We pre-select several candidate rules highlighting biases that are useful for language processing and known to exist in humans, or are otherwise reasonable.
Our experiments show that these rules cover many cases of the learners’ behavior.
The ﬁrst two tasks study biases in arithmetic reasoning: Count-or-Memorization quantiﬁes learners’ preference for counting vs. a simple memorization and Add-or-Multiply further probes the learners’ preferences in arithmetic operations.
We believe these tasks are interesting, as counting is needed in some NLP problems like processing linearized parse trees (Weiss et al., 2018).
The third task,
Hierarchical-or-Linear , contrasts hierarchical and linear inductive reasoning.
The hierarchical reasoning bias is believed to be fundamental in learning some syntactical rules in human acquisition of syntax (Chomsky, 1965; 1980).
Finally, with the Composition-or-Memorization task, we investigate biases for systematic compositionality, which are central for human capabilities in language.
Figureillustrates these four tasks.
By a k we denote a sequence that contains token a repeated k times.
For training, we represent sequences in a standard way: the tokens are one-hot-encoded separately, and we append a special end-of-sequence token to each sequence.
Input and output vocabularies are disjoint.
Count-or-Memorization :
In this task, we contrast learners’ preferences for counting vs. memorization.
We train models to ﬁt a single training example with input a l and output b
l (i.e., to perform the mapping a l → b l ) and test it on a m with m ∈
[ l − 10 , l +10] .
If a learner learns the constant function, outputting b l independently of its inputs, then it follows the mem strategy.
On the other hand, if it generalizes to the a m → b m mapping, then the learner is biased toward the count strategy.
Add-or-Multiply : This task is akin to the motivating example in Section 1.
The single training example represents a mapping of an input string a l to an output string b 2 l .
As test inputs, we generate a m for m in the interval [ l − 3 , l + 3] .
We consider the learned rule to be consistent with mul if for all m , the input/output pairs are consistent with a m → b 2 m .
Similarly, if they are consistent with a m → b m + l , we say that the learner follows the addition rule, add .
Finally, the learner can learn a constant mapping a m → b 2 l for any m .
Again, we call this rule mem .
Hierarchical-or-Linear :
For a ﬁxed depth d , we train learners on four training examples x d yx d → y where x, y ∈ { a, b } .
Each training example has a nested structure, where d deﬁnes its depth.
A learner with a hierarchical bias ( hierar ), would output the middle symbol.
We also consider the linear rule ( linear ) in which the learner outputs the ( d + 1) th symbol of its input.
To probe learners’ biases, we test them on inputs with different depths m ∈
[ d − 2 , d + 2] .
Note that to examine the linear rule (i.e. if the learner outputs the ( d + 1) th symbol of any test input of depth m ), we need m ≥ d 2 .
Similar to the previous tasks, there is no vocabulary sharing between a model’s inputs and outputs (input and output tokens a and b are different).
Composition-or-Memorization : We take inspiration from SCAN (Lake & Baroni, 2017), a benchmark used for studying systematic generalization of seq2seq learners.
The input vocabulary has N symbols a i that are one-to-one mapped into N output symbols
b i (i.e., a i → b i ).
In addition, there is a modiﬁer token thrice : when thrice precedes an input symbol a i , the corresponding output is repeated three times: thrice a
i
→ b i b i b i .
We train a learner on all non-compositional examples ( a i → b i ) and M ( M < N ) compositional examples ( thrice a i → b i b i b i ).
At test time, we feed the learner with the remaining compositional examples ( thrice a
i , i > M ).
If the learner generalizes to the mapping thrice a i → b i b i b i for i > M , we consider it to be biased toward a compositional reasoning ( comp ).
As an alternative generalization, we consider a mapping where all inputs containing a i are mapped into b i : thrice a i → b i ( i > M ).
We call this generalization memorization ( mem ).
M ETHODOLOGY 4. S EQUENCE - TO - SEQUENCE LEARNERS
We experiment with three standard seq2seq models: LSTM-based seq2seq (LSTM-s2s)
(Sutskever et al., 2014), CNN-based seq2seq (CNN-s2s) (Gehring et al., 2017), and Transformer (Vaswani et al., 2017).
All share a similar Encoder/Decoder architecture (Sutskever et al., 2014).
LSTM-s2s and are implemented as LSTM cells (Hochreiter & Schmidhuber, 1997). Encoder encodes its inputs incrementally from left to right.
We experiment with architectures without (LSTM-s2s no att.)
and with (LSTM-s2s att.) an attention mechanism (Bahdanau et al., 2014). For the ﬁrst three tasks, both Encoder and Decoder are single-layer LSTM cells with hidden size of 512 and embedding of dimension 16.
CNN-s2s Encoder and Decoder are convolutional networks (LeCun et al., 1990), followed by GLU non-linearities (Dauphin et al., 2017) and an attention layer.
To represent positions of input tokens,
CNN-s2s uses learned positional embeddings.
Encoder and Decoder networks have one layer with 512 ﬁlters and a kernel width of 3.
We set the embedding size to 16.
Transformer Encoder and Decoder are implemented as a sequence of (self-)attention and feedforward layers.
We use sinusoidal position embeddings.
Both Encoder and Decoder contain one transformer layer.
The attention modules have 8 heads, feed-forward layers have dimension ofand the embedding is of dimension 16.
In Appendix we report experiments where we vary hyperparameters of the learners.
4. T RAINING AND EVALUATION
For all tasks, we follow the same training procedure.
We train with Adam optimizer (Kingma & Ba, 2014) for 3000 epochs.
The learning rate starts at 10 − 5 and increases for the ﬁrst 1000 warm-up updates till reaching 10 − 3 .
We include all available examples in a single batch.
We use teacher forcing (Goodfellow et al., 2016) and set the dropout probability to 0.5.
For each learner, we perform training and evaluation 100 times, changing random seeds.
At generation, to calculate FPA, we select the next token greedily.
We use the model implementations from fairseq (Ott et al., 2019).
As discussed in Section 2, when calculating L , we use the training examples as the ﬁrst transmitted block at t = 1 .
In Count-or-Memorization and Add-or-Multiply this block contains one example, in Hierarchical-or-Linear it has 4 examples, and in Composition-or-Memorization it has N + M examples.
Next, we transmit examples obtained from the candidate rules in a randomized order, by blocks of size 1, 1, 1, and 4 for Count-or-Memorization , Add-or-Multiply , Composition-orMemorization , and Hierarchical-or-Linear respectively.
At each step, the learner is re-trained from the same initialization, using the procedure and hyper-parameters as discussed above.
Our training setup is typical for seq2seq training.
It does not include any additional pressure towards description length minimization.
Description length is solely used as a measure of inductive biases.
E XPERIMENTS
Count-or-Memorization We investigate here learners’ biases toward count and mem rules.
We provide a single example a l → b l as the training set, varying l ∈ { 10 , 20 , 30 , 40 } .
We report the learners’ performances in Table 1a.
We observe that, independently of the length of the training example l , CNN-s2s and Transformer learners inferred perfectly the mem rule with FPA- mem > 0.
(i.e. more than 90% of the random seeds output b l for any given input a m ).
However, LSTM-based learners demonstrate a more complex behavior.
With l = 10 , both learners (with and without attention) exhibit a preference for mem .
Indeed, while these learners rarely generalize perfectly to any of the hypothesis ( 0 .
FPA (no att.), 0 . 2 / 0 .
FPA for mem / count (att.)), they have signiﬁcantly lower L - mem .
As l increases, LSTM-based learners become more biased toward count .
Surprisingly, for l ≥ 30 , most learner instances show sufﬁciently strong inductive biases to infer perfectly the non-trivial count hypothesis.
With l = 40 , 99% of random seeds of
LSTM-s2s att.
and all (100%) of LSTM-s2s no att.
seeds generalized perfectly to count .
Further, we see that if L shows similar trends, it has a higher sensitivity.
For example, while both
LSTM-based learners have a similar FPA with l = 40 , L demonstrates that LSTM-s2s no att.
has a stronger count bias.
Add-or-Multiply In this task, we examine learners’ generalization after training on the single example a l → b 2 l .
We vary l ∈ { 5 , 10 , 15 , 20 } .
In Table 1b, we report FPA and L for the three generalization hypotheses, add , mul , and mem .
We observe, similarly to the previous task, that
CNN-s2s and Transformer learners always converge perfectly to memorization.
[Table]
In contrast, LSTM-based learners show non-trivial generalizations.
Examining ﬁrst LSTM-s2s att.
, when l = 5 , we note that mem has a high FPA and an L considerably lower than others.
This is consistent with the learner’s behavior in the Count-or-Memorization task.
As we increase l , more interesting behavior emerges.
First, L - mem decreases as l increases.
Second, mul -type preference increases with l .
Finally, L - add presents a U-shaped function of l .
That is, for the medium example length l , the majority of learners switch to approximating the add rule (for l = 10 ).
However, when l grows further, a considerable fraction of these learners start to follow a mul -type rule.
Strikingly, 98% of LSTM-s2s att.
seeds generalized perfectly to the non-trivial mul rule.
As for LSTM-s2s no att.
, we do not observe a strong bias to infer any of the rules when l = 5 .
However, when increasing l , the LSTM-s2s no att.
behaves similarly to LSTM-s2s att.
: at ﬁrst it has a preference for add (FPA- add =0.95, for l = 10 ) then for mul (e.g. FPA- mul =0.94, for l = 20 ).
Hierarchical-or-Linear
We look now at learners’ preference for either hierar or linear generalizations.
The architectures we use were only able to consistently learn the training examples with the depth d not higher than 4 .
Hence, in this experiment, we set d to 4 .
We report in Table 1c the learners’ FPA and L .
We observe that CNN-s2s exhibits a strikingly different bias compared to all other learners with a perfect agreement with the linear rule.
In contrast, Transformer learners show a clear preference for hierar with a high FPA (0.69) and a low L (1.21).
Surprisingly, this preference increases with the embedding size and Transformers with embedding size ≥ 64 admit an FPA- hierar of 1.00 (see Appendix for more details).
LSTM-s2s att.
learners demonstrate also a similar preference for hierar with an FPA of 0.30 and a considerably lower L than L - hierar .
Finally, while only 5% of LSTM-s2s no att.
instances generalized to perfect hierar (and none to linear ), L conﬁrms their preference for the hierar hypothesis.
Composition-or-Memorization
In this task, we set the number of primitives N = 40 and vary the number of compositional examples M ∈ { 6 , 24 , 36 } seen during training .
Results are reported in Table 1d.
First, we observe that FPA is only informative for CNN-s2s when trained with a large
M .
Indeed, for M = 6 , CNN-s2s does not infer any of the candidate rules.
However, according to description length L , we note a signiﬁcant preference for mem over comp .
More compositional examples CNN-based learners see at training, more biased they become toward comp .
The remaining learners have zero FPA for both candidate rules.
However, according to description length, LSTMbased learners have preferences similar to CNN-s2s, although weaker.
That is, they show a preference for mem for low M , that declines in favor of comp as M increases.
In contrast, Transformers show a strong bias for mem with all tested M .
Overall, across all the above experiments, we see that seq2seq learners demonstrate strikingly different biases.
In many cases, these biases lead to non-trivial generalizations when facing ambiguity in the training data.
This spans tasks that probe for memorization, arithmetic, hierarchical, and compositional reasoning.
We found that a single example is sufﬁcient for LSTM-based learners to learn counting, addition, and multiplication.
Moreover, within the same task, they can switch from one explanation to another, depending on the training example length, with Addition-or-Multiplication being the task where this switch happens twice.
In contrast, CNN-s2s and Transformers show a strong bias toward memorization.
Furthermore, all learners except for CNN-s2s demonstrate a strong bias toward the hierarchical behavior.
In the task of compositional generalization, CNN-s2s shows a strong bias toward compositional reasoning that appears after a few compositional training examples.
On the other hand, Transformers show a preference for memorization over compositional generalization.
We see that the conclusions derived from comparing the description length of the candidate rules are in agreement with the results under accuracy-based metrics, but provide a more nuanced picture.
Robustness to hyper-parameters We observe that learners’ biases depend, in many cases, on the length/number of the input examples.
In Appendix, we examine impact of other hyper-parameters.
In particular, we study impact of (1) learners’ architecture size, by varying the number of layers, hidden and embedding sizes, and (2) dropout probability.
Our results show that in some cases a learner’s architecture size can inﬂuence the strength of inductive biases, but rarely modify them: among the 136 tested settings, we observe only 3 cases of switching the preferences.
We also found that large dropout probabilities are necessary to prevent mem -type generalization.
Furthermore, in Appendix, we show that a variant of Transformer learners, namely the joint sourcetarget self-attention learner (He et al., 2018; Fonollosa et al., 2019), displays the same preferences as the standard Transformer learners.
This variant resembles the “decoder-only” architecture used in language modeling (Radford et al., 2019; Brown et al., 2020).
This result demonstrates that our tasks and bias measures could be applied for studying inductive biases of language model architectures.
R ELATED W ORK
Dessì & Baroni (2019) found that, unlike LSTM-s2s learners, CNN-s2s can perform compositional generalization on SCAN.
Our experiments indicate that this only happens when enough compositional examples are provided in the training.
Moreover, in such a case, attention-enabled LSTM-s2s also start to prefer compositional generalization over memorization.
McCoy et al.
(2020) studied inductive biases of recurrent neural architectures in two synthetic tasks,
English question formation and tense inﬂection.
They found that only tree-based architectures show robust preference for hierarchical reasoning, in contrast to LSTM-s2s learners that generalized linearly.
Our experiments on the hyperparameter robustness, reported in Appendix, indicate that the preferences over linear/hierarchical reasoning are strongly affected by the dropout probability, with learners shifting to linear behavior with low probabilities.
As McCoy et al.
(2020) experimented with a low dropout probability of 0.1, we believe this explains the misalignment of the conclusions.
Overall, our study shows that the inductive biases is more complicated than it seemed in these prior works and a more careful analysis is crucial.
We believe that our extremely controlled setup with very little confounds is a good addition for those studies.
Another line of research investigates theoretically learners’ capabilities, that is, the classes of the hypothesis that a learner can discover (Siegelmann & Sontag, 1992; Suzgun et al., 2019; Merrill et al., 2020).
For example, Weiss et al.
(2018) demonstrated that LSTM cells can count.
In turn, we demonstrate that LSTM-s2s learners are not only capable but also biased toward arithmetic behavior.
D ISCUSSION AND C ONCLUSION
In this work, we studied inductive biases of standard seq2seq learners, Transformer-, LSTM-, and
CNN-based.
To do so, we introduced four new tasks, which allowed us to cover an interesting spectrum of behaviors useful for language learning.
In particular, we considered arithmetic, hierarchical, and compositional “reasoning”.
Next, we connected the problem of ﬁnding and measuring inductive biases to Solomonoff’s theory of induction and proposed to use a dataset’s description length under a learner as a tool for sensitive measurement of inductive biases.
In our experiments, we found that the seq2seq learners have strikingly different inductive biases and some of them generalize non-trivially when facing ambiguity.
For instance, a single training example is sufﬁcient for LSTM-based learners to learn perfectly how to count, to add and to multiply by a constant.
Transformers and, to a lesser degree, LSTM-s2s demonstrated preferences for the hierarchical bias, a bias that has been argued to govern children’s acquisition of syntax.
Interestingly, such biases arose with no explicit wiring for them.
Our results support then Elman et al.
(1998)’s theory which states that human’s inductive biases can arise from low-level architectural constraints in the brain with no need for an explicit encoding of a linguistic structure.
However, how the brain, or, more generally, a learner is wired to admit a speciﬁc inductive bias is still an important open question.
Across our experiments, we also observed that description length is consistent with “intuitive” measurements of inductive biases, and, at the same time, it turned out to be more sensitive.
This also indicates that, in the presence of ambiguity in the training data, a learner is more likely to follow the alternative with the shorter description length (i.e. the simplest one) when applied on unseen data, showing consistency with the prescriptions of the theory of induction (Solomonoff, 1964).
A similar simplicity preference is argued to play a role in human language acquisition (Perfors et al., 2011).
Our work provides simple tools to investigate learners’ biases.
We ﬁrst show that FPA is an intuitive measure to study biases when provided with simple tasks.
Second, we present description length as a robust measure to fairly compare learners’ biases.
This metric considers learners’ size and their ease of learning as opposed to accuracy-based metrics.
Besides, it is a model- and task-agnostic measure that succeeds in unveiling learners’ biases even when presented with more complex tasks with spurious correlations.
Our ﬁndings can guide for architecture selection in the low-data regimes where inductive biases might have a higher inﬂuence on model’s generalization performance.
Large sparse datasets can also beneﬁt from predictable behavior in few-shot scenarios akin to what we consider.
Finally, our results demonstrate that relatively large deep learning models can generalize non-trivially from as little as one example – as long as the task is aligned with the their inductive biases.
We believe this should reinforce interest in future work on injecting useful inductive biases in our learners and, we hope, our ﬁndings and setup can provide a fertile ground for such work
A C AN SEQ 2 SEQ LEARNERS MULTIPLY BY 3?
In our experiments on the Multiply-or-Add task we saw that LSTM-s2s learners are able to learn to multiply by 2 from a single example.
A natural further question is whether these learners can learn to multiply by larger numbers?
Here we only provide a preliminary study in a hope to inspire more focused studies.
We build a task that is similar to Multiply-or-Add , but centered around multiplication by 3 instead of 2.
The single training example represents a mapping of an input string a l to an output string b 3 l .
As test inputs, we use a m with m coming from an interval [ l − 3 , l + 3] .
Since 3 x can be represented as a combination of addition and multiplication in several ways ( 3 x = x + 2 x = 2 x + x ), we consider 4 different candidate rules.
As before, by mem we denote the constant mapping from a m → b 3 l .
mul1 represents the mapping a m → b 2 l + m .
mul2 corresponds to a m
→
b l +2 m and mul3 denotes a m → b 3 m .
The “explanation” mul1 is akin to the add rule in the Multiply-or-Add task.
We use the same hyperparameters and training procedure described in Section 4.
We report the results in Table 2.
Like the results observed in the Multiply-or-Add task, both CNN-s2s and
Transformer learners show a strong preference for the mem rule while LSTM-based learners switch their generalization according to the length of the training example l .
Indeed, for CNN-s2s and Transformer, we note an FPA- mem >0.97 independently of l (with L - mem signiﬁcantly lower than others).
LSTM-s2s att.
learners start inferring the mem rule for l = 5 (FPA=0.64, L =2.44), then switch to comparable preference for mul and mul3 when l = 10 , and ﬁnally show a signiﬁcant bias toward the mul3 hypothesis for l ∈ { 15 , 20 } (e.g.
FPA- mul3 =0.76 for l = 15 ).
LSTM-s2s no att.
learners are also subject to a similar switch of preference.
That is, for l = 5 , these learners have a signiﬁcant bias toward mul1 (FPA=0.49).
Strikingly, when l = 10 , 92%
LSTM-s2s no att.
learners inferred perfectly the mul2 rule after one training example.
Finally, we observe again another switch to approximate mul3 for l ∈ { 15 , 20 } .
Overall, while CNN-s2s and Transformer learners show a signiﬁcant and robust bias toward mem , LSTM-based learners generalize differently depending on the training input length.
In particular, our results suggest that these learners avoid adding very large integers by switching to the multiplicative explanation in those cases.
Answering our initial question, we see that LSTM-based learners can learn to multiply by 3 from a single example.
5 Only 21% of learners succeeded in learning the training example in this setting. Only 51% of learners succeeded in learning the training example in this setting.
B R OBUSTNESS TO CHANGES IN ARCHITECTURE
In this section, we examine how changing different hyper-parameters affects learners’ preferences for mem- orization, arithmetic, hierarchical and compositional reasoning.
In particular, we vary the number of layers, hidden and embedding sizes of the different learners and test their generalization on Count-or-Memorization , Add-or-Multiply , Hierarchical-or-Linear , and Composition-or-Memorization tasks.
In all the experiments, we ﬁx l = 40 and l = 20 for Count-or-Memorization and Add-or-Multiply respectively, d = 4 for Hierarchical-or-Linear and M = 36 for Composition-or-Memorization .
Finally, we keep the same training and evaluation procedure as detailed in Section 4.2 of the main text.
However, we use 20 different random seeds instead of 100.
B. N UMBER OF HIDDEN LAYERS ( N Layer )
We experiment with the standard seq2seq learners described in the main paper and vary the number of layers N
Layer ∈ { 1 , 3 , 10 } .
Results are reported in Table 3.
First, when looking at the interplay between mem and count (Table 3a), we observe that, independently of N
Layer , more than 97% of CNN-s2s and Transformer learners inferred perfectly the mem rule (i.e. output b 40 for any given input a m ).
Further, if the preference for count decreases with the increase of N Layer , LSTM-s2s att.
learners display in all cases a signiﬁcant bias toward count with a large FPA and an L signiﬁcantly lower than L - mem .
However, we note a decline of the bias toward count when considering LSTM-s2s no att.
learners.
For N Layer = 1 , 100% of the seeds generalize to perfect count , versus 6% for N Layer = 10 .
Note that this lower preference for count is followed by an increase of preference for mem .
However, there is no signiﬁcant switch of preferences according to L .
Second, we consider the Add-or-Multiply task where we examine the three generalization hypothesis add , mul and mem .
Results are reported in Table 3b.
Similarly to the previous task, Transformer and CNN-s2s learners are robust to the number of layers change.
They perfectly follow the mem rule with FPA- mem =1.00.
However,
LSTM-based learners show a more complex behavior: If single-layer LSTM-s2s att.
and no att.
demonstrate a considerable bias for mul (FPA- mul >0.94), this bias fades in favor of add and memo for larger architectures.
Third, in Table 3c, we observe that larger learners are slightly less biased toward hierar and linear .
However, we do not observe any switch of preferences.
That is, across different N layers , CNN-s2s learners prefer the linear rule, whereas Transformers and, to a lesser degree, LSTM-based learners show a signiﬁcant preference for the hierar rule.
Finally, we do not observe an impact of N Layer on the Composition-or-Memorization task (see Table 3d), demonstrating the robustness of biases w.r.t.
learners’ size.
In sum, we note little impact of N Layer on learners’ generalizations.
Indeed, LSTM-based learners can learn to perform counting from a single training example, even when experimenting with 10-layer architectures.
For most tested N Layer , they favor mul and add over mem .
In contrast, Transformer and CNN-s2s perform systematic memorization of the single training example.
Furthermore, independently of N Layer , Transformer and LSTM-based learners show a bias toward the hierar hypothesis over the linear one, while CNN-based learners do the opposite.
Finally, the compositional experiments further support how inductive biases are barely inﬂuenced by the change of the number of layers.
B. H IDDEN SIZE ( S Hidden )
We experimented in the main text with standard seq2seq learners when hidden size S Hidden = 512 .
In this section, we look at the effect of S Hidden varying it in { 128 , 512 , 1024 } .
We report learners performances in
Table 4.
First, Table 4a demonstrates how minor effect hidden size has on learners counting/memorization performances.
Indeed, for any given S Hidden , between 84% and 100% of LSTM-based learners learn perfect counting after only one training example.
Similarly, and with even a lower variation, more than 91% of Transformer and CNN-s2s learners memorize the single training example outputting b 40 for any given a m .
The same observation can be made when studying the interplay between the hierar and linear biases (Table 4c) and comp and mem biases (Table 4d).
Concretely, Tables 4c and 4d show that learners’ generalizations are stable across S Hidden values with the exception of LSTM-based learners that loose signiﬁcance for some tested S
Hidden values.
Yet, there is no switch of preference.
Finally, as demonstrated in Table 4b, all Transformer and CNN-s2s learners perform perfect memorization when tested on the Add-or-Multiply task independently of their hidden sizes.
Both LSTM-based learners are signiﬁcantly biased toward mul for S Hidden ∈ { 512 , 1024 } .
However, when experimenting with smaller
[Table] [Table] [Table] S Hidden (=128), we detect a switch of preference for LSTM-s2s no att.
learners.
The latter start approximating add -type rule (with signiﬁcantly lower L ).
Lastly, we do not distinguish any signiﬁcant difference between add and mul for LSTM-s2s att.
when S Hidden = 128 .
Taken together, learners’ biases are quite robust to S Hidden variations.
We however note a switch of preference from mul to add for LSTM-s2s no att.
learners when decreasing S Hidden .
Furthermore, we see a loss of signiﬁcant preference in ﬁve distinct settings.
B. E MBEDDING SIZE ( S Emb )
We look here at the effect of the embedding size, S Emb , on learners’ generalizations.
In particular, we vary S
Emb ∈ { 16 , 64 , 256 } .
Results are reported in Table 5.
Across sub-tables 5a, 5b and 5c, we see small inﬂuence of S Emb on learners’ biases.
For example, if we consider the Count-or-Memorization task when varying S Emb (see Table 5a), between 95% and 100% of LSTM-s2s no att.
learners inferred perfectly the count hypothesis.
More striking, between 99% and 100% of LSTM-s2s att.
learners learned the count rule after one training example.
The same trend is observed for the remaining learners and across the other tasks; Add-or-Multiply (Table 5b) and Hierarchical-or-Linear (Table 5c).
Yet, we still discern in some cases, systematic, but low, effects of S Emb .
First, the larger S Emb is, the lower FPA- mul of LSTM-s2s no att.
learners is (from 0.94 for S Emb = 16 to 0.84 for S Emb = 256 ).
However, LSTM-s2s no att.
learners still have considerable preference for mul for any tested S Emb .
Second, we see an increase of Transformer’s preference for hierar with the increase of S Emb .
Surprisingly, for S Emb ≥ 64 , 100% of
Transformer learners generalize to perfect hierar hypothesis.
Finally, when considering the Composition-or-Memorization task, we observe in Table 5d, for all learners, a tendency to prefer comp generalization when S Emb is large.
For example, if, for LSTM-based learners, we note 0 .
comp -FPA when S Emb = 16 , comp -FPA ≥ 0 .
85 with S Emb = 256 .
Interestingly, Transformers switch their preference, showing a signiﬁcant bias toward compositional reasoning with S Emb = 256 .
In this section, we studied the impact of the number of layers, hidden and embedding sizes on learners’ generalizations.
We found that, if these hyper-parameters can inﬂuence, in some cases, the degree of one learner’s preference w.r.t.
a given rule, inductive biases are quite robust to their changes.
In particular, among all tested combinations, we observe only 3 cases of preference switch (out of 136 ).
C R OBUSTNESS TO CHANGES IN TRAINING PARAMETERS
We examine here the effect of the training parameters on learners’ biases.
As previously, we only consider the Count-or-Memorization task with l = 40 , the Add-or-Multiply task with l = 20 , the Hierarchical-or-Linear task with d = 4 and the Composition-or-Memorization task with M = 36 .
[-.11 ∗ 31.
-]
We experiment with the architectures detailed in the main text; however, we use 20 different random seeds instead of 100.
We consider in this section two different hyperparameters: (1) the choice of the optimizer, and (2) the dropout probability.
Optimizer We experiment with replacing the Adam optimizer (Kingma & Ba, 2014) with SGD (Bottou et al., 2018).
We found experimentally that learners failed to learn the training examples consistently in most of the settings.
Yet, when successful, they showed the same preferences.
In particular, Transformer and CNN-s2s were the only learners that had good performances on Count-or-Memorization and Add-or-Multiply train sets (success rate higher than 70% ).
These learners showed a prefect generalization to the mem rule in both tasks.
Prior works had shown that SGD leads to better generalization compared to other adaptive variants (Wilson et al., 2017).
However, based on our few successful instances with SGD, we do not observe a difference between SGD and
Adam results.
One main difference compared to the prior work is the deﬁnition of generalization. For instance, in the Count-or-Memorization task, memorizing the training example could be seen as a perfect generalization as it explains the training set fully. Hence, preferring memorization when trained with SGD, as opposed to counting, does not contradict the advantage of SGD when testing generalization performances. Dropout We then examine how the dropout probability affects learners’ preferences.
We use, as mentioned in the main paper, Adam optimizer and vary the dropout probability dropout ∈ { 0 . 0 , 0 .
Results are reported in Table 6.
Both Count-or-Memorization (Table 6a) and Add-or-Multiply (Table 6b) tasks show the same trend.
First,
Transformer and CNN-s2s learners prefer consistently the mem rule.
Second, when looking at LSTM-based learners, we distinguish a more complex behavior.
For dropout ≥ 0 .
2 , LSTM-based learners show a signiﬁcant preference for arithmetic reasoning ( count for the Count-or-Memorization task and mul for the Add-or- Multiply task).
However, when dropout = 0 . 0 , we see different preferences.
In particular, both LSTM-based learners show a preference for mem (not signiﬁcant for LSTM-s2s att.), LSTM-s2s no att. learners are signiﬁcantly [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Transformer
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
FPA
[Table]
linear
[Table]
no att.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
att.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
comp
[Table]
no
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
{+.11 ∗ 31.
+}
[Table]
2.12 ∗ 23.
LSTM
[Table]
s2s att.
[Table]
[Table]
[Figure]
[Table]
[Figure]
biased toward add whereas LSTM-s2s att.
do not show any signiﬁcant bias with a slight preference for mem .
In sum, the lower dropout is, the more likely learners will overﬁt the mem rule.
The same observation holds for the Composition-or-Memorization task in Table 6d.
That is, dropout has no impact on CNN-s2s and Transformer biases (the former show a signiﬁcant preference for comp while the latter prefers mem for any tested dropout value), but does impact LSTM-based learners’ biases, such as a dropout = 0 .
0 favors memorization.
Finally, we consider the Hierarchical-or-Linear task (see Table 6c).
We observe that, for any dropout value,
CNN-s2s and Transformer inductive biases remain the same.
Indeed, CNN-s2s learners show a consistent preference for linear with FPA ≥ 0 .
75 while Transformers prefer hierar (note that this preference is not very large for dropout = 0 .
with an FPA- hierar of 0.05, compared to an FPA- hierar of 1.00 and 0.
for dropout = 0 .
2 and 0 .
5 respectively).
On the other hand, dropout has a larger impact on LSTM-based learners.
When dropout = 0 .
5 , both LSTM-s2s prefer the hierar hypothesis.
However, for dropout = 0 . 0 , both learners do not show any signiﬁcant preference for any of the rules (with 0 FPA for both rules and close L values).
D SCAN
In the main text, we studied preferences of the learners towards compositional generalization in the Composition- or-Memorization task.
Here, we resort to a larger-scale SCAN task (Lake & Baroni, 2017), which can be thought of as a generalization of Composition-or-Memorization with multiple modiﬁers.
In SCAN, inputs are sequences that represent trajectories and outputs are step-wise instructions for following these instructions (see Figure 2).
We experiment with the SCAN-jump split of the dataset, where the test set (7706 examples) is obtained by ﬁltering all compositional uses of one of the primitives, jump .
The train set (14670 examples) contains all uses of all other primitives (including compositional), and lines where jump occurs in isolation (only as a primitive).
This makes the training data ambiguous only for the jump instruction.
However, learners can transfer compositional rules across the primitives.
We refer to this split simply as SCAN.
We consider two generalizations: (1) all input sequences with jump are mapped to a single instruction JUMP as memorized from the training examples, or (2) underlying compositional grammar.
We call them mem and comp , respectively.
We used the original test as a representative of the comp candidate explanation and generated one for mem ourselves.
Sequence-to-sequence learners When experimenting with SCAN, we use the following models:
LSTM-s2s : We chose the architecture used in Lake & Baroni (2017).
In particular, Encoder and Decoder are two hidden layers with 200 units and embedding of dimension 32.
This architecture gets 0.98 test accuracy on the i.i.d.
(simple) split of SCAN, averaged over 10 seeds.
CNN-s2s : We use one of the successful CNN-s2s models of Dessì & Baroni (2019) that has 5 layers and embedding size of 128.
We also vary kernel size in { 3 , 5 , 8 } as it was found to impact the performance on SCAN (Dessì & Baroni, 2019).
These architectures reach test accuracy above 0.98 on the i.i.d.
split of SCAN, averaged over 10 seeds.
Transformer :
We believe our work is the ﬁrst study of Transformer’s performance on SCAN.
For both Encoder and Decoder, we use 8 attention heads, 4 layers, embedding size of 64, FFN layer dimension of 256.
This architecture gets .94 test accuracy on the i.i.d.
split of SCAN, averaged over 10 seeds.
Training and evaluation We follow the same scenario described in the main paper with two differences: (a) when calculating L , we use blocks of size 1024, (b) during training, we sample 10 4 batches with replacement, similar to (Lake & Baroni, 2017).
We use batches of size 16 for LSTM-s2s & CNN-s2s, and 256 for Transformer.
We repeat the training/evaluation of each learner 10 times, varying the random seed.
Again, we use Adam optimizer and a learning rate of 10 − 3 .
Due to the large number of test examples and low performance of the learners, all FPA scores would be equal to zero.
Hence, we follow Lake & Baroni (2017) and use per-sequence accuracy, i.e. the ratio of the sequences with all output tokens predicted correctly.
Results We report our results in Table 7.
First, we see that CNN-s2s learners have a strong preference to comp , both in accuracy and description length.
Furthermore, we observe that with an increase in the kernel size, description length of mem increases, while description length of comp decreases, indicating that the preference for comp over mem grows with the kernel width.
We believe this echoes ﬁndings of (Dessì & Baroni, 2019). [Table] [Table]
While accuracy is well below 0 .
01 for all other learners/candidate combinations (and rounded to 0.00), according to the description length, Transformer and LSTM-based learners also have preference for comp over mem .
This can be due to the transfer from compositional training examples, that can make comp explanation most “simple” given the dataset.
Hence, the failure for systematic generalization in SCAN comes not from learners’ preferences for mem .
We believe this resonates with the initial qualitative analysis in (Lake & Baroni, 2017).
E S TUDYING INDUCTIVE BIASES OF LM- LIKE ARCHITECTURES
While in this paper we focused on studying inductive biases of seq2seq architectures, we believe our entire framework can be used for investigating biases of architectures used in language modeling (LM).
As language modeling emerges as a general few-shot learning mechanism where textual prompting is used as an inter- face (Brown et al., 2020), it becomes important to understand how models generalize in such a setup.
In this
Section, we demonstrate the applicability of our tasks and measures to such learners and scenarios.
We propose to study inductive biases of LM-like architectures by using prompt-based versions of our tasks akin to GPT2 (Radford et al., 2019) & GPT3 (Brown et al., 2020). As an example, we can train an LM-like architecture to continue the sequence “<sos>aaa<sep>” as "<sos>aaa<sep>bbb<eos>" and then prompt it with "<sos>aaaa<sep>" and check the output: is it "<sos>aaaa<sep>bbb<eos>" ( mem ) or "<sos>aaaa<sep>bbbb<eos> ( count )? Such formulation makes it trivial to adapt our tasks (Section 3) for the architectures used in language modeling. However, training differs from that of a language model due to the presence of the prompt.
In the following, we experiment with a variant of Transformer, namely joint source-target self-attention learner introduced by He et al.
We use the code provided in (Fonollosa et al., 2019).
The joint source-target self-attention learner differs from the structure of Transformer in three ways.
First, instead of attending to the last layer of the encoder in standard Transformers, here, each layer in the decoder attends to the corresponding layer in the encoder.
Second, the separate encoder-decoder attention module and the decoder self-attention are merged into one attention, called mixed attention.
The mixed attention is hence used to extract information both from the source sentence and the previous target tokens.
Finally, joint source-target self-attention shares the parameters of attention and feed-forward layer between the encoder and decoder.
The use of mixed attention and the sharing of parameters make the encoder/decoder distinction artiﬁcial.
Indeed, we can see this learner as a "decoder-only" LM-like architecture that is adapted to performing seq2seq tasks in a way we described in the previous paragraph.
We use the same hyper-parameters of Transformer presented in the main text.
Preliminary experiments showed that joint source-target self-attention learners do not succeed in the Hierarchical-or-Linear and Composition-orMemorization tasks (i.e., less than 70% of the seeds learned the training set) when embedding size S emb
{+= 
+}
(the one used in the main experiments).
We hence set S emb = 64 .
The remaining hyper-parameters remain unchanged.
Note that, according to Table 5, there is no switch of biases for any setting when considering S emb = 64 compared to S emb = 16 .
Speciﬁcally for Transformer, the only difference is that with S emb = 64 , the latter shows even a stronger bias towards the hierarchical rule in the Hierarchical-or-Linear task.
Finally, similar to Sections B and C, we only consider the Count-or-Memorization task with l = 40 , the Add-or- Multiply task with l = 20 , the Hierarchical-or-Linear task with d = 4 and the Composition-or-Memorization task with M = 36 and run 20 seeds per setting.
Results are reported in Table 8.
[Table]
{+First, when considering the arithmetic tasks (i.e., Count-or-Memorization and Add-or-Multiplication ), we note a clear preference for the memorization rule (with a zero L and 1 . 
+}
FPA) similar to Transformer behavior (see Table 5).
Second, joint source-target self-attention learners are, similarly to Transformers, biased towards the hierar rule with 100% of the seeds generalized perfectly to the hierar rule and a signiﬁcantly lower L .
Finally, for the Compositional-or-Memorization , while only 1 seed generalized perfectly to the comp rule, L shows a signiﬁcant preference for the mem rule, in agreement with Transformers’ biases.
Overall, our experiments in this Section indicate that the inductive biases of the "decoder-only" Transformer learner are very close to that of a standard Transformer seq2seq learner.
On a higher level, we have demonstrated that our tasks and the description length measure can be applied for studying inductive biases of architectures that are often used in language modeling, indicating a high universality of the approach.