Quantifying Adaptability in Pre-trained Language Models with 500 Tasks
janeyu@
Meta AI mkhabsa@fb.com
Luke Zettlemoyer
Meta AI lsz@fb.com
Alon Halevy
Meta AI ayh@fb.com
Jacob Andreas
MIT jda@mit.edu
Abstract
When a neural language model (LM) is adapted to perform a new task, what aspects of the task predict the eventual performance of the model?
In NLP, systematic features of LM generalization to individual examples are well characterized, but systematic aspects of LM adaptability to new tasks are not nearly as well understood.
We present a large-scale empirical study of the features and limits of LM adaptability using a new benchmark, T ASK B ENCH 500, built from 500 procedurally generated sequence modeling tasks.
These tasks combine core aspects of language processing, including lexical semantics, sequence processing, memorization, logical reasoning, and world knowledge.
Using
T ASK B ENCH 500, we evaluate three facets of adaptability, ﬁnding that: (1) adaptation procedures differ dramatically in their ability to memorize small datasets; (2) within a subset of task types, adaptation procedures exhibit compositional adaptability to complex tasks; and (3) failure to match training label distributions is explained by mismatches in the intrinsic difﬁculty of predicting individual labels.
Our experiments show that adaptability to new tasks, like generalization to new examples, can be systematically described and understood, and we conclude with a discussion of additional aspects of adaptability that could be studied using the new benchmark.
Introduction
Much of the recent research effort in NLP has shifted from training task-speciﬁc models to adapting pre-trained language models (LMs) by ﬁnetuning their parameters or input prompts for downstream tasks (Devlin et al., 2019; Raffel et al., 2020; Li and Liang, 2021;
Lester et al., 2021).
This paradigm is general, in the sense that a large number of distinct NLP tasks beneﬁt from pretraining (Peters et al., 2018; Devlin et al., 2019;
Raffel et al., 2020).
But many questions about the nature and limits of LM adaptation remain unanswered.
For example: given a new task, can we predict how quickly (and how effectively) pre-trained
LMs can be adapted to perform it?
From among the variety of different adaptation techniques (e.g. ﬁnetuning or prompt-tuning), can we predict which one will be most effective?
Today, new pre-training and adaptation schemes are evaluated using small suites of curated tasks, typically featuring classiﬁcation, textual inference, and question answering (Wang et al., 2018, 2019).
These benchmarks have been extremely successful in identifying new tools for adaptation, but they are poorly suited for answering larger, structural questions like the ones above.
We present a large-scale study of LM adaptability using a new suite of benchmark tasks called
T ASK B ENCH 500.
1 T ASK B ENCH 500 consists of 500 procedurally generated tasks involving lexical semantics, factual information, memorization of random relations, list processing, and logical composition (Fig. 1).
Analogous to past work that uses synthetic data to characterize LM performance on single examples (Weston et al., 2016;
Lake and Baroni, 2018; Saxton et al., 2019; Kim and Linzen, 2020; Keysers et al., 2020; Liu et al., 2021a), T ASK B ENCH 500 enables systematic study of LM adaptability at the task level.
In this paper, we use it to study three aspects of adaptability:
Memorization : When can adaptation successfully memorize new functions (e.g., to update factual knowledge about entities, or learn arbitrary new token correspondences)?
We ﬁnd that LMs’ ability to memorize new input–output mappings is strongly inﬂuenced by task type .
Datasets of lexical relations (like antonym pairs) are easier to memorize than factual information (like name– occupation pairs).
Both are easier to memorize than lists of random word pairs.
These ﬁndings are particularly striking in the case of prompt tun-
1) Define tasks
[Figure]
2) Make datasets for each task
[Figure]
3) Adapt model to each task
Finetune
[Figure]
, which in standard conﬁgurations struggles to memorize even small random word pair lists.
Composition : Is LM performance on simple tasks predictive of their performance on compositions of those tasks?
(If the father and occupation relations are easy to learn via adaptation, does this imply that the father’s occupation relation is also easy to learn?)
We ﬁnd a nuanced answer.
LMs exhibit compositional adaptation to lexical and factual relations (like father’s occupation ), with success on composed tasks strongly correlated ( r 2 > 0 . 5 ) with success on atomic tasks.
However, when composing these relations with sequence processing operations, success on the base task does not predict success on the composed task.
Distribution matching :
In models ﬁne-tuned on datasets exhibiting a distribution of acceptable answers (e.g., translating ungendered pronouns into gendered ones), do model predictions match these distributions?
We ﬁnd that LMs are often unable to match label distributions in datasets used for adaptation .
In particular, when labels in the ﬁnetuning dataset are drawn from a uniform mixture of labels from two tasks (e.g., labeling half of the words with their antonym and half with their synonym ), models disproportionately assign mass to labels from the task that is easier to learn.
Each of these forms of adaptability corresponds to a central challenge in NLP: reliable updating of deployed models, composition of previously learned skills, and fair and predictable output from models trained on curated data.
Our study of memorization, composition, and distribution matching have direct analogs to previous studies of sample expressivity (Zhang et al., 2017), compositional generalization (Lake and Baroni, 2018; Kim and
Linzen, 2020; Keysers et al., 2020), and calibration (Guo et al., 2017).
However, we study these phenomena at the task level, rather than the example level.
Our experiments highlight important qualitative differences between current adaptation paradigms; identify several novel challenges for
LM adaptation, and offer a new benchmark for approaches aimed at meeting these challenges.
Background
Fine-tuning and prompt search
In languages for which large digitized corpora are available, most NLP system development today involves adaptation of a pre-trained model to a downstream task of interest.
Pre-training typically involves reconstruction of masked or corrupted text sampled from a large corpus (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020).
Adaptation to a new task typically involves one of three approaches: (1) ﬁne-tuning of all of a pre-trained model’s parameters (possibly in conjunction with a specialized decoder) on a task-speciﬁc training set (Devlin et al., 2019); (2) manual prompt engineering of an input template that induces pre-trained model predictions to perform the task of interest (Brown et al., 2020;
Petroni et al., 2019); or (3) automated prompt tuning of these templates, in either the discrete space of tokens (Shin et al., 2020) or con- tinuous space of token embeddings (Li and Liang, 2021; Lester et al., 2021;
Liu et al., 2021b).
The latter two approaches have grown more popular as pre-trained models have grown larger.
The performance of both prompt-search approaches still lags ﬁne-tuning (Raffel et al., 2020; Brown et al., 2020; Lester et al., 2021), though the difference between approaches appears to shrink as model size increases (Lester et al., 2021).
Measuring generalization and adaptability
The success of the training paradigm described above stems from its generality—a large number of NLP tasks appear to beneﬁt from some combination of pre-training and adaptation.
Previous attempts to quantify this generality have typically relied on benchmarks like GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019), each of which aggregates ten natural language processing tasks designed to probe different aspects of language understanding.
Similar benchmarks have also been built for non-English languages (Xu et al., 2020; Kakwani et al., 2020;
Park et al., 2021;
Hu et al., 2020).
However, the heterogeneity and small number of distinct tasks represented in existing benchmarks makes it difﬁcult to make ﬁner-grained predictions, e.g. by identifying speciﬁc features of tasks that contribute to the success or failure of adaptation.
This challenge has a direct analog to the problem of characterizing generalization at the example level in models trained for a single task.
Model performance on natural test sets is often loosely correlated with accuracy on individual examples featuring rare syntactic constructions or word collocations (McCoy et al., 2019).
A great deal of past work has focused on improving evaluation using synthetic evaluation sets (Jia and Liang, 2017; Naik et al., 2018;
Lake and Baroni, 2018; Richardson et al., 2020).
These datasets have been used to study long-range agreement (Marvin and Linzen, 2018), compositional generalization (Lake and Baroni, 2018;
Ruis et al., 2020; Keysers et al., 2020), and mathematical reasoning (Saxton et al., 2019).
But no analogous notion of systematicity, or tool for studying it, currently exists for tasks rather than examples.
Thus, building on this past work, we describe how to construct synthetic data distributions that enable systematic study of adaptation to new tasks rather than generalization to new examples .
Like previous research that uses procedural data generation procedures to study models in NLP, we focus on coverage rather than naturalness, using datasets designed to complement, rather than replace, existing naturalistic benchmarks.
A 500-task benchmark
Our goal is to study the generalizability of task adaptation paradigms.
In particular, we would like to identify which attributes of a task make it easy or difﬁcult to learn, across different models and training schemes.
While this work shares many of its high-level goals with existing benchmarks built from collections of real-world datasets, the makeup and difﬁculty of these datasets is often difﬁcult to characterize precisely: differences in annotation standards, annotation quality, and dataset size mean that models often exhibit very different performance on datasets designed to evaluate model performance on the same abstract task.
In addition, existing datasets cover an exceedingly small subset of the space of all tasks that future NLP practitioners might wish to perform.
To account for all these limitations, we propose to generate tasks synthetically as described below.
The space of tasks
T ASK B ENCH 500 is constructed compositionally: we begin by deﬁning a space of atomic tasks , which are combined using a set of composition operators to produce more complex tasks.
Every task takes as input a word or word sequence, and outputs either a boolean value or a set of words/word sequences.
We refer to any task that outputs booleans as a predicate task , and any task that outputs sets of words or word sequences as a relation task .
A subset of relation tasks involve modeling relations between single words at the input and output; we refer to these as word-level tasks and the remaining relation tasks (that take sequences as input or output) as sequential tasks .
The choice of atomic tasks and composition functions aims to capture aspects of real language processing tasks.
Accordingly, the set of atomic tasks comprises of:
Lexical tasks , which test knowledge of lexical semantics.
These include lexical relations like synonym , or lexical predicates like is-noun .
These tasks are constructed from WordNet relations (Fellbaum, 1998).
Factual tasks , which test factual knowledge.
These include factual relations like father-of , or factual predicates like
is-human .
These tasks are constructed from Wikidata properties (Vrandeˇci´c and Krötzsch, 2014).
Random relation tasks , which test memorization ability.
These are created by mapping a word in the vocabulary to a singleton set containing a random other word.
We createrandom relations with different random seeds.
To recursively create arbitrarily complex tasks, we deﬁne a set of composition functions .
These take tasks as arguments and return other tasks.
These functions fall into two categories:
Word-level compositions , which test ability to combine word-level information in different ways, such as through set or logical operations.
These functions take word-level tasks and return other word-level tasks.
Examples include intersection and chaining .
Sequential compositions , which test ability to operate on sequences.
These functions convert word-level tasks to sequence-level tasks.
There are two functions in this category: map takes a word-level relation task and returns a task that maps a sequence of n words to a set of all possible sequences resulting from
applying f W to each input word.
filter takes word-level predicate tasks and returns a sequence consisting only of words for which the task returns true , preserving the original ordering of those words.
The full list of atomic tasks and composition function can be found in Appendix Tables 4 and 5.
We surmise that typical NLP tasks may require some combination of lexical knowledge, factual knowledge, sequential processing, and other task-speciﬁc reasoning; our data distribution lets us evaluate all these aspects separately and in combination.
Datasets for tasks
We create datasets D ( f ) =
{ ( x i , y i ) : x ∼ X f , y ∼ Unif ( f ( x i )) } for each task f , where X f is the input distribution for the task, and recalling that f ( x i ) returns a set of possible outputs associated with the input x i .
For all tasks, we randomly partition the dataset into
D train ( f ) and D test ( f ) splits.
[Table] [Table] [Table] For lexical atomic tasks and their compositions, we directly use the most common words in the task’s input language for X f .
We create tasks in
English and Spanish.
For factual atomic tasks and their compositions, we sample the entities from
Wikidata that participate in the relation or predicate deﬁned by the task (e.g. for the child-of task, we sample only entities that have children).
For sequential tasks , we use a random sampler, which samples n random words from the vocabulary and concatenates them.
Figure 1 shows examples of tasks and associated datasets.
More details on dataset construction can be found in Appendix A.
Experimental Setup Model & Training
For all experiments, we adapt a pre-trained T5-base model (Raffel et al., 2020).
We examine two types of training paradigms: ﬁne-tuning and prompt-tuning.
During ﬁne-tuning, we update all model parameters on the training set.
During prompt-tuning, we follow Lester et al.
and introduce a new set of prompt-tokens { p 1 , · · · , p n } to the vocabulary, which will be prepended to every sample from the task during inference, i.e., each sample input x becomes p 1 p 2 · · · p n x .
Let θ denote the parameters of the original pretrained LM.
During training, the entire model is frozen and only the word embeddings of the new tokens { θ p 1 , · · · , θ p n } ⊂ θ are updated.
We use prompts of length n = 100 for all experiments.
We also study each paradigm on various quantities of training data, and separately evaluate their memorization and generalization adaptabilities.
In particular, for word-level tasks the test-set words are disjoint from the train-set words, so evaluating on the test set will strictly measure generalization capacity.
We optimize all models using
AdamW. See Appendix B for full hyperparameters
. Evaluation
For each task f and model M [ θ ] (with parameters θ ), we measure the model’s average per-token accuracy on both training and test splits of the dataset D ( f ) .
As each task deﬁnes multiple acceptable outputs for each input, we credit models for producing any acceptable output.
Letting y (cid:48)
=
M ( x ) , we measure the fraction of positions i at which any valid answer y i matches the
[Figure]
predicted y (cid:48)
i :
[Equation]
Further details can be found in Appendix B.
Given a pretrained model M [ θ pretrain ] , an adaptation procedure T , and a task suite f , let
M [ θ T , D ( f ) ] denote the model trained using T on training data D ( f ) .
We then deﬁne the adaptability of a (pretrained model, adaptation paradigm, task suite) as:
[Equation]
We denote by adapt mem the value of this metric over training data ( D eval =
D train ), and by adapt gen the metric over test data ( D eval
= D test ).
Memorizing datasets
Our ﬁrst experiment investigates the extent to which task adaptation paradigms can memorize different types of tasks.
We are interested in memorization because many real NLP tasks involve some degree of memorization.
For example, translation builds on memorizing lexical associations between words in various languages, and semantic similarity and paraphrasing require memorizing word meanings and/or groupings of semantically similar words.
Method
We use training-set adaptability ( adapt mem ) as an indicator of a model’s memorization ability (Fig. 2).
We train on a set ofexamples, and plot the value of Eq.
(2) on each atomic task as models are adapted via ﬁne-tuning or prompt-tuning.
This allows us to visualize both the ﬁnal training-set performance, as well as the time it took to achieve that performance, both of which we use to quantify memorization ability.
Results
Figure 2 shows the training curves for ﬁne-tuning (left) and prompt-tuning (right), on different types of tasks.
Solid lines show adapt mem , while dashed lines show adapt gen .
Under both adaptation paradigms, we ﬁnd that lexical tasks are easier to memorize than factual tasks, while random tasks are the hardest to memorize.
However, for ﬁne-tuning, we ﬁnd that models can (eventually) learn to perfectly memorize all types of tasks—even entirely random word associations.
However, different types of tasks converge at different rates—lexical tasks converge ﬁrst, followed by factual tasks, followed by random tasks.
Prompt-tuning, with many fewer parameters than ﬁne-tuning, is much less expressive.
As shown in Fig.
2, none of the tasks types converge to 100% accuracy across tasks.
Prompt-tuning overall also takes signiﬁcantly longer to converge; in particular, on random tasks, the ﬁnetuned model generally converges at ∼ 20k updates, while the prompt-tuned model takes over 200k updates to even begin performing nontrivially.
[Table]
However, despite being much worse at memorization, prompt-tuned models still generalize almost as well as fully ﬁne-tuned models, at least on atomic tasks.
This suggests that the inability to memorize arbitrary functions is not necessarily a problem for prompt-tuning in general, and more broadly that overﬁtting the training set—at least during ﬁne-tuning—may not be necessary for generalization.
In Appendix E, we run a version of this experiment on permuted task labels in order to better disentangle the effect of learning novel tasks vs. retrieving existing ones.
We ﬁnd that, for both prompt-tuning and ﬁne-tuning, pre-trained models can more easily adapt to existing relations than to novel (permuted) ones, but they are still able to adapt to new tasks, especially compared to non-pretrained models.
Composing tasks
In the previous section, we found that while prompttuning cannot memorize arbitrary tasks like ﬁnetuning, it can still generalize well on simple atomic tasks, almost comparably to ﬁne-tuning.
In this section we investigate whether this ﬁnding extends to more complex tasks.
Speciﬁcally, we examine the behavior of prompt-tuned and ﬁne-tuned models when adapted to compositions of atomic tasks.
Many prior studies of compositionality focus on instance-level compositionality (Lake and Baroni, 2018; Keysers et al., 2020): they test whether models can generalize to new instances by combining information from previously-seen instances within the same task .
For example, Lake and Baroni (2018) study whether models can learn to jump left , after learning to jump , run , and run left .
In our work, we instead focus on task-level compositionality, studying whether models can adapt to new tasks that are compositions of simpler tasks on which they are known to perform well.
Thus, while a model exhibiting compositional generalization will correctly compose fragments of previously observed training examples, a training procedure exhibiting compositional adaptability will perform well on tasks involving compositions of previously learned skills.
Method
We study adaptation to complex tasks by relating performance on atomic tasks with performance on depth-2 compositional tasks .
We also study each paradigm under few-shot learning, by creating a random 32-sample subset of each training dataset, and training on that subset.
To mitigate the effect of the random seed, we report average performance over 4 different subsets.
What allows models to adapt to these complex tasks?
We hypothesize that their adaptability is (in part) compositional—when they can adapt to simple tasks, they can also adapt to compositions of those tasks.
For each training paradigm T and each composition function C , we run linear regression to estimate the Pearson correlation coefﬁcient r 2 between adaptability to a compositional task
C
[Equation]
and average adaptability to the task’s atomic components,
[Equation]
Figure 3 depicts the procedure graphically.
Can language models learn compositional tasks?
The average model adaptability to compositional and atomic tasks, under each training paradigm, is reported in Table 1.
We observe that the gap between full-data prompt-tuned models and full-data ﬁne-tuned ones is much larger on compositional tasks than atomic ones.
Thus, prompttuned models can only generalize comparably to ﬁnetuned ones for sufﬁciently “simple” tasks.
Interestingly, this distinction disappears under few-shot learning.
Though both adaptation paradigms generalize much worse in the few-shot
[Figure] [Figure]
[Figure]
setting compared to the full setting, they appear to be comparable to each other in the few-shot setting, even on compositional tasks.
This may simply imply that few examples are insufﬁcient to learn the nuances of complex tasks, and that simply learning a few prompt tokens is sufﬁcient to capture what can be learned from the limited data samples.
Do language models adapt compositionally?
We visualize each regression model in Fig.
Higher r 2 indicates higher correlation between atomic and compositional versions of tasks.
Note that all model training paradigms demonstrate some degree of word-level compositionality ( r 2 > 0 . 5 )—when they succeed at word-level compositional tasks ( union , chaining ), they succeed at the atomic constituents to those tasks, and vice versa.
However, this does not appear to be the case for sequential map .
In the full-data regime, both ﬁne-tuning and prompt-tuning have near-zero rvalues.
In the few-shot regime, the r 2 value, while nontrivial, is also quite low.
Note the slopes of the learned regression lines—the model appears to be unable to learn the sequential versions of tasks, despite succeeding at their atomic versions.
To explain this result, we hypothesize that a major obstacle to sequence-level compositional adaptability is segmentation of sequences into atomic units.
This is especially the case for factual tasks: for example, the sequence Pauline Payne Whitney
Charles Lloyd could be segmented as [Pauline
Payne Whitney] [Charles Lloyd] or [Pauline
Payne] [Whitney Charles Lloyd] , etc.
To test whether segmentation is a bottleneck, we train on a version of sequential tasks where we give the language model explicit markers of word/entity boundaries (e.g. the language model is given Pauline
Payne Whitney # Charles Lloyd as input).
We found that, with separators, performance on the map tasks increases substantially and the model demonstrates compositional adaptability ( r 2 > 0 . 5 ) to these tasks in 3 of the 4 adaptation paradigms.
This setting is plotted in Fig.
3 as Map (+separators) .
Under this setting, full ﬁne-tuning is the only training paradigm that does not demonstrate compositional adaptability.
To better understand this phenomenon, we exclusively plot points from the
Map (+separators) setting in Appendix Fig.
We ﬁnd that the distribution of points in the full ﬁnetuning case shows that points tend to fall within the upper-left triangle.
This indicates that for a signiﬁcant number of tasks, models adapt to their sequential versions despite failing at atomic versions.
In these cases, the model does not simply adapt com- [Figure] positionally, but can take advantage of additional information present in sequences (e.g., seeing more tokens, more examples of the word-level function) to outperform compositional adaptation.
Learning new distributions
Previous sections investigated the degree to which models could ﬁt particular tasks using a binary metric that assigned credit to any acceptable answer.
Our ﬁnal set of experiments explores a ﬁnergrained notion of correctness: when there are multiple acceptable answers, as is often the case in real
NLP tasks, when does the output distribution of a model match the distribution empirically observed during adaptation?
Method
We speciﬁcally investigate whether models are biased towards predicting “easy” labels, in the sense measured in Section 5.
We consider all possible pairs of atomic tasks f 1 , f 2 (for which f 1 and f 2 take in overlapping sets of inputs).
Let f e to be the easier task in this pair and f h be the harder task, relative to a model
M and training paradigm T , in the sense that adapt gen ( M , T , f e ) > adapt gen ( M , T , f h ) .
We compose f e and f h using union to create compositional task ∪ ( f e , f h ) , and construct the training dataset for this task to be balanced — such that the model sees an equal number of examples of form ( x, f e ( x )) as ( x, f h ( x )) .
Now let M ∪ ( f e ,f h ) denote a model adapted to this task.
During test-time, we provide M ∪ ( f e ,f h ) with novel inputs x (cid:48) from the domain of both f e and f h , and record the average probability mass it assigns to all y ie ∈ f e ( x (cid:48) ) versus all
y ih ∈
f h ( x (cid:48) ) .
Finally, we average these datasetwide probabilities over all pairs of tasks, to get an aggregated probability mass assigned to all easier tasks and all harder tasks in a task pair, invariant of the actual underlying task identity.
More details on this procedure can be found in Appendix D.
Results
Overall, as seen in Fig.
4, across all tasks and training paradigms, the model tends to assign a higher probability to the easier relation.
As a concrete example, when trained to predict either antonyms or lexical entailments, the average probability mass placed on the antonyms of a word from the held-out set (easier relation) is 13%, while the average probability mass placed on the entailments of a word (harder relation) is 8%.
Thus, despite having a perfectly balanced ﬁnetuning set, pretrained models still predict label distributions in a way that align with their inductive biases (measured via the “intrinsic difﬁculty” of individual labels).
This holds for all task adaptation methods, including full ﬁne-tuning, meaning even paradigms and models that can ﬁt more complex tasks still have residual biases from pretraining that affect their predictions.
This also suggests wider-reaching consequences for model fairness and equity: simply debiasing a ﬁne-tuning dataset is insufﬁcient to overcome biases from pretraining.
Conclusion
In this paper, we construct T ASK B ENCH 500, a synthetic task set which serves as a testbed for task adaptability.
We focus on three axes of adaptability: ability to memorize, ability to (compositionally) generalize, and ability to ﬁt to novel distributions.
We study two adaptation paradigms: ﬁnetuning and prompt-tuning, ﬁnding that: 1. unlike ﬁne-tuning, prompt-tuning cannot memorize completely arbitrary tasks beyond a small number of examples, 2.
all adaptation paradigms demonstrate compositional adaptation to word-level compositions, but not sequence-level compositions, and 3.
no paradigm is able to perfectly replicate the downstream distribution—all paradigms learn output distributions that align with its inductive biases.
In future work, T ASK B ENCH 500 can be used to study other factors that may affect adaptability, such as length of the prompt in prompt-tuning, similarity between the task distribution and the pretraining distribution, or ﬁner-grained distinctions between tasks (beyond lexical/factual/random, or composition type) that predict task adaptability.
T ASK B ENCH 500 can also be used to explore the limitations of prompt engineering on a GPT3-scale model.
Finally, the current set of tasks and primitives in T ASK B ENCH 500 are by no means complete.
Future work can expand on these primitives and study the relationships between the tasks put forth here and real NLP tasks.
Acknowledgements
This work was supported by the MIT–IBM Watson AI lab.
Part of the work was done using computing resources provided by a hardware donation from
NVIDIA under the NVAIL program, and by the
Lincoln Laboratory Supercloud.
BZL is supported by an NDSEG Fellowship.
Impact Statement
This paper introduces a new procedure for deﬁning task suites.
This procedure is then used to create a 500-task benchmark, which measures the adaptability of pre-trained language models to new tasks.
Because the benchmark is created procedurally from databases of words and entities, we anticipate that there should be little to no identifying information or toxic and hateful content.
Our datasets should also contain less social bias compared to natural datasets.
However, like with all benchmarks, overﬁtting to static datasets can inhibit progress in NLP.
Moreover, even though this dataset is procedurally generated, we cannot avoid all biases.
The resources upon we build our benchmark are themselves biased—for example, lexical databases (like WordNet) are much richer for certain languages (like
English) than others, and WikiData currently features many more men than women.
Our benchmark currently only features English and Spanish tasks, with a heavy bias towards standard English.
This can encourage development of methods that underserve non-standard-English-speaking communities.
We hope to mitigate the aforementioned issues by releasing the code to procedurally generate task suites.
We emphasize that the benchmark is dynamic: consisting of not just the static task suite that we are currently releasing, but more importantly the procedure for creating new tasks suites.
We encourage future researchers to develop analogous task suites for low-resource languages, nonstandard English dialects, and more balanced sets of entities.
A More details on T ASK B ENCHcreation procedure A. Task creation details
For atomic lexical tasks, we take a subset of relations speciﬁed in either Wordnet (Fellbaum, 1998) or SentiWordNet (Esuli and Sebastiani, 2006).
For atomic factual tasks, we take a subset of tasks from
Wikidata (Vrandeˇci´c and Krötzsch, 2014).
We also have 3 broad categories of composition functions: set operations, logical operations, and sequential operations.
The full list of atomic tasks can be found in Table 4 and the list of composition functions can be found in Table 5.
We enumerate all possible depth-2 word level compositions of each task, and the sequential versions of them (i.e. if the task is a relation, inserting it into a map , or if the task is a predicate, inserting it into a filter ), up to 500 tasks.
We also apply some basic heuristics to ﬁlter identical tasks: for example, we ﬁlter symmetric relations, e.g. union(B,A) is identical to union(A,B) , or avoid the use of logical operations alongside set operations, e.g. lor(in(x,A), in(x,B)) is identical to in(x,union(A,B)) ).
Our full list of tasks can be found in Tables 4, 6, 7 and 8.
Sequential compositions
Sequential composition functions convert word-wise tasks to sequencelevel tasks.
We speciﬁcally consider only two sequential functions: map and filter .
Note that compositions of multiple map s or multiple filter s can instead be expressed as compositions of multiple word-level functions.
For example, map{ λ x.occupation(x)}(map{ λ x. father(x)}(S))
(for an input sequence S ) is equivalent to map{ λ x.occupation(father(x))}(S)
Speciﬁcally, we deﬁne the following top-level sequential operator
[Equation]
where f M is a word-wise relation and f F is a wordwise predicate.
All recursively-deﬁned sequential operators follow this form.
The following are the recursive rules for mapping nested maps and ﬁlters into a function of this form: in the base cases, map
[Equation]
in the recursive cases,
[Equation]
A. Dataset creation details
Note that many tasks created through composition will be degenerate or identical to other tasks, even with our heuristic ﬁlters.
We do a preliminary ﬁlter for degenerate tasks by removing tasks for which we have less than 100 samples.
We also manually inspect all depth-2 word-level lexical compositions to ensure they are nontrivial and unique.
Word-level lexical tasks
For English lexical tasks, we use words that appeared more thantimes in the Brown corpus (Francis and Kucera, 1979) as our inputs x .
For Spanish lexical tasks, we in use words that appeared at least once in the
CESS Spanish Treebank (Martí et al., 2007) as our inputs.
This results in a a total of 9143 English words and 5298 Spanish words.
We then construct outputs for each input word using either WordNet or SentiWordNet.
From each task, we ﬁlter out samples for which the relations map to an empty set— thus, for a task like intersection(synonym(x), antonym(x)) , most samples will be ﬁltered out as the set of synonyms are usually disjoint from the set of antonyms.
(This task ends up getting ﬁltered out entirely, as the ﬁnal number of samples is under 100.)
Word-level factual tasks
We use a dump of
Wikidata from 2017, taken from Sorokin and
Gurevych (2018).
5 We convert each word-level factual task into SPARQL queries which returns a set of input-output data pairs from Wikidata.
For factual relations R , we create two queries: a sample query which gives us a set of entities that participate in the relation, from which the inputs x are derived, and a function query that maps speciﬁc inputs x to its set of output entities R ( x ) .
For factual predicates P , we create three queries: a positive sample query which gives samples x for which P ( x ) = true , a negative sample query which gives samples x for which P ( x ) = false ,
[Table]
and a function query that maps speciﬁc inputs x to its output boolean value P ( x ) .
The SPARQL query is generated recursively given the speciﬁcation of the task.
We deﬁne a function task2sparql(T(x),y) which converts tasks
T(x) to SPARQL fragments (where the second argument to the function is the variable name we deﬁne for the output).
We then convert the output of this function into a well-formed query using:
SELECT ?
x
WHERE <task2sparql(T(x),y)> for sample queries and
SELECT ?
y
WHERE <task2sparql(T(x),y)> for function queries.
Note for function queries that the input x is provided to us (and is not a variable).
The general rules specifying the task2sparql function are given in Table 2.
Sequential tasks
In practice, naively concatenating outputs from a random word sampler to create sequences will return degenerate or trivial sequences for many functions (for example, map{ λ x. child(x)} is not meaningful for sequences consisting of words that don’t refer to humans).
Thus, we deﬁne a sequence sampler in Algorithm 1 that takes in a sequential function (given in the form from Eq.
(5)), an input length n and an output length m ≤ n , which will always sample sequences with length n such that the output, when the function is applied to the sequence, is of length m .
At a high level, this algorithm samples n input words which are in the domain of the map relation, and for which the ﬁlter predicate returns true , and m − n input words for which the ﬁlter predicate returns false , then permutes and concatenates them.
B Experimental Setup Details Hyperparameters
We adapt a pre-trained T5base model (24-layer, 220M parameters) to our [Table] tasks.
We use an AdamW optimizer with a learning rate of 1.0 for all prompt-tuning experiments, and learning rate of 1e-3 for all ﬁne-tuning experiments.
We use batch sizes of 64 for word-level tasks, and 32 for sequential tasks.
We run all experiments up to 100 epochs, and run 3–4 trials for each fewshot experiment to estimate average performance over possible choices of few-shot training samples.
These hyperparameters were chosen by trial and error on top of default conﬁgurations.
Reproducibility
For each task, we adapt our model using a single 32GB
NVIDIA V100 GPU, or a single 40GB NVIDIA
A100 GPU.
Training time varies by training dataset size and maximum number of epochs, but on average (using the hyperparameters speciﬁed above) is less than a few hours per task.
Prompt-tuning is also more efﬁcient than ﬁne-tuning, updating the parameters of only 100 prompt tokens vs. the full 220M parameters in the model.
Evaluation of Sequential Tasks
When evaluating accuracies of sequential tasks (Eq.
(1)), note that we must align words in the generated sequence y (cid:48)
i with words in the ground-truth sequence y i .
However, this can be nontrivial, especially under the setting where word and entity boundaries are not explicitly generated by the model.
We cannot rely on whitespaces to segment words as a single word can span multiple white-spaces; for example, an entity Will Smith constitutes a single word.
Instead, given a ground-truth sequence of n words (note ground-truth segmentations are present in the dataset), we optimize over all possible length- n segmentations of the generated sequence.
C Compositionality Experiment: Additional Results
Additional results for the compositionality experiment, including all composition functions, and the formula for the best-ﬁt regression line in each case, are reported in Table 3.
Furthermore, the map task with explicit segmentation (+separators) is plotted in isolation in Fig.
D Prediction distribution experiment: Additional details
We adapt the model to the task ∪ ( f e , f h ) , constructing the training dataset for ∪ ( f e , f h ) to be balanced — such that the model sees an equal number of examples of form ( x, f e ( x )) as ( x, f h ( x )) .
Let M ∪ ( f e ,f h ) denote a model adapted to this task.
Note that the domains of either function are not always identical, for example the set of entities in the domain of political-party-of(x) (mostly politicians) is different from the set of entities in the domain of position-played-on-sports-team(x) (mostly athletes).
We create a balanced training set by ﬁrst taking all items in the intersection of both domains, then sampling an equal number number of items in either domain.
Furthermore, to minimize the effect of the order seen during training, we shufﬂe the entire dataset after creating all example-label pairs.
During test-time, we give M ∪ ( f e ,f h ) a novel input x (cid:48) and record the average probability mass it assigned to all y ie ∈ f e ( x (cid:48) ) vs. all y ih ∈
f h ( x (cid:48) ) .
Note we evaluate only on inputs x (cid:48) which are in the domain of both f e and f h .
Under the rare scenario that a prediction is in both target tasks for a particular word (i.e. y is in both f e ( x (cid:48) ) and f h ( x (cid:48) ) ), we count that towards both tasks, and increment the probability mass on both tasks by the probability the model assigned to y .
Instead of averaging across outputs in either set f e ( x (cid:48) ) , f h ( x (cid:48) ) , we also looked at the probabilities assigned to highest -scoring predictions from each set.
The overall trends were similar: the model tends to assign greater mass to the highest-scoring prediction from the easier task compared to highestscoring prediction from the harder task.
E Permuting task labels: disentangling effect of “learning” vs. “retrieval”
We hypothesize two ways that pre-trained models might adapt to new tasks: (1) through learning the underlying rules and patterns governing the task, or (2) through learning how to “retrieve” the correct label from memorized pre-training data.
These hy-
[Figure]
potheses, respectively, suggest two different roles for pre-training: (1) providing a “generally good” initialization from which many different tasks can be learned, or (2) imbuing the LM with memorized knowledge that can later be retrieved.
To determine which effect is at play (for which types of tasks), for each atomic task, we permute the labels associated with each input, then run each adaptation paradigm on the permuted version of the task.
Notably, permuted labels differ from random tasks as the input and label distributions are restricted to be identical to original task.
Because the model would be unable to generalize to permuted labels, we only look at memorization ability.
The setting is similar to Section 5.
We compare the rate of adaptation for (A) a non-pretrained model to a permuted task, (B) a pre-trained model to a permuted task, and (C) a pre-trained model to nonpermuted task.
If a pre-trained model is better able to adapt to a task than the non-pretrained model (B > A) , this indicates that pre-training helps models learn new tasks on the ﬂy, supporting hypothesis 1.
If a pre-trained model can better adapt to a nonpermuted task than it can to a permuted task (C > B), this indicates that adaptation requires something learned during pre-training, supporting hypothesis 2.
Results are shown in Fig.
6 (which, from left to right, shows settings A-C respectively).
We ﬁnd that for ﬁne-tuning and prompt-tuning, both hypotheses are supported.
For both lexical and factual tasks, pre-trained models can memorize novel word relations faster than non-pre-trained models.
However, pre-trained models can still adapt to nonpermuted tasks faster than permuted ones.
Furthermore, note that for ﬁne-tuning, the order of convergence of the three task types is reversed when going from permuted tasks to non-permuted tasks.
In particular, random relations are easier to learn than permuted lexical or factual tasks.
This suggests that models can more easily to adapt to random labels than labels that are known to be false .
[Table]
.f
M
[Table]
x )
[Table]
separators
[Table] [Table]
functions and their compositions, under various training paradigms.
We train a linear regressor to predict a model’s generalization adaptability on a composite function based on its adaptabilities on the atomic constituents.
Finally, we report the average generalization adapt- ability of composite tasks, for each training paradigm, under each type of composition.
[Table]
[Table] [Table]
[Table]
expectinexpensivedirectbonessullenbreedswitchingeight
[Table]
bonesbreedswitchingeight
[Table] [Table]
[Table] [Table]