What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environment (2022)

Abstract: The instruction learning paradigm—where a model learns to perform new tasks from task descriptions alone—has become popular in research on general-purpose models. The capabilities of large transformer models as instruction learners, however, remain poorly understood. We use a controlled synthetic environment to characterize such capabilities. Specifically, we use the task of deciding whether a given string matches a regular expression (viewed as an instruction) to identify properties of tasks, instructions, and instances that make instruction learning challenging. For instance, we find that our model, a fine-tuned T5-based text2text transformer, struggles with large regular languages, suggesting that less precise instructions are challenging for models. Instruction executions that require tracking longer contexts of prior steps are also difficult. We use our findings to systematically construct a challenging instruction learning dataset, which we call Hard RegSet. Fine-tuning on Hard RegSet, our large transformer learns to correctly interpret (with at least 90% accuracy) only 65.6% of test instructions, and 11%-24% of the instructions in out-of-distribution generalization settings. We thus propose Hard RegSet as a challenging instruction learning dataset, and a controlled environment for studying instruction learning.

## introduction recent years have seen an increased interest in instruction learning @xcite where a model learns to perform unseen tasks at test time in a zero-shot manner given only a prompt containing instructions. this style of learning is an important feature of flexible and general intelligent systems. instruction learning stands in contrast to the traditional machine learning paradigm of example learning. in example learning, the model has access to input-output pairs during training time. for instance, in sentiment analysis with the goal to classify product reviews, the model has access to many examples of labeled reviews from which to learn the task. in instruction learning, a model that has never seen labeled sentiment analysis data must perform sentiment analysis given only the explicit instruction "tell me whether this review is positive". in other words, the model learns to interpret the instruction language (here english) in order to execute an instruction for a task it has never seen. most recent work on instruction learning has been conducted in the context of natural language instructions (e.g., @xcite @xcite . in this context, the complexity of natural language makes it difficult to draw clear conclusions about the kinds of instructions transformers can learn to interpret. to remedy this, we adopt a synthetic approach by building a controlled instructional environment based on interpreting regular expressions (regexs), and use well-studied properties of regexs to characterize transformer instruction learning capabilities. importantly, while findings on synthetic data do not necessarily translate to the real world, our work identifies potentially in-teresting hypotheses that can be further investigated on natural data. previous work on datasets such as @xcite and rule-taker @xcite has also taken a similar approach of simplifying a difficult, fuzzy problem into a synthetic one. synthetic generation is a lowcost alternative to large-scale dataset building that allows a high level of formal precision that is infeasible with human-authored data. this allows us to distill atomic characteristics of instruction following in the synthetic domain and make informed conjectures about the natural language setting. a regex is a specification of a formal language, i.e., a set of strings of symbols. to avoid the confusion between an instructional language such as english and a formal language specified by a regex, we refer to the latter as an r-language (for regular language). in our work, we view a regex as an instruction for the task of deciding whether a string belongs to the r-language of the regex. we choose to study regexs because they are well known, unambiguous (the r-language decision problem is binary and always well-defined), and easy to compute (there exist linear-time algorithms to recognize whether a string belongs to a regex), while also incorporating fundamental computational operations including iteration, disjunction, conjunction, and nesting. this environment allows us to study instruction learning phenomena more precisely. we construct regset, a regex instruction learning dataset ( §4) with instances of the form (regex, string) → t/f, and find that a large t5-based text2text model fine-tuned on regset is unable to correctly interpret many instructions in the test set. inspecting the r-languages, regexs, and string instances, we identify a number of properties that predict which regex instructions the model struggles with ( §6). selecting regexs with these properties, we construct a hard variant of the dataset which we call hard regset ( §7). our fine-tuned model achieves good performance on only 65.6% of the test regexs in hard regset, leaving room for improvement. we find that instruction learning models struggle with non-starfree r-languages. this provides evidence that even large transformers struggle with periodic r-languages, a theoretical limit of transformers' attention mechanism @xcite ) that @xcite further study in smaller models, in the example learning setting. our findings in §6 suggest four general implica-tions that we expect will extend beyond the synthetic regex environment. first, instruction learning is harder when the underlying tasks require modular counting (e.g., keeping track of whether a quantity is even or odd). second, it gets harder as the instructions themselves become less precise, that is, they can be executed correctly in several different ways, forcing the execution engine to make and track choices. third, as expected, it's harder for instructions involving executing and composing many individual operations or steps. lastly, it's harder when the correct execution of instructions requires keeping in memory a larger context of the partial execution thus far and making choices dependent on this longer history. in summary, our main contributions are: ## related work learning instructions. large language models like gpt3 @xcite have some ability to understand task instructions expressed in natural language @xcite . this has led to research on whether models can be fine-tuned to follow instructions reliably, creating the sub-field of instruction learning. our formulation and approach aligns with @xcite who build zest, a benchmark for natural language instruction following, and report that a fine-tuned t5 model @xcite does poorly. we differ in the use of a highly-controlled synthetic environment that allows analyzing formal properties of instructions. numerous studies @xcite @xcite have followed up with improved models. though our work differs in domain and goals, we adopt their approach of fine-tuning on instruction-annotated datasets to train an instruction learner. earlier work has shown that transformers can emulate (the results of) deductive reasoning over a set of natural language-like rules with limited expressivity @xcite . similarly, regex instructions include compositions of simple operations such as disjunction and concatenation as well as more complex rules such as iteration (the kleene star "*" operator) and nesting ("()"). ## regex datasets our datasets consist of collections of triples according to the formulation given in §3.2, where each triple contains a regex r, a string @xmath0, b} * , and a label ℓ ∈ {0, 1} indicating whether @xmath1. some examples are given in we sample regexs r such that all corresponding r-languages l r are distinct within the dataset. furthermore, r is chosen such that it uses the minimum number of compositional operators needed to represent l r . see appendix b for details of our regex sampling method. constraining the regexs to express unique r-languages avoids over-representing only a few r-languages, as would happen with a naive sample. we limit number of compositional operators in each regex to at most 6, sampling approximately uniformly at random with respect to the number of operators. for each r, we sample both strings that match r and strings that do not. 4 we limit string length to at most 15 and sample from each length approximately uniformly at random. ## experimental setup model. we select byt5-large (1.2b parameters) @xcite as our base model 6 . byt5 is a t5-based pre-trained transformer model with character-level tokenization-which is helpful in avoiding issues with tokenizing synthetic strings. we use a pre-trained model because pre-training often imbues performance benefits even for unrelated tasks @xcite . 7 we feed r x to the model as a string (separated with a space), and train the model to output the strings "true" or "false". we train for 200 epochs, using validation accuracy to select the best model, using a learning rate of 5 • 10 -5 and batch size 32. 8 metrics. consider a regex evaluation set d, and a model m . we define m 's accuracy on an instance (r, x) ∈ d, denoted acc m (r, x), as 1 if m correctly predicts whether x matches r, and 0 otherwise. m 's accuracy on a regex r is acc m (r) = mean x:(r,x)∈d acc m (r, x). to measure how well m learns to interpret each regex we use metrics that operate at the level of regexs. due to the synthetic-and thus noisefree-nature of our datasets, we expect m to perfectly interpret every expression given sufficient training data, at least in the i.i.d. setting and also in reasonable generalization settings. to measure how well m does relative to this upper bound, we define two metrics that give equal weight to all regexs in d, regardless of how many test strings each has. our main metric is mean regex performance at k, defined as the fraction of regexs on which m 's accuracy is at least k (treated as a percentage): as a shorthand for {r | (r, x) ∈ d}. we will drop the subscript m from perf m when the model is clear from the context. perf@100 thus refers to the fraction of regexs r that are learned perfectly (as assessed by all strings tested for r in d). since this metric is somewhat strict from a machine learning perspective, we use perf @90 as our main metric, and also track a more lenient metric, perf@80. we also report a secondary metric, mean regex accuracy, defined as m 's accuracy on a regex r, averaged across all r present in d: acc @xmath2. as before, we drop the subscript m from acc m when the model is clear from the context. note that this metric does not distinguish the case of two regexs learned to accuracies of 90% and 50% from two regexs learned to an accuracy of 70% each. further, when each regex in the evaluation set has the same number of strings, acc m simplifies to the standard instance-level accuracy of the dataset rather than a regex-level metric. for these reasons, this metric is less desirable, but we include it for completeness. we train our model on the exploration train set and evaluate on the exploration test set. the model achieves a high mean regex accuracy acc of 97.1%. the number of regexs learned to at least 90% accuracy (perf@90), however, is a more modest 89.6%. by design, the exploration test set's large size allows us to analyze errors in detail and assess which attributes contribute to the hardness of instruction learning. ## hard regset: a new challenge based on our findings, we select the attributes that contribute most to difficulty, namely starfreeness (or rather, non-starfreeness), r-language size, and es; following the idea of salient variable sampling @xcite . in selecting attributes we balance the trade-off between accuracy reduction and aggressiveness of the filter and ignore attributes with small effects like number of compositions. filtering the exploration test set for non-starfree expressions with @xmath3.2. the 8 lowest-scoring regexs from this group are shown in hard regset is split into train, validation, and test sets foot_4 which match the sizes of the exploration train, validation, and test sets. ## conclusion instruction learning is an important step towards general-purpose ai systems. understanding the limits and capabilities of current instruction learning models is key to improving them. our exploration regset dataset provide a controlled environment for discovering what makes instruction learning hard for today's transformers. we identify several such attributes in our setting and use our findings to make informed speculations about the difficulty of instruction learning in general. we also use our findings to systematically construct hard regset, a challenging synthetic instruction learning benchmark. our t5-large based model leaves much room for improvement on this dataset, failing to meet the 90% accuracy bar in over 34% of the regexs in an iid setting and faring much worse in our ood generalization settings. we offer our dataset as a challenge to help the community progress towards building reliable instruction learning systems.