Symbol tuning improves in-context learning in language models (2023)

Abstract: We present symbol tuning - finetuning language models on in-context input-label pairs where natural language labels (e.g., “positive/negative sentiment”) are replaced with arbitrary symbols (e.g., “foo/bar”). Symbol tuning leverages the intuition that when a model cannot use instructions or natural language labels to figure out a task, it must instead do so by learning the input-label mappings. We experiment with symbol tuning across PaLM models up to 540B parameters and observe benefits across various settings. First, symbol tuning boosts performance on unseen in-context learning tasks and is much more robust to underspecified prompts, such as those without instructions or without natural language labels. Second, symbol-tuned models are much stronger at algorithmic reasoning tasks, with up to 18.2% better performance on the List Functions benchmark and up to 15.3% better performance on the Simple Turing Concepts benchmark. Finally, symbol-tuned models show large improvements in following flipped-labels presented in-context, meaning that they are more capable of using in-context information to override prior knowledge.

## introduction a key feature of human intelligence is that humans can learn to perform new tasks by reasoning using only a few examples. scaling up language models has unlocked a range of new applications and paradigms in machine learning, including the ability to perform challenging reasoning tasks via few-shot examples given in-context @xcite @xcite . language models, however, are still sensitive to the way that prompts are given, indicating that they are not reasoning in a robust manner. for instance, language models often require heavy prompt engineering @xcite or phrasing tasks as instructions @xcite @xcite , and they exhibit unexpected behaviors such as performance on tasks being unaffected even when shown exemplars with random labels @xcite or flipped labels @xcite . in this paper, we propose a simple finetuning procedure that we call symbol tuning, which significantly improves the ability of language models to reason with and learn from input-label mappings presented in-context. in the symbol-tuning procedure, we finetune language models on input-label pairs presented in-context where natural language labels are remapped to arbitrary symbols. 1 the intuition is that when models cannot rely on instructions or relevant natural language labels to figure out a given task, they must instead do so by reasoning with input-label mappings presented in-context in order to learn the mappings that reveal the task. we perform symbol tuning using a mixture of 22 nlp datasets with various arbitrary symbols as labels and experiment using instruction-tuned palm models (flan-palm) with several sizes (8b, 62b, 62b-cont, 540b). first, symbol tuning improves performance of baseline models on unseen in-context learning tasks across various settings (with/without instructions, with/without relevant labels), with larger performance gains when instructions or natural language labels are not given in the prompt. for example, when prompts do not contain instructions or relevant labels, symbol tuning yields a +11.1% average performance improvement across eleven evaluation tasks for flan-palm-62b-cont. ## symbol tuning despite their ability to perform some reasoning tasks after being shown in-context exemplars @xcite , language models are still sensitive to the way in which these tasks are presented in prompts @xcite @xcite , suggesting that they are not reasoning in a robust way. instruction tuning has been shown to improve performance and allow models to better follow incontext exemplars @xcite @xcite @xcite . one shortcoming, however, is that models are not forced to learn to use the exemplars because the task is redundantly defined in the evaluation example via instructions and natural language labels. for example, in the left-hand side of figure 1 , although the exemplars can help the model understand the task, they are not strictly necessary since the model could ignore the exemplars and just read the instruction. to make the model better at in-context learning, we propose symbol tuning, in which the model is finetuned on exemplars where the instructions are removed and natural language labels are replaced with semantically-unrelated labels (e.g., "foo," "bar," etc.). in this setup, the task is unclear without looking at the in-context exemplars. for example, if the prompt from the previous paragraph was changed to "<sentence>. answer: {foo, bar}" (as shown in the right-hand side of figure 1 ), multiple in-context exemplars would be needed in order to figure out the task. because symbol tuning teaches the model to reason over in-context exemplars, symbol-tuned models should have better performance on unseen tasks that require reasoning between in-context exemplars and their labels. characters (1-3 letter combinations) words (mit list of 10,000 words ) characters (3-4 letter combinations) words (mit list of 100,000 words) integers (1-4 digits) integers (5 digits) finetuning (~30k symbols) evaluation (~270k symbols) figure 3 : we use a set of ∼300k arbitrary symbols from three categories (integers, character combinations, and words). ∼30k symbols are used during tuning and the rest are held out for evaluation. see appendix e.1 for more details on the symbols that we used. ## symbol-tuned models are better in-context learners during symbol tuning, models must learn to reason with in-context exemplars in order to successfully perform tasks because prompts are modified to ensure that tasks cannot be learned from natural language labels or instructions. symbol-tuned models should thus perform better in settings where tasks are unclear and require reasoning between incontext exemplars and their labels. additionally, since symbol tuning is meant to improve the ability to follow in-context exemplars, it should not modify prior knowledge and should thus retain the same performance in settings where exemplars are not as necessary to complete the task. to explore these settings, we define four icl settings that vary the amount of reasoning required between inputs and labels in order to learn the task (based on the availability of instructions/relevant labels), as shown in figure 4 . the easiest of these settings uses prompts where both instructions and relevant labels are available (as in-context exemplars are not necessary to learn the task), while the hardest setting uses prompts where instructions and relevant labels are both unavailable. in symbol-tuned models also perform somewhatcomparably in settings with only relevant labels or only instructions, unlike baseline models whose performance in settings with only relevant labels is always better than in settings with only instructions. performance in settings with relevant labels actually decreases for flan-palm-8b after symboltuning, however, which may suggest that symbol tuning a small model can override its prior knowl-edge due to overfitting. overall, the improvements demonstrate the strong potential of symbol tuning to improve model performance, especially when tasks require learning from in-context exemplars. ## symbol tuning improves algorithmic reasoning symbol tuning is designed to force the model to learn from input-label mappings in the in-context exemplars because the symbols are unrelated to the task and no instructions are provided (and thus the model cannot rely on any other guidance to determine the task). for this reason, we posit that symbol tuning should not only improve the model's ability to map natural language inputs to arbitrary symbols, but also its ability to learn other forms of input-label mappings such as algorithms. to test this, we experiment on algorithmic reasoning tasks from big-bench @xcite . we first experiment on a set of list function tasks @xcite where the model needs to identify a transformation function (e.g., remove the last element in a list) between input and output lists containing nonnegative integers. these tasks were evaluated in a four-shot setting, following our evaluation setup in section 3.2. additionally, we test models on a set of simple turing concepts @xcite where models need to reason (a-e): categories of list functions tasks @xcite . (f): simple turing concepts task @xcite . accuracy per list function category is averaged across all subtasks (categories and per-task results are shown in appendix f.1). with binary strings to learn the concept that maps an input to an output (e.g., swapping 0s and 1s in a string). these tasks have predetermined shots for each evaluation example. we selected these algorithmic tasks because they test the model's ability to generalize to different task types (the symbol-tuning tasks were classification problems with discrete labels, while these tasks are more open-ended generation problems 2 ) and do not require world knowledge (symbol tuning does not increase a model's prior knowledge). in figure 5 , we show model performance on the twenty list function tasks with the highest human accuracy baselines 3 (rule, 2020) separated into five categories (category details are described in appendix f.1) and the turing concepts containing 3 or fewer instructions in the as ii subset of the simple turing concepts task. on the list function tasks, symbol tuning results in an average performance 2 although chain-of-thought @xcite can be viewed as an open-ended generation problem, in appendix c.2, we found that symbol-tuning did not significantly affect a model's cot reasoning abilities, possibly because our symbol-tuning data did not incorporate any cot prompts. 3 we do not directly compare with the human baselines because our evaluation format was different. to test this, we flip the labels of both in-context exemplars and the evaluation example for the tasks described in section 3.2 (we remove tasks with more than two labels from this experiment since it is unclear how to best "flip" more than two labels). for example, for the sst2 dataset, all exemplars that are labeled as having "positive" sentiment will now be labeled as having "negative" sentiment. a perfect model that can follow these flipped labels should achieve 100% accuracy on these tasks if its accuracy in the standard icl setting is also 100%. as shown in figure 6 , symbol tuning restores the ability to follow flipped labels that was lost during instruction tuning. we see that there is a similar trend across all model sizes-instructiontuned models are generally unable to follow flipped labels (as demonstrated by their performance being far below random guessing), but symbol-tuned models are much more capable of doing so. we found that after symbol tuning, flan-palm-8b sees an average improvement across all datasets of 26.5%, flan-palm-62b sees an improvement of 33.7%, and flan-palm-540b sees an improve-ment of 34.0%. for some datasets (e.g., or, subj, tc), symbol-tuned models can now override priors and follow flipped labels (i.e., achieve much better performance than random guessing), despite instruction-tuned models not being able to do so for any datasets. additionally, symbol-tuned models match or beat pretraining-only models in terms of average performance, indicating that symbol tuning has, to some extent, restored the model's original ability to follow flipped labels. these results further indicate another type of generalized in-context learning capability, as we did not include any flipped labels during symbol tuning. although the performance improvement from symbol tuning is large, we note that more work should be done in this area since performance on the flipped-labels settings is, on average, not significantly better than random guessing. ## limitations while our study presents a simple yet effective method of improving in-context learning for language models, there are several limitations to our work. an open question is how to apply symbol tuning in a generative setting-we symbol tuned models on a range of classification tasks with discrete labels so that we can remap labels to arbitrary symbols, but we did not tune on generation tasks since it is unclear how to remap outputs to symbols in those settings. future work could thus explore whether symbol tuning can be applied in a generative setting. additionally, our symbol-tuning procedure used 22 nlp datasets-while we ablated the number of datasets in appendix b.4 and saw that increasing the number of datasets used for symbol tuning generally improves performance, we did not experiment with adding more tasks. prior work, however, has demonstrated that scaling up finetuning methods can improve their impact on language models @xcite , so a natural extension would be to examine whether further scaling up the symbol-tuning method would have a similar result. furthermore, we applied symbol tuning to a family of instruction-tuned language models. it is unknown, however, whether the effects of symbol tuning that we showed may be affected by changes to the pretraining objective, model architecture, or training process. similarly, symbol tuning may have different effects on language models that are not instruction tuned, as we did not specifically experiment on this factor. for this reason, future work may investigate how these factors impact the effectiveness of symbol tuning for improving in-context learning abilities in language models. because we only experimented with one family of language models, it is still unclear whether symbol tuning is effective for other models. applying symbol tuning to other language models would likely require adjustments to the finetuning procedure to be successful (e.g., number of finetuning steps, mixing with previous data, number datasets), but a definitive conclusion about these factors cannot be drawn without further experimentation. we thus note that a crucial direction for future work is to explore how well symbol tuning translates to other language models. ## conclusions in this paper, we presented symbol tuning, a new method of tuning models on tasks where natural language labels are remapped to arbitrary symbols. symbol tuning is based off of the intuition that when models cannot use instructions or relevant labels to determine a presented task, it must do so by instead learning from in-context exemplars. we tuned four language models (flan-palm-8b, flan-palm-62b, flan-palm-62b-cont, and flan-palm-540b) using our symbol-tuning procedure, utilizing a tuning mixture of 22 datasets and approximately 30k arbitrary symbols as labels. experimentally, we showed that symbol tuning can significantly improve a model's ability to learn from in-context exemplars in not only natural language settings, but also on algorithmic tasks. first, we showed that symbol tuning improves performance on unseen in-context learning tasks, especially when prompts do not contain instructions or relevant labels. we also found that symbol-tuned models were much better at algorithmic reasoning tasks, despite the lack of numerical or algorithmic data in the symbol-tuning procedure. finally, in an in-context learning setting where inputs have flipped labels, symbol tuning (for some datasets) reunlocks the ability to follow flipped labels that was lost during instruction tuning. through symbol tuning, we aim to have increased the degree to which models can examine and learn from input-label mappings during in-context learning. we hope that our results encourage further work towards improving language models' ability to reason over symbols presented in-context.