Meta-learning via Language Model In-context Tuning (2022)

Abstract: The goal of meta-learning is to learn to adapt to a new task with only a few labeled examples. Inspired by the recent progress in large language models, we propose \textit{in-context tuning} (ICT), which recasts task adaptation and prediction as a simple sequence prediction problem: to form the input sequence, we concatenate the task instruction, labeled in-context examples, and the target input to predict; to meta-train the model to learn from in-context examples, we fine-tune a pre-trained language model (LM) to predict the target label given the input sequence on a collection of tasks.We benchmark our method on two collections of text classification tasks: LAMA and BinaryClfs. Compared to MAML which adapts the model through gradient descent, our method leverages the inductive bias of pre-trained LMs to perform pattern matching, and outperforms MAML by an absolute 6% average AUC-ROC score on BinaryClfs, gaining more advantage with increasing model size. Compared to non-fine-tuned in-context learning (i.e. prompting a raw LM), in-context tuning meta-trains the model to learn from in-context examples. On BinaryClfs, ICT improves the average AUC-ROC score by an absolute 10%, and reduces the variance due to example ordering by 6x and example choices by 2x.

## introduction few-shot learning (fsl) refers to a system's ability to quickly adapt to new tasks when very few labeled examples are available for training. fsl is a key feature of human learning @xcite , but current machine learning systems often rely on large amounts of labeled training data @xcite @xcite . recently, prompting large pre-trained language models (lms) for fsl has achieved remarkable progress @xcite . lm prompting with in-context learning reduces the "task learning and predict" process to a simple sequence prediction problem. to perform a new task, brown et al. (2020) prompt a raw lm (i.e., a pre-trained lm not fine-tuned on any labeled data) with the concatenation of the task instruction, some input-output examples, and the target input to be predicted on; then they extract the answer from the lm's continuation of the concatenated sequence (figure 1 left). for example, to coax the model into performing sentiment classification on the target input "this movie is a waste of time", we prompt the lm with the sequence "i like the movie! positive review? yes. horrible movie! positive review? no. this movie is a waste of time. positive review? ___", and predict "positive" if the next word is more likely to be "yes" rather than "no". however, raw lms are not optimized for incontext fsl during pre-training, and exhibit undesirable behavior when used for fsl. for example, @xcite observed that lms suffer from the "recency bias", which assigns higher probability to labels that appear closer to the target input. as a result, the accuracy becomes extremely sensitive to the ordering of the in-context examples. previous work has also shown that prompting raw lms is often oversensitive to example choices and instruction wording @xcite @xcite . we address this weakness through a metalearning lens and directly fine-tune the lm for fsl. under the meta-learning framework, we metatrain a model to learn to adapt to new tasks from a few examples on a wide range of tasks, so that it learns to leverage the few-shot examples to adapt to new tasks at test time. since lm prompting already reduces the "task learning and predict" process to a simple sequence prediction problem, we meta-train a lm by directly fine-tuning it to optimize for this sequence prediction problem on a wide range of tasks (figure 1 left). since we fine- instruction x1 y1 x' y' x2 y2 meta-update via gradient descent in-context tuning θ := θ-δ few-shot adaptation via in-context learning maml y1 x1 instruction y2 x2 instruction θ′ := θ-δ y' x' instruction calculate loss with θ′ meta-update: optimize to minimize the loss. to adapt the model initialization to a new task t , a task-specific model θ initialized with θ is updated with gradient descent using task examples from t . meta-training of maml involves bi-level optimization, where the inner optimization learns a task-specific model θ using task examples from t , and the outer optimization learns a meta-initialization θ to minimize few-shot prediction loss of θ on task t . in-context tuning (ours) (left): our approach adapts to new tasks via in-context learning, and learns a single model θ shared across all tasks that is directly optimized with the fsl objective (section 2.2). because model parameters are frozen during task adaptation, our approach does not involve bi-level optimization during meta-training. we benchmark our algorithm on lama @xcite , a dataset for testing models' factual knowledge, and binaryclfs @xcite , a wide range of binary classification tasks each annotated with a few language descriptions of the task. compared to prompting raw lms, in-context tuning improves performance by 7.6 precision@1 points on lama and 10.6% auc-roc score on binaryclfs. in addition, in-context tuning mitigates the over-sensitivity of raw lm prompting, significantly reducing the variance of the performance with respect to example ordering (by 68% on lama and 83% on binaryclfs), example choices (by 56% on lama and 40% on binaryclfs), and instruction wording (by 19% on lama). our approach also out-performs maml, which adapts the model by gradient descent on a few examples and learns an initialization that can adapt to a new task through a few gradient steps @xcite . since our approach better takes advantage of the inductive bias of lms to extrapolate from in-context examples, our approach out-performs first-order maml by 2.8 points on lama and 5.1 points on binaryclfs, with increasing advantage as models become larger. given the empirical effectiveness of in-context tuning (section 4.1), we conjecture that the fewshot learning potential of large lms (e.g., gpt-3) may be broadly underestimated if prompted without any direct optimization for fsl. we also conjecture that in-context tuning can mitigate various undesirable properties of lm prompting, such as over-sensitivity to example ordering, example choices, and instruction wording (section 4.2). ## approach we introduce the problem setup (section 2.1), describe our in-context tuning algorithm (section 2.2), compare our algorithm to gradient-based adaptation methods (section 2.3) and other baselines (section 2.4). ## results in-context tuning out-performs maml and various baselines on the two text classification metadatasets (section 4.1). it also significantly reduces model sensitivity to instruction wording, example choices, and example ordering compared to prompting raw lms (section 4.2). ## conclusion in this work, we propose meta-learning via incontext tuning, which recasts the few-shot learning process of task adaptation and task-specific prediction as a simple sequence prediction problem, where few-shot labeled examples are concatenated with the target example to form the model input. in-context tuning out-performs a wide variety of baselines in terms of accuracy, including raw lm prompting, maml and instruction tuning. meanwhile, sensitivity study shows that our fsl approach of in-context tuning is significantly less sensitive to few-shot examples and instruction wording compared to raw lm prompting. given the empirical effectiveness of in-context tuning, we conjecture that the few-shot learning potential of large lms (e.g., gpt-3) might be broadly underestimated, and that in-context tuning can eliminate well-known artifacts of few-shot lm prompting such as over-sensitivity to example ordering, example selection and instruction wording.