Pre-Training to Learn in Context (2023)

Abstract: In-context learning, where pre-trained language models learn to perform tasks from task examples and instructions in their contexts, has attracted much attention in the NLP community. However, the ability of in-context learning is not fully exploited because language models are not explicitly trained to learn in context. To this end, we propose PICL (Pre-training for In-Context Learning), a framework to enhance the language models’ in-context learning ability by pre-training the model on a large collection of “intrinsic tasks” in the general plain-text corpus using the simple language modeling objective. PICL encourages the model to infer and perform tasks by conditioning on the contexts while maintaining task generalization of pre-trained models. We evaluate the in-context learning performance of the model trained with PICL on seven widely-used text classification datasets and the Super-NaturalInstrctions benchmark, which contains 100+ NLP tasks formulated to text generation. Our experiments show that PICL is more effective and task-generalizable than a range of baselines, outperforming larger language models with nearly 4x parameters. The code is publicly available at https://github.com/thu-coai/PICL .

## introduction pre-trained language models (plms; @xcite have shown strong abilities of learning and performing unseen tasks conditioning on several task examples or instructions in its context, which is called in-context learning (icl; @xcite . compared to conventional fine-tuning methods, icl adapts plms to downstream tasks only through inference, without parameter updates, which is computationally cheaper in practice and is closer to general ai. however, plms trained on massive corpora to predict the next word given previous words are not explicitly taught to learn in the context. this makes icl a surprising emergent ability but also indicates that the icl ability of plms is not fully exploited. @xcite has shown that by directly training to do icl in a meta-learning paradigm, models show strong performance on learning simple function classes in the context. in practical nlp scenarios, previous works @xcite also enhance the icl performance by metafine-tuning plms on a large collection of downstream tasks and evaluating them on unseen tasks. however, the low diversity of human-annotated downstream tasks restricts the performance of the meta-tuned model. direct training on downstream tasks also brings undesired bias on specific input formats, label spaces, or domains, which hurts the generalization of plms. to enhance the icl ability while maintaining generalization, we propose picl (pre-training for in-context learning), a framework that exploits the plm's icl ability by pre-training models on data automatically constructed from the general plain-text corpus. our framework is based on a simple observation that many paragraphs in the text documents contain "intrinsic tasks". as shown in the left part of figure 1 , each paragraph in the document contains an intrinsic task. when doing language modeling on each paragraph, models implicitly perform the corresponding intrinsic tasks simultaneously. this shares a similar idea with the prompt-learning paradigm @xcite , where downstream data examples from nlp tasks are transformed into text sequences, and the model learns to perform the original tasks when trained on the text sequences with language modeling. different from the downstream data, text paragraphs contain more diverse intrinsic tasks and have little bias on input formats, label spaces, or domains because they are free-form texts from the large-plm knowing that argentina won the world cup, he was excited to tears. after breaking up with his girlfriend, he was so miserable. when i saw the avatar movie, i was impressed by its special effects. when facing the audience, she was too nervous to say a word. xbox, the famous video gaming brand created by microsoft, is sad to share that marc whitten, chief product officer of xbox, will be leaving the team. "i have had the extreme pleasure over the last 14 years because of the amazing team and the greatest community i've had the opportunity to work with." said marc. phil spencer, head of microsoft studios, said, "i've had the pleasure of working with marc and he has always led xbox forward with a focus on our fans. " other team members said, "we wish marc well, while looking forward to the next chapter of xbox." ## method we first present an overview of picl and then describe the details in the following sections. as shown in the right part of figure 1 , we construct the pre-training instances from a corpus c consisting of paragraphs split from full documents by "\n". for each paragraph z 0 in c, we first use a retriever r to find k paragraphs {z 1 , z 2 , • • • , z k } sharing the same intrinsic task (sentiment analysis) with z 0 . then the retrieved paragraphs are treated as demonstrations and concatenated with z 0 to form a pre-training instance: finally, we adopt a language modeling objective to pre-train the model on the constructed instances. in this way, the pre-training stage can be regarded as a meta-training process, where the model learns to solve the intrinsic task in z 0 conditioning on its context z k ⊕ z k-1 ⊕ • • • ⊕ z 1 . since c is a large-scale general corpus, it contains a variety of intrinsic tasks and little domain bias, which ensures the generalization of the pre-trained model. ## conclusion this paper presents picl, a framework that exploits the in-context learning ability of plms by pre-training models on concatenations of text paragraphs sharing the same "intrinsic tasks" gathered from the large-scale general corpus. in picl, models learn to perform various intrinsic tasks conditioning on their context while preserving their generalization due to the little bias of the pre-training data. extensive experiments show that picl improves the icl performance on various datasets against several baselines, enabling a 770 m model to outperform a larger model with about 4x parameters while maintaining good generalization across a wide range of tasks. for future work, we would like to consider adding human instructions to our pre-training framework to enhance more abilities of plms like zero-shot instruction following.