O pen ICL : An Open-Source Framework for In-context Learning (2023)

Abstract: In recent years, In-context Learning (ICL) has gained increasing attentionand emerged as the new paradigm for large language model (LLM) evaluation. Unlike traditional fine-tuning methods, ICL instead adapts the pre-trained models to unseen tasks without any parameter updates. However, the implementation of ICL is sophisticated due to the diverse retrieval and inference methods involved, as well as the varying pre-processing requirements for different models, datasets, and tasks. A unified and flexible framework for ICL is urgently needed to ease the implementation of the aforementioned components. To facilitate ICL research, we introduce OpenICL, an open-source toolkit for ICL and LLM evaluation. OpenICL is research-friendly with a highly flexible architecture that users can easily combine different components to suit their needs. It also provides various state-of-the-art retrieval and inference methods to streamline the process of adapting ICL to cutting-edge research. The effectiveness of OpenICL has been validated on a wide range of NLP tasks, including classification, QA, machine translation, and semantic parsing. As a side-product, we found OpenICL to be an efficient yet robust tool for LLMs evaluation. OpenICL is released at https://github.com/Shark-NLP/OpenICL .

## introduction the rise of large language models (llms) @xcite @xcite has shown impressive emergent in-context learning (icl) ability @xcite . different from finetuning which requires parameter updates, icl can perform inference with model parameters frozen. icl sidesteps the resource-intensive nature of fine-tuning, yet still yields comparable results to fine-tuned models in specific tasks @xcite @xcite . however, we observed a lack of a unified framework for icl. implementations from existing projects are often high-customized to their own needs, thus making further development and comparisons with previous approaches a challenge. the basic icl pipeline contains two steps: retrieval and inference. given a testing input x ′ , in the retrieval stage, several examples from the training set are retrieved as in-context demonstrations. in the inference stage, these demonstrations are prepended before x ′ and fed into the llm to generate the prediction. researchers have explored various methods for both retrieval(e.g., bm25 @xcite , topk @xcite and votek @xcite ) and inference(e.g., perplexity-based @xcite , channel-based @xcite , and chain-of-thoughts @xcite ). however, these methods are often implemented under different frameworks, and/or evaluated using different llms and tasks. these inconsistencies make systematic evaluations and comparisons of various methods challenging, thus hindering the development of better icl methods. to address this issue, we present openicl, an open-source and easy-to-use toolkit for icl. openicl has many state-of-the-art retrieval and inference methods built in to facilitate systematic comparison and fast research prototyping. openicl also provides a unified and flexible interface for the development and evaluation of new icl methods. users can easily incorporate different retrieval and inference methods, as well as different prompt instructions, into their pipelines. to validate openicl's implementation and design, we use openicl to evaluate llms on several nlp tasks, including classification, question answering, translation, and semantic parsing. our contributions are summarized as follows: in-context learning besides the classic "pretrain and fine-tune" paradigm, brown et al. aside from those early successes, researchers have developed more sophisticated icl methods that require some intermediate reasoning steps. among them, chain-of-thoughts (cot) is the first attempt that significantly surpasses the previous state-of-the-art methods on many reasoning tasks @xcite . after that, different variants of cot have been proposed to strengthen its performance, such as self-ask @xcite , icap @xcite , least-to-most prompting @xcite , and selection-inference @xcite . despite the surprising performance, icl has been criticized for being very sensitive to the choice and ordering of in-context examples @xcite . to address this problem, different criterion and context construction methods have been proposed. @xcite and @xcite select examples that are closer to the test input in the embedding space; a line of work @xcite @xcite select the most representative examples in the training set to encourage diversity of in-context examples; @xcite observe that minimum description length (mdl) principle can be an effective criterion for in-context example selection. prompt learning prompt learning @xcite is a special case of icl without any incontext examples. prompt learning comprises various topics including manual template engineering @xcite , automated template learning @xcite @xcite ), and answer engineering @xcite . we refer the readers to the usage of open-prompt @xcite which is a toolkit specially designed for prompt learning. in comparison, openicl focuses more on integrating various exemplar retrieving approaches and inference strategies for in-context learning. note that openicl can also seamlessly support prompt learning by setting the number of in-context examples to zero and specifying the manual or pre-searched prompt templates by openprompt for different tasks. ## openicl in this section, we first explain openicl's design principles. then, we will briefly describe openicl's two major components, namely, the retriever and inferencer. ## toolkit walkthrough in this section, we demonstrate openicl by walking readers through several typical icl use cases. example 1. we first demonstrate how to use openicl to develop a typical icl pipeline for language classification using a few lines of code and conduct evaluation on the popular sentiment classification dataset sst-2 @xcite . as shown in figure 2 , the pipeline begins with a datasetreader which loads the dataset given its name on huggingface dataset hub foot_2 or local file path. users need to specify the names of columns where the input ("text") and output ("label") are stored. secondly, a customized prompttemplate is instantiated with a dictionary that defines the prompts for each class label. the @xmath0, separately. after that, we initiate the retriever based on topk @xcite and set the number of in-context examples to 8 ("@xmath1. we select perplexity-based method to initiate the inferencer and use gpt2-xl as the llm. having 1 from openicl import datasetreader, prompttemplate 2 from openicl import topkretriever, pplinferencer, accevaluator 1 from datasets import load_dataset 2 from openicl import datasetreader, prompttemplate 3 from openicl import randomretriever, geninferencer, bleuevaluator 4 5 @xmath2, 'de-en').map(lambda example: example['translation']) 6 7 @xmath3, @xmath4, @xmath5:</@xmath6: </@xmath7, 10 {'de':'</@xmath8, 'en':'</@xmath9, @xmath10, @xmath11, prompttemplate, bm25retriever, cotinferencer 2 3 @xmath12, @xmath13, 4 @xmath14, @xmath15: </@xmath16: </@xmath17, 7 {'question':'</@xmath18, 'answer':'</@xmath19, 8 @xmath20, @xmath21.", 14 "\ntherefore, the answer (arabic numerals) is"] retriever inferencer lm metric random topk mdl direct channel ppl cot gpt2 … gpt3 xglm t5 wmt16 (de-en) piqa sst2 gigaword gsm8k 75.51 90.61 58.15 32.87 13.95 … accuracy bleu rouge-l … … figure 5: evaluation results. we conduct experiments on five representative tasks with openicl and use different retrievers, inferencers, language models, and other components. in terms of model usage, we adopt gpt-neo (2.7b) for sst2, piqa, and gigaword, xglm (7.5b) for wmt16 (de-en), and text-davinci-003 version of gpt-3 (175b) for gsm8k. all these been set, we can run the inference by invoking the inferencer (line 17) and calculating the accuracy of the model's prediction(line 18). example 2. figure 3 shows how to use openicl to work with generation problems. we consider the popular machine translation dataset wmt16 (bojar et al., 2016). as in example 1, we can easily load the dataset, define the prompt template, and initiate the retriever, by feeding new parameters to the function, respectively. the major api difference from example 1 is that (i) we add some pre-processing for the translation task (line 5); (ii) pplinferencer is replaced by inferencer tailored for generation (line 16); (iii) we use bleu to evaluate model performance. example 3. openicl also supports more advanced icl methods, as shown in figure 4. users can seamlessly switch to cot by only modifying two lines of code: line 14 defines the template for cot and line 15 initiates the inferencer with gpt3 using openai's api. similar multi-step icl methods such as self-consistency (wang et al., 2022) and selection-inference (creswell et al., 2022) can also be easily implemented by inheriting the superclass inferencer designed in openicl. ## evaluation to demonstrate openicl's flexibility we conducted experiments on a diverse set of datasets, llms, and icl methods. we consider piqa @xcite for commonsense reasoning, sst-2 @xcite for sentiment analysis, gsm8k @xcite for arithmetic reasoning, wmt16 de-en @xcite for machine translation and gigaword @xcite for summarization. we've also tested various llms, including gpt-neo (2.7b) @xcite ), text-davinci-003 version of gpt-3 (175b), and xglm (7.5b) @xcite . we use ope-nai's official api foot_3 to access gpt-3. the detailed setups and results are shown in figure 5 . as we can see, components of openicl can be easily chained to support different evaluation needs and replicate results of state-of-the-art methods. ## conclusion we present openicl, an open-source toolkit for in-context learning. openicl provides a convenient and flexible interface for in-context learning practice and research. our modular design allows it to support a wide range of llms, tasks, and icl methods with ease. we implement both model parallelism and data parallelism to make inference of large models more efficient. openicl is highly extensible, and we will continue to update it to keep pace with the latest research. despite the promising results, icl is still in its early stages, and many challenges remain. we believe openicl will be a valuable resource for researchers and practitioners to facilitate their research and development.