P ara ICL : Towards Parallel In-Context Learning (2025)

Abstract: Large language models (LLMs) have become the norm in natural language processing (NLP), excelling in few-shot in-context learning (ICL) with their remarkable abilities. Nonetheless, the success of ICL largely hinges on the choice of few-shot demonstration examples, making the selection process increasingly crucial. Existing methods have delved into optimizing the quantity and semantic similarity of these examples to improve ICL performances. However, our preliminary experiments indicate that the effectiveness of ICL is limited by the length of the input context. Moreover, varying combinations of few-shot demonstration examples can significantly boost accuracy across different test samples. To address this, we propose a novel method named parallel in-context learning (ParaICL) that effectively utilizes all demonstration examples without exceeding the manageable input context length. ParaICL employs parallel batching to distribute demonstration examples into different batches according to the semantic similarities of the questions in the demonstrations to the test question. It then computes normalized batch semantic scores for each batch. A weighted average semantic objective, constrained by adaptive plausibility, is applied to select the most appropriate tokens. Through extensive experiments, we validate the effectiveness of ParaICL and conduct ablation studies to underscore its design rationale. We further demonstrate that ParaICL can seamlessly integrate with existing methods.

## introduction in recent years, scaling up the parameters of generative language models has significantly enhanced their language generation capabilities @xcite @xcite . large language models (llm) have demonstrated their adeptness across a wide range of tasks via few-shot in-context learning (icl) @xcite @xcite . in fewshot icl, models are expected to generate outputs directly based on a sequence of given examples without any parameter modifications, making it the most efficient method for adapting to new tasks. however, the effectiveness of icl is notably influenced by the few-shot demonstration examples used @xcite . various methods have been developed to select the most effective fewshot demonstration examples for icl. @xcite demonstrated that scaling up the number of demonstration examples can improve the icl performance. @xcite and @xcite proposed to utilize the most relevant examples for each test sample during inference. nevertheless, existing few-shot icl methods mostly focus on either increasing the number of demonstration examples or selecting a few that are similar to the test samples to improve performance. we have designed two preliminary experiments to identify key factors for effectively using demonstration examples. firstly, we increase the number of demonstration examples. the results, as shown in figure 1 , reveal that the performance of mistral-7b-instruct-v0.2 on gsm8k and winogrande does not consistently improve with more examples. this is partly because longer input lengths, resulting from more examples, can lead to suboptimal results in llms @xcite . therefore, controlling the input context length is essential in icl, making the number of few-shot demonstration examples a critical factor. secondly, we focus on the selection of demonstration examples. we experiment with llama-2-7b-chat on 100 test samples from winogrande with 32 different 10-shot example combinations. as shown in figure 2 , we notice that varying combinations lead to different accuracy improvements. these combinations enabled the model to accurately answer 80% of the test samples, a significant increase compared to the 50.9% average accuracy of each individual 10-shot combination. this highlights the fact that varying demonstration examples can enhance the model's accuracy on different test samples. therefore, we should leverage all available demonstration examples when possible. to combine the best of both worlds, we introduce parallel in-context learning (paraicl). paraicl effectively utilizes the maximum number of demonstration examples without extending the input context length, thus avoiding the potential reduction in model performance due to larger context sizes. given a set of question-answer pairs as demonstration examples, paraicl first assigns them into various batches based on the semantic similarity between the demonstrations' questions and the sample test question. consequently, each batch maintains a controlled context length while utilizing all demonstration examples. these batches are then processed by a causal model in parallel to obtain the next token distribution for each batch. afterward, a weighted average of these distributions is calculated, considering the semantic relation of each batch to the test question. the final step involves selecting the token with the highest weighted average probability for continued generation. we conduct extensive experiments across a range of reasoning, natural language inference, and coding tasks to validate the effectiveness of paraicl. we further demonstrate that paraicl is compatible with both open-and closed-source causal language models. our study includes abundant ablation studies and analyses to justify the design of paraicl and demonstrate how it can be integrated with other icl methods. in summary, our main contributions are the following: (1) we introduce paraicl, a simple but effective method that leverages all available demonstration examples while maintaining the input context length manageable. (2) we conduct thorough experiments to prove the effectiveness of our method, along with ablation studies to justify its design. (3) we illustrate how our method can enhance and work in conjunction with other methods. ## related work icl has surged as a transformative approach in the nlp domain, with its significance evident across various applications, including knowledge grounding @xcite , code generation @xcite , and other industrial applications @xcite . despite its promise, challenges such as sensitivity to the prompts and context window length limitations, highlight areas for further research and development in making icl more robust and versatile. as aforementioned, various studies have underscored that icl exhibits significant sensitivity to the quality of prompts, particularly concerning the demonstration examples provided @xcite @xcite . @xcite discovered that increasing the number of examples only leads to slight improvements. @xcite proposed to select examples that have the highest semantic similarity to the test question. contrarily, @xcite found that leveraging a diverse set of demonstration examples could improve in-context compositional generalization. @xcite proposed iterative demonstration selection, which considers both the diversity and similarity dimensions of icl demon-stration selection for llms. additionally, chia et al. (2023) explored the potential of contrastive examples in improving the reasoning capabilities of causal language models. nevertheless, these methods primarily focus on choosing a subset of demonstration examples from a pool of candidates based on certain perspectives. furthermore, these methods necessitate a substantial number of candidates to efficiently select the best examples, which constrains their applicability across all situations. another line of research is dedicated to utilizing extended context in icl. @xcite proposed parallel context window (pcw), a method that alleviates the context window limitations for any open-source llms without necessitating additional training. however, pcw overlooked the semantic connections between the demonstration examples and the test samples. as evidenced by @xcite , merely increasing the quantity of demonstration examples does not correlate with a substantial improvement in performance. in fact, this strategy might detract from overall performance due to the inclusion of incorrect or misleading content within some of the examples. the implementation of our paraicl method is also inspired by the ensemble of models @xcite , which use multiple slightly different models to contribute probability "votes" before producing the final answer by (weighted) averaging the votes' probabilities. vastly different from the ensemble of models, paraicl uses the same language model prompted with different sets of in-context exemplars to produce varying vote probabilities before aggregating the output distributions to produce the final answer. as such, paraicl does not require many models and is efficient to compute. ## methodology we first formulate the problem setting for few-shot icl in section 3.1. following this, in section 3.2, we introduce paraicl, a novel method that efficiently processes demonstration examples in batches and effectively utilizes a novel parallel semantic decoding strategy for generation. subsequently in section 3.3, we introduce the weighted average semantic objective alongside the adaptive plausibility constraint. finally we define the parallel semantic decoding method, which employs the weighted average semantic objective subject to the plausibility constraint. a demonstration of paraicl can be found in figure 3 . ## conclusions in this work, we introduce a novel approach known as parallel in-context learning (paraicl), designed to enhance the effectiveness of few-shot in-context learning. paraicl aims to fully leverage all available demonstration examples while keeping within the limits of a manageable input context size. it starts by executing parallel batching, grouping demonstration examples into various batches based on the semantic similarities between the questions in the demonstrations and the test questions. afterward, for each batch, normalized semantic scores are calculated. the process culminates in the decoding of the final tokens, optimizing a weighted average semantic objective under an adaptive plausibility constraint. paraicl has been proven to yield consistent performance improvements across a diverse set of benchmarks and exhibits a high degree of compatibility for integration with other methodologies.