Naive B ayes-based Context Extension for Large Language Models (2024)

Abstract: Large Language Models (LLMs) have shown promising in-context learning abilities. However, conventional In-Context Learning (ICL) approaches are often impeded by length limitations of transformer architecture, which pose challenges when attempting to effectively integrate supervision from a substantial number of demonstration examples. In this paper, we introduce a novel framework, called Naive Bayes-based Context Extension (NBCE), to enable existing LLMs to perform ICL with an increased number of demonstrations by significantly expanding their context size. Importantly, this expansion does not require fine-tuning or dependence on particular model architectures, all the while preserving linear efficiency. NBCE initially splits the context into equal-sized windows fitting the target LLM’s maximum length. Then, it introduces a voting mechanism to select the most relevant window, regarded as the posterior context. Finally, it employs Bayes’ theorem to generate the test task. Our experimental results demonstrate that NBCE substantially enhances performance, particularly as the number of demonstration examples increases, consistently outperforming alternative methods. The NBCE code will be made publicly accessible. The code NBCE is available at: https://github.com/amurtadha/NBCE-master

## introduction large language models (llms) have demonstrated remarkable capabilities in in-context learning (icl), a paradigm that enables them to excel in various unseen tasks based on task examples or instructions within their context @xcite . unlike traditional finetuning methods, icl leverages llms for downstream tasks solely through inference, eliminating the need for parameter updates and making it computationally efficient, bringing us closer to the goal * these authors contributed equally to this work. of general ai. this approach has gained prominence as llms continue to grow in scale @xcite @xcite . the 2048-token context limit in popular llms like gpt-3 poses challenges for scaling up icl with more demonstration examples in icl, due to architectural constraints and computational complexity. recent studies @xcite @xcite improve icl through meta-learning and fine-tuning on downstream tasks, but the limited diversity of annotated tasks and biases hinder generalization. another line of research has explored various approaches to retraining longrange language models with extrapolation, extending them to 128 times the limit of existing llms @xcite . however, these approaches require additional training over several steps, which can be time-consuming. recently, @xcite introduced structured prompting, encoding demonstrations with specific position embeddings for collective attention via a scaled mechanism. extending this, @xcite proposed parallel context windows, utilizing individual encoding of examples with designed position and attention mechanisms. addressing this issue is crucial for leveraging icl effectively, especially in scenarios with ample examples. in this paper, we introduce a novel framework called naive bayes-based context extension (nbce) for large language models to significantly expand the number of demonstrations by orders of magnitude while greatly enhancing stability. instead of simply merging all demonstrations, we partition the vast number of demonstrations into multiple groups, each independently processed by the language model. this approach ensures that the encoding complexity scales linearly with the number of groups, avoiding the quadratic complexity associated with considering all examples simultaneously. following @xcite ; @xcite , we align the position embeddings of grouped prompts to the right, placing them next to the test input. subsequently, we leverage the naive bayes to encode the input by conditioning it on these grouped prompts. we conducted experiments across various tasks, including text classification, multi-choice, and open-ended tasks. nbce effectively scales up the number of demonstrations, outperforming conventional in-context learning across different model sizes and tasks, while also significantly enhancing stability. in brief, the contributions can be summarized as follows: ## approach an example of our proposed nbce is depicted in 1. assume that we have a sequence, denoted as t , which we intend to generate. furthermore, we have multiple relatively independent context sets, denoted as s 1 , s 2 , . . . , s n (e.g., n different paragraphs), each of which is sufficiently long and does not split a sentence into fragments. suppose that the total length of these context sets exceeds the training length, but when combined with an individual s k and t , they still fall within the training length. our objective is to generate t based on the information contained in s 1 , s 2 , . . . , s n . in essence, we seek to estimate the conditional probability of t given s 1 , s 2 , . . . , s n , which can be represented as in straightforward terms, naive bayes can be understood as a combination of two key elements: bayes' formula and an independence assumption: thus, we have: to commence this refinement, we shall introduce the following notations: however, the reformulation may prompt the emergence of two inherent inquiries: to delve deeper into these inquiries, we conducted a series of experiments employing the 7b model and garnered preliminary insights. in the realm of reading comprehension, a consistent trend of robust performance emerges when employing max pooling with a β value of 0.25 in conjunction with greedy search. conversely, outcomes generated via random sampling frequently yield results that are challenging to interpret. the observed disparities in outcomes can be attributed to the inherent characteristics of these two methods. random sampling, characterized by its selection of tokens based on their probability distribution, tends to exhibit lackluster performance, signaling that the output of max pooling may not align with a plausible probability distribution. in contrast, greedy search operates distinctively by prioritizing the token with the highest probability, disregarding the holistic distribution. its commendable performance suggests that the token with the highest probability is more likely to be the accurate choice. larger probabilities are indicative of lower uncertainty. to enhance the performance of random sampling, we modify the pooling method to directly output the probability distribution with the lowest uncertainty: by substituting this expression into eq.9, we arrive at the conclusive formulation of the nbce. it is noteworthy that while the initial inspiration for this approach stemmed from naive bayes, the generalized equation 9 transcends the conventional boundaries of traditional naive bayes, yet maintains its inherent interpretability. eq.9 assumes an intuitive form: predictions originating from various contextual sources are collectively amalgamated (or weighted) through the utilization of the method denoted as p (with a weight factor of β + 1). subsequently, this amalgamation is counterbalanced by subtracting the prediction in the absence of context, weighted by β. the rationale behind subtracting the context-less prediction lies in enhancing the model's reliance on contextual information, reducing its dependency on inherent knowledge @xcite . the choice of values for β can be tailored to different scenarios. for tasks necessitating comprehensive reading comprehension and robust context integration, a larger β value may be deemed appropriate. conversely, tasks leaning towards creative writing may benefit from a smaller β value. in our experiments, we set β = 0.25. ## experimental setup in this section, we describe the experimental settings adopted in our work, including the datasets, llms and comparative approaches used to evaluate our approach. ## evaluation we evaluate our proposed solution based on two primary criteria: ## conclusion this paper introduces a novel framework called naive bayes-based context extension (nbce) for large language models. nbce innovatively incorporates a voting mechanism to select the most appropriate window context, and then utilizes bayes' theorem to generate the task text. our results show that nbce outperforms its alternative pcw across a diverse set of multi-class classification tasks. for future work, while pcw shows effective without additional training, icl could potentially benefit from more demonstrations in fine-tuning settings; however, further investigation is required to fully comprehend the extent of its advantages.