Enhancing In-Context Learning via Implicit Demonstration Augmentation (2024)

Abstract: The emergence of in-context learning (ICL) enables large pre-trained language models (PLMs) to make predictions for unseen inputs without updating parameters. Despite its potential, ICL’s effectiveness heavily relies on the quality, quantity, and permutation of demonstrations, commonly leading to suboptimal and unstable performance. In this paper, we tackle this challenge for the first time from the perspective of demonstration augmentation. Specifically, we start with enriching representations of demonstrations by leveraging their deep feature distribution. We then theoretically reveal that when the number of augmented copies approaches infinity, the augmentation is approximately equal to a novel logit calibration mechanism integrated with specific statistical properties. This insight results in a simple yet highly efficient method that significantly improves the average and worst-case accuracy across diverse PLMs and tasks. Moreover, our method effectively reduces performance variance among varying demonstrations, permutations, and templates, and displays the capability to address imbalanced class distributions.

## introduction large pre-trained language models (plms) have showcased exceptional abilities in in-context learning (icl) @xcite @xcite , which assists the model in discerning the underlying patterns within demonstrations and make more accurate predictions @xcite . as a new paradigm, icl offers compelling advantages, allowing for natural language interaction with plms @xcite , as well as reduced computational costs @xcite . while promising, icl's performance is highly dependent on provided demonstrations and templates @xcite ; @xcite , resulting in subpar and unstable performance. this promotes research aimed at improving the quality @xcite , quantity @xcite , and permutations @xcite of demonstrations. other research avenues include prediction adjustment @xcite @xcite and learning process design (e.g., channel models @xcite and meta-training frameworks @xcite ). despite ongoing efforts, icl still struggles with efficiently and reliably capturing sufficient knowledge from context, leaving performance stability as a persistent bottleneck. in this study, we propose enriching contextual knowledge for plms by augmenting demonstrations. we first attempt to enhance the representation of demonstrations by transforming them along semantic directions sampled from the deep feature space of demonstration examples, as depicted in figure 1 . this operation stems from the observation that the deep features in a network are usually linearized @xcite @xcite , implying the existence of numerous semantic directions within the deep feature space, hence potentially enabling us to incorporate richer contextual knowledge without extending input length. from this novel perspective, we theoretically prove that when the number of augmented pieces approaches infinity, its effect approximately equals a logit adjustment operation. specifically, we derive a refined softmax function that integrates the statistical properties of demonstrations. consequently, rather than explicitly executing the augmentation procedure, we can efficiently conduct implicit demonstration augmentation using the derived prediction function, obtaining an improved icl method with theoretical guidance. we conduct extensive experiments across seven plms and various classification tasks. the empirical results demonstrate that our approach remarkably enhances prediction accuracy and reduces performance variability across different demonstrations, permutations, and templates. notably, our method is straightforward, effective, and generalizable, enabling seamless integration with other icl methods to enhance their performance. our contributions can be summarized as follows: nevertheless, recent research has uncovered certain limitations. to begin with, the volume of input knowledge for each query is constrained by the maximum input length of plms @xcite , and the computational cost increases as the number of demonstrations grows @xcite , making it challenging to integrate significant knowledge from demonstrations to plms. additionally, icl's performance is sensitive to the input of plms @xcite , thus exhibiting high variance and poor worst-case accuracy @xcite . researchers have explored various techniques to address the biases and instability of icl. these techniques encompass learning process design @xcite , demonstration retrieval @xcite , prompt engineering @xcite , and prediction calibration @xcite . however, these methods have yet to fully address the issue of severely limited knowledge transfer from demonstrations to large plms. ## conclusion this study introduces idaicl, a novel icl approach designed to enhance demonstrations by utilizing semantic directions sampled from the deep feature distribution of demonstration examples. our augmentation strategy enriches the knowledge available to plms without extending the context length. a new prediction function is then theoretically established considering the number of augmented pieces approaching infinity. this eliminates the need for explicit augmentation and allows for direct utilization of this derived function for predictions. our extensive experiments, spanning various tasks and plms, demonstrate that idaicl significantly enhances both prediction accuracy and stability when compared to other icl baselines.