Retrieval-Augmented Few-shot Text Classification (2023)

Abstract: Retrieval-augmented methods are successful in the standard scenario where the retrieval space is sufficient; whereas in the few-shot scenario with limited retrieval space, this paper shows it is non-trivial to put them into practice. First, it is impossible to retrieve semantically similar examples by using an off-the-shelf metric and it is crucial to learn a task-specific retrieval metric; Second, our preliminary experiments demonstrate that it is difficult to optimize a plausible metric by minimizing the standard cross-entropy loss. The in-depth analyses quantitatively show minimizing cross-entropy loss suffers from the weak supervision signals and the severe gradient vanishing issue during the optimization. To address these issues, we introduce two novel training objectives, namely EM-L and R-L, which provide more task-specific guidance to the retrieval metric by the EM algorithm and a ranking-based loss, respectively. Extensive experiments on 10 datasets prove the superiority of the proposed retrieval augmented methods on the performance.

## introduction few-shot text classification, which entails learning a new task based on limited training data, has been advanced by pre-trained language models (plms) @xcite and prompt engineering @xcite . however, since training numerous parameters of plms on scarce data is prone to produce over-fitting @xcite and unstable generalization, only using the trained parameters for inference usually leads to unsatisfactory performance on unseen test data. on the other hand, retrieval-based methods have witnessed success on various natural language processing tasks, thanks to their capability of incorporating retrieved memory alongside parameters for better generalization. these methods retrieve relevant examples as memories from a large-scale corpus through either a static retrieval metric @xcite or a joint learningbased metric @xcite and then the retrieved examples are used to make a prediction. in this way, their generalization ability is achieved by not only the model parameters but also the retrieved memory. despite the theoretical potential of promoting generalization by using retrieved memory, previous retrieval-augmented methods empirically struggle to showcase compelling ability in few-shot learning scenarios, where the retrieval space (i.e., the fewshot training data) is limited. specifically, static retrieval may lack neighbors with high metrics in the case of limited retrieval space. even though such neighbors exist, static retrieval cannot be reliable for retrieving really helpful samples for target prediction, because its metric is not task-specific. in particular, for joint learning-based retrieval which minimizes the standard cross-entropy based loss, although the retrieval metric is updated towards the downstream task, it suffers from the gradient vanishing problem during the optimization process as quantitatively measured in fig. 2 (see §5.2 later). as a result, in a few-shot scenario, the retrieval metric might be not optimized well due to insufficient training data. to overcome the aforementioned challenges, we propose two novel training objectives, namely expectation maximization-based loss (em-l) and ranking-based loss (r-l), for learning to retrieve examples from a limited space more effectively. both objectives are committed to obviating the gradient vanishing problem and prioritizing more beneficial examples for specific downstream tasks. in the em-l approach, the retrieved examples are treated as latent variables, and an iterative process of expectation-step and maximization-step is employed until convergence @xcite . the posterior distribution of the latent variable is estimated to measure the importance of candidate examples in the e-step, while the m-step maximizes the expectation log-likelihood. by approximating the retrieval metric according to the posterior probability, more productive examples could be recalled for downstream tasks with limited training data. following a similar idea, r-l optimizes an additional ranking loss function to provide more direct supervision to the examples retriever, which draws inspiration from pair-wise ranking algorithm @xcite @xcite . such a tailored loss measures the consistency between the retrieval metric and the auxiliary function associated with each example for classification purposes. minimizing the loss could effectively strengthen the supervision signals for the example retriever. our experimental evaluation on ten text classification datasets demonstrates the superiority of em-l and r-l over existing retrieval methods within a limited retrieval space. the comparative analyses further confirm that em-l and r-l alleviate the weak supervisory signals and gradient vanishing issue suffered by joint learning-based retrieval. our contributions could be summarized as follows: ## conclusion this paper studies the retrieval-augmented methods for few-shot text classification and demonstrates the challenges which hinder their success: it is impossible to retrieve semantically similar examples by using an off-the-shelf metric and it is difficult to optimize a plausible metric by minimizing the standard cross-entropy loss. accordingly, it proposes two novel training objectives, em-l and r-l, which provide stronger supervision signals to train the retrieval metric effectively in few-shot scenarios. it is worth mentioning that the idea of searching within limited examples bears similarity to the concept of demonstration selection in recent large language models (llms). exploring the application of our methods in llms holds promise for future research. input: startup times are incredibly long : over two minutes. the sentiment polarity of startup times @xmath0.