Multi-Scale Prompt Memory-Augmented Model for Black-Box Scenarios (2024)

Abstract: Black-box few-shot text classification handles text classification in limited data without accessing the parameters and gradients of language models (LMs). Existing black-box optimization methods have demonstrated strong few-shot learning capabilities. However, they still require numerous LMs’ calls to search optimal prompts, thus resulting in overfitting performance and increasing computational cost. To address this issue, we present MuSKPrompt (Multi-scale Knowledge Prompt for Memory Model), an efficient multi-scale knowledge prompt-based memory model in black-box few-shot text classification task. MuSKPrompt extracts instance-level and class-level knowledge at different scales and stores them in memory banks during training. Then, it references multi-scale memory banks to perform quick inference on new samples via a novel scoring module. MuSKPrompt achieves competitive performance in limited data through multi-scale instance-level and class-level knowledge. Moreover, it realizes gradient-free optimization with zero training parameters in the black-box scenario. Experiments on different benchmarks and parameter analysis demonstrate the effectiveness and efficiency of MuSKPrompt in black-box few-shot text classification tasks.

## introduction over the past few years, large language models (llms) have achieved significant success @xcite @xcite . @xcite propose in-context learning (icl), which aids lms in adapting to downstream tasks by providing a few context examples before the input. due to its demonstrated good performance, prompt-based learning has become a popular method for low-resource adaptation of llms to downstream tasks. however, icl exhibits significant instability, and its performance depends @xcite . on the selection of context examples @xcite . to that end, numerous studies have focused on optimizing continuous prompts @xcite , ensuring that most parameters of llms remain unchanged. however, they are ineffective in certain situations where access to internal parameters and gradients of llms is restricted, a.k.a. the black-box scenario. many existing llms provide only apis, thus making gradient-based prompt learning challenging @xcite . furthermore, deploying llms in the black-box manner across various industries has become a trending topic @xcite @xcite . existing approaches @xcite employ gradient-free optimization methods to search for optimal continuous prompts. rlprompt @xcite and tempera @xcite use reinforcement learning to find optimal discrete prompts. however, these methods require many queries to lms, resulting in low computational efficiency. promptboosting @xcite constructs numerous weak learners by pairing the prompt pool generated by t5 with the output distribution of the lms and then integrates these weak learners using the adaboost algorithm. it requires a larger lms to generate the prompt pool. to this end, knn prompting @xcite introduces a non-parametric memory module and achieves excellent performance in text classification tasks. the memory module, storing knowledge from downstream tasks, has demonstrated excellent performance in text classification tasks. in natural language processing (nlp), memory models have been applied to various tasks such as named entity recognition and machine translation @xcite . however, the application of memory models to few-shot classification in the black-box scenario remains unexplored. the term "multi-scale" in this paper is referenced from previous works @xcite . moreover, previous works @xcite have demonstrated that prompts composed of more examples can assist lms in learning deeper knowledge. @xcite demonstrate that in shallow layers, label words accumulate information from demonstrations to form semantic representations, while deep layers integrate information from these label words. based on these insights, we propose muskprompt, a novel non-parametric memoryaugmentation method. multi-scale prompts allow lms to extract deep and shallow instance-level and class-level knowledge. we store this knowledge in a memory bank and accomplish text classification tasks through a scoring module. experimental results demonstrate that muskprompt achieves competitive performance in few-shot text classification tasks in the black-box scenario. importantly, because it doesn't require training to find optimal prompts, it simply stores instance-level knowledge obtained through fewer than 10 api calls for each instance, as shown in the contributions of this paper can be summarized as follows: in-context learning with the success of icl @xcite @xcite , investigating why icl is suitable for llms across various tasks has become a prominent research topic. recent work @xcite @xcite found that in-context demonstrations with different numbers and permutations can influence the model's performance on downstream tasks exhibiting significant instability and vulnerability. to address this issue, @xcite proposes to select diverse and representative contextual examples to achieve more robust and superior performance. self-adaptive icl @xcite employs a topk @xcite selection module and a minimal description length ranking module, aiming to select highquality contextual examples. furthermore, to exploit the capabilities of llms in icl, @xcite and @xcite introduce metalearning, which enables llms to better adapt to icl settings. @xcite , @xcite and @xcite found significant label bias in icl, propose to calibrate the bias via either probing the bias or reversing the conditional prediction formulation. our work adopts contextual examples as prompts to enhance the few-shot learning capability of lms. memory models similar methods for constructing memory modules have been proposed in the fields of computer vision @xcite and nlp @xcite @xcite @xcite . retroprompt @xcite constructs an open knowledge base, enabling the model to retrieve relevant examples from the training corpus as enhanced prompts. knn prompting @xcite introduces a retrieval mechanism in icl, addressing issues related to label bias and context length limitations. our approach falls into the category of memory models. the main difference is the use of a memory bank that stores class-level and instance-level knowledge at multi-scale to guide lms during reasoning. ## problem formulation for notation, we define the training set in few-shot learning as d @xmath0, y i } ## method in this section, we introduce muskprompt, a framework that employs multi-scale prompts to encapsulate knowledge, thereby guiding lms to excel in various few-shot text classification tasks within the black-box scenario. the primary objective of muskprompt is to extract knowledge at both the class and instance levels from context examples across multiple scales. the non-parametric memory bank stores this knowledge at multiple scales and varying levels to enhance the diversity of the knowledge base. finally, the scoring module utilizes this enriched few-shot knowledge to complete classification tasks. as shown in figure 1 , muskprompt consists of the following modules. ## conclusion in this paper, we present muskprompt, a novel memory-augmented framework. it does not require access to the internal information of lms and achieves remarkable performance with few-shot training samples. muskprompt acquires instancelevel and class-level knowledge of various depths through multi-scale prompts. this knowledge is then stored in a memory bank. the scoring module efficiently performs classification tasks using the stored knowledge, without training any parameters. the significant improvements brought solely by memory knowledge will encourage future memory model solutions for few-shot classification in the black-box scenario.