Latent Learningscape Guided In-context Learning (2024)

Abstract: The growing interest in leveraging large language models is driven by their exceptional imitation and reasoning capabilities. In-context learning (ICL), a streamlined method, has shown potential in boosting these models’ performance without modifying their underlying parameters, especially when supplied with suitable demonstrations. However, existing methods mainly choose demonstrations by comparing surface-level semantic similarities (e.g., based on embedding) and fall short of identifying the most fitting ones. This paper introduces the concept of a “latent learningscape”, a more nuanced representation that describes the characteristic of the demonstrations. Building on this concept, we develop a results-driven approach to characterize the latent learningscape features of demonstrations, which then inform the creation of more effective prompts. Through comprehensive testing across datasets in arithmetic, commonsense, and symbolic reasoning tasks, our approach outperforms leading models, showing an average increase in scores by 7.4 percentage points.

## introduction the captivating imitation and reasoning abilities exhibited by large language models @xcite @xcite have sparked growing interest in their applications. in-context learning (icl), as a lightweight method that refrains from altering model parameters, is increasingly proved by experiments to be able to enable large language models to achieve impressive results when appropriate demonstrations are supplied. these demonstrations, when appropriately chosen, effectively stimulate the latent capabilities within large models, thereby enhancing traditional methods manul q1: q2: ❌ ✅ q2 a2 x q1 a1 x llm …… the answer is 11 …… the answer is 9 we first introduce the term latent learningscape, which refers to the knowledge points, problemsolving approaches, or concepts that aid in solving problems. as depicted in fig. 1 , unlike demonstrations selected solely based on the similarity of surface-level semantics by embedding-based methods, q1 share a same problem-solving approach with x, could truely serve as an effective demonstration. in other words, q1 and x share a common latent learningscape. the problem-solving approach mentioned here is one of the latent learningscapes we refer to. when demonstrations possess or encompass the latent learningscape inherent in the test question, demonstrations can play a effective role. in leveraging icl, we encounter the following challenges: (1) how to identify the latent learningscape and the latent learningscape features behind demonstrations. (2) how to construct prompts that encompass sufficient latent learningscape features for correctly solving reasoning problems. in light of the challenges mentioned earlier, identifying the latent learningscape behind demonstrations without human intervention is difficult. however, it is possible to characterize the latent learningscape features of demonstrations by defining the differences between demonstrations. therefore, we propose a result-driven prompt construction method, without the explicit identifying the latent learningscape behind demonstrations. instead, we leverage the corrective nature of demonstrations to characterize the latent learningscape. specifically, we first establish two distinct question pools: the a-pool, containing questions answered correctly by the llm, and the c-pool, containing challenging questions answered incorrectly by the llm. next, we use questions from the a-pool as demonstrations to answer the questions in the c-pool, aiming to explore the representational capabilities of the demonstrations in the a-pool. we store the latent features of these demonstrations in vector form and use them to distinguish between demonstrations. then, we construct prompts using linearly independent vector groups to encompass a comprehensive range of latent learningscape features. finally, the answer to the current problem is obtained by a majority vote among the different groups of demonstrations defined by different linearly independent vectors. to demonstrate the effectiveness of our method, we conduct extensive experiments on reasoning problem datasets, including arithmetic, commonsense, and symbolic reasoning datasets. the re-sults indicate that our proposed method achieves the best or competitive performance compared to state-of-the-art methods, with an average score improvement of 7.4 percentage points. additionally, we conduct ablation experiments to validate the effectiveness of different parts of our method and its performance across different llms. in summary, our contributions include: our framework fully capitalizes on the excellent characteristics of latent learningscape features, constructing prompts that encompass a rich range of latent learningscape. this approach allows us to achieve superior results with lower energy consumption. ## method our approach consists of three steps, namely: (1) basic pools construction: establish the appropriate question pool and the challenging question pool. ## main experiments we conducted experiments using the widely adopted large language model, chatgpt(gpt-3.5turbo-0613), across three distinct categories encompassing a total of nine datasets to validate the feasibility and efficacy of our proposed method. the first categories include five mathematical datasets: gsm8k @xcite , aqua @xcite , mathqa @xcite (first 400 questions), svamp @xcite , and type 1 of numglue @xcite . the second category involves two commonsense reasoning datasets, namely commonsenseqa @xcite and strategyqa @xcite . the third category comprises two symbolic reasoning algorithm 2 latent learningscape feature characterization input: appropriate question pool a, challenging question pool c output: demonstration pool d 1: for each (q i , r i , a i ) in a do 2: ## conclusion in this paper, we introduced a latent learningscape that is more closely aligned with the essence of in-context learning abilities, representing knowledge points, problem-solving approaches, concepts, and themes that truly enable effective in-context learning. building upon this foundation, we proposed a result-driven three-stage method. firstly, we constructed latent learningscape features for demonstration example with the potential to serve as demonstrations. next, we generated several demonstration example sets covering a comprehensive latent learningscape by constructing linearly independent vector groups, which were used as demonstrations in the few-shot prompt. experimental results demonstrate that our method indeed enhances the model's in-context learning abilities, achieving significant performance improvements without a substantial increase in energy consumption. moreover, compared to the highest energyconsuming methods, our approach not only outperforms them but also achieves better results. ablation experiments confirm the effectiveness of our method across different models, indicating that we have genuinely identified examples with demonstration potential and, by defining their latent learningscape features, combined them into more valuable compositions.