Skills-in-Context: Unlocking Compositionality in Large Language Models (2024)

Abstract: We investigate how to elicit compositional generalization capabilities in large language models (LLMs). Compositional generalization empowers LLMs to solve complex problems by combining foundational skills, a critical reasoning ability akin to human intelligence. However, even the most advanced LLMs currently struggle with this form of reasoning. We examine this problem within the framework of in-context learning and find that demonstrating both foundational skills and compositional examples grounded in these skills within the same prompt context is crucial. We refer to this prompt structure as skills-in-context (SKiC). With as few as two exemplars, this in-context learning structure enables LLMs to tackle more challenging problems requiring innovative skill combinations, achieving near-perfect systematic generalization across a broad range of tasks. Intriguingly, SKiC also unlocks the latent potential of LLMs, allowing them to more actively utilize pre-existing internal skills acquired during earlier pretraining stages to solve complex reasoning problems. The SKiC structure is robust across different skill constructions and exemplar choices and demonstrates strong transferability to new tasks. Finally, inspired by our in-context learning study, we show that fine-tuning LLMs with SKiC-style data can elicit zero-shot weak-to-strong generalization, enabling the models to solve much harder problems directly with standard prompting.

## introduction large language models (llms) have achieved great success in solving natural language processing (nlp) tasks ( smith ideally, if an llm has already learned a rich set of knowledge and foundational skills, it should be able to solve any problem whose solutions are composable from these skills. to unlock such great potential, the key is to teach the llms how to use these skills to construct a solution to more difficult problems. towards this goal, there have been a series of in-context learning strategies developed to improve the reasoning and composition capabilities. notably, chain-of-thought (cot) prompting (wei et al., 2022b) significantly improves the reasoning performance of llms by demonstrating how to approach a complex problem through a sequence of basic steps. follow-ups such as least-to-most prompting (zhou et al., 2022a) and decomposed prompting (khot et al., 2022) propose a two-stage strategy, which first decomposes the problem into sub-problems, and then solve and combine them sequentially. although these methods significantly boost the performance in solving many challenging compositional generalization tasks, they usually fail over problems that are significantly harder than the ones they have seen. moreover, least-tomost prompting and decomposed prompting are restricted to solving problem classes that can be decomposed as a sequence of sub-problems. and for problems with general computation graphs (dziri et al., 2023), it is generally less intuitive, if not possible, to construct the prompting exemplars. in this paper, we examine how to elicit strong compositional abilities in llms within the frame-figure 1 : skills-in-context prompting. the prompt consists of three blocks: (i) the (basic) skills for solving a complex task, (ii) examples of how to compose the skills, and (iii) the problem to be solved. the above prompt will be fed into an llm to generate the output -see figure 26 for an example of the output. note that the compositional exemplars demonstrate how to explicitly ground the reasoning steps onto the basic skills (highlighted in colors). work of in-context learning. we find that the key insight is to teach the llm to explicitly ground each of its reasoning steps on the (more foundational) skills. to this end, it is crucial to demonstrate both the foundational skills and the compositional examples grounded in these skills within the same prompt context. we refer to this (one-stage) prompting structure as skills-in-context (skic). specifically, the skic prompt is constructed from three main blocks (figure 1 ). the first block contains a short (non-exhaustive) list of skills that llms may need to use in order to solve a more complex problem, which include the instructions of the skills. these skills can be distilled either manually or automatically via llms. the second part consists of a few (generally two) exemplars that demonstrate how to compose skills into a complex solution. the last part is the testing problem. interestingly, with both the skills and their explicit compositions presented in the context, the llms successfully learn how to ground reasoning steps onto the skills that they have already mastered, yielding much stronger generalization abilities. it allows llms to achieve near-perfect systematic generalization across a broad range of tasks. in addition, it also allows the llms to gen-eralize beyond the skills provided in the context and solve problems by more actively and explicitly using the vast reservoir of the internal skills they acquired during the prior pre-training stage. it clearly demonstrates that skic structure unleashes strong synergies between skills and their composition capabilities, which teaches llms to generalize to unseen (harder) problems that require innovative compositions of skills. furthermore, the skic structure is robust across different skill constructions (e.g., handcrafted or discovered by llms) and exemplar choices and demonstrates strong transferability to new tasks. finally, inspired by our in-context learning study, we show that fine-tuning llms with skic-style data can elicit zero-shot weak-to-strong generalization, enabling the models to solve much harder problems directly with standard prompting. ## skic: elicit compositionality with in-context skills and grounding while humans naturally exhibit compositional generalization in problem-solving, llms often struggle to compose basic skills to solve more difficult problems (dziri et al., 2023). empowering llms with the ability to compose skills that they have seen to solve more complex tasks is important to mirror human intelligence and to reach superintelligence. in this work, we investigate how to elicit compositionality of llms in in-context learning (icl) setting. in particular, we want to reveal how a meticulously designed prompt structure could greatly enhance the compositional ability. the insights obtained in the icl setting can also inspire how to further improve the fine-tuning (sec. 4). ## analysis of compositional abilities we perform experiments in two settings, where more details can be found in appendix d: systematic generalization: composition over in-context skills, where all the needed skills are provided in the context. we evaluate the exemplars in skic are either a subset of or the same as what have been used in baselines. ## beyond in-context learning inspired by the above in-context learning study, we show that instruction tuning data constructed with skic structure can be utilized to fine-tune llms, enhancing their easy-to-hard generalization capabilities. specifically, we generate training data by using gpt4 to produce answers for gsm8k problems with skic prompts. this ensures that the reasoning steps for each gsm8k problem are explicitly grounded on basic skills, as illustrated in figure 33 -34. using the gsm8k data annotated with skic-structure reasoning steps, we fine-tune llama2 models and evaluate their performance on math dataset, which consists of significantly more challenging evaluation problems compared to the training problems from gsm8k. the results, shown in figure 7 , indicate that fine-tuning with skic data significantly improves accuracy on math compared to training data annotated with cot reasoning steps (also by gpt4). this demonstrates that models fine-tuned with skic reasoning steps achieve better generalization to complex and challenging test cases. these findings suggest that skic could potentially replace cot in instruction tuning, eliciting stronger reasoning capabilities and enabling better weak-to-strong generalization. ## related work there has been a long history of studies on compositional generalization (lake and baroni, 2018; jia and liang, 2016; andreas, 2019; lake and ba- ## conclusion in this work, we examine how to elicit compositional generalization abilities in llms. specifically, within the in-context learning framework, we find that it is crucial to explicitly ground each of the reasoning steps on the foundational skills. to facilitate this, it is important to demonstrate both the foundational skills and the compositional examples grounded in these skills within the same prompt context. we refer to this prompt structure as skillsin-context (skic). skic demonstrates strong (nearperfect) systematic generalization abilities across many tasks and enhanced complex reasoning capabilities. notably, with skic, the llms could generalize beyond the skills provided in the prompting context and learns to activate the skills and knowledge that are acquired through earlier pre-training stages for solving unseen complex problems. furthermore, skic structure could be utilized in finetuning to improve the easy-to-hard generalization.