Tutor- ICL : Guiding Large Language Models for Improved In-Context Learning Performance (2024)

Abstract: There has been a growing body of work focusing on the in-context learning (ICL) abilities of large language models (LLMs). However, it is an open question how effective ICL can be. This paper presents Tutor-ICL, a simple prompting method for classification tasks inspired by how effective instructors might engage their students in learning a task. Specifically, we propose presenting exemplar answers in a *comparative format* rather than the traditional single-answer format. We also show that including the test instance before the exemplars can improve performance, making it easier for LLMs to focus on relevant exemplars. Lastly, we include a summarization step before attempting the test, following a common human practice. Experiments on various classification tasks, conducted across both decoder-only LLMs (Llama 2, 3) and encoder-decoder LLMs (Flan-T5-XL, XXL), show that Tutor-ICL consistently boosts performance, achieving up to a 13.76% increase in accuracy.

## introduction with the rapid advancement of large language models (llms), in-context learning (icl), which involves performing various tasks by learning from only a small number of examples within the context of a single prompt, has become a dominant paradigm in natural language processing @xcite . with icl, the likelihood of any answer for the test example is conditioned on the provided icl exemplars @xcite . the underlying assumption of icl is that llms can thoroughly review these exemplars, identify the hidden patterns crucial for input-label mappings, and consequently make correct predictions @xcite . however, recent studies have provided some evidence that llms, particularly smaller ones (less than or around 10 billion parameters), are unable to fully utilize the provided exemplars. for example, @xcite found that smaller llms allocate less attention mass to icl examples than larger models. similarly, @xcite indicated that smaller llms have a lower ability to adjust their semantic priors based on the provided icl examples than their larger counterparts. we also conduct a preliminary experiment indicating that llms do not always produce the correct answer even when it is provided as one of the exemplars (section 5.1 and ## related work in-context learning (icl) the existing literature on in-context learning research can be broadly divided into two categories: (1) analytical studies, which aim to uncover the underlying mechanisms of how llms perform icl @xcite @xcite , and (2) improvement studies, which seek to enhance icl performance through various methods such as exemplar selection @xcite , exemplar ordering @xcite , and instruction calibration @xcite . the strand of work that is most closely related to ours focuses on changes to the prompt template itself, as explained next. studies on icl template components a number of studies have examined the effects of the components within prompt templates. @xcite , referred to as decon-icl hereafter, showed the benefits of briefly repeating text and the importance of reiterating inline instructions. @xcite proposed re2 as a simple strategy of reading the question again to improve icl performance. @xcite examined the effects of flipped or semantically-unrelated labels on icl. most of these studies rely on a standardized prompt template that includes four components: task instructions, exemplar inputs, exemplar labels, and inline instructions @xcite . the key distinction of our work lies in our out-ofthe-box approach. rather than focusing solely on these four standard components, we take it one step further by exploring the incorporation of new elements. we verify that these additional elements can significantly improve icl performance. prompting multi-step reasoning a lot of work in prompt engineering has focused on eliciting intermediate reasoning steps to enhance performance, as seen in various techniques like chainof-thought @xcite , tree-of-thought @xcite , and visualization-of-thought @xcite , among others. while effective, this comes with limitations, such as high computational costs from generating longer outputs and difficulties in acquiring high-quality exemplars for icl. in contrast, tutor-icl bypasses the need for generating additional intermediate steps or new exemplars, providing a more cost-effective approach to improving performance. ## tutor-icl our work is primarily motivated by @xcite and @xcite , which show that simply repeating the same instruction or question could boost llm performance. this indicates that providing more careful guidance could further enhance llms' icl performance, highlighting a potential area for improvement. tutor-icl (figure 1 ), effectively guides llms throughout the icl process by incorporating three novel ideas: (1) comparative answer format, which provides the answers in a comparative form to elicit deeper thinking from multiple answer perspectives; (2) glanceat-the-test framework, which informs llms of the test instance in advance, leading to a more efficient search and focus on relevant exemplars; (3) summarization step, which makes llms to summarize the given exemplars before attempting the test instance, similar to human practice.