Chain-of-Quizzes: Pedagogy-inspired Example Selection in In-Context-Learning (2024)

Abstract: In-context learning (ICL) has emerged as a powerful tool for enhancing large language models (LLMs) in addressing downstream tasks. In this paper, we explore the vital task of example selection in ICL by mimicking the human learning process. We propose a Chain-of-Quizzes (CoQ) framework inspired by educational theories such as Bruner’s Spiral Learning and Mastery Learning theory. Specifically, our framework employs the LLMs to answer the quiz (question in the example) to sift ‘good’ examples, combines these examples iteratively with the increasing complexity, and utilizes a final exam to gauge the combined example chains. Our extensive experiments on diverse reasoning datasets show the proposed approach outperforms baseline models. These findings underscore the framework’s potential for future research.

## introduction with the scaling up the model size and corpus size @xcite @xcite , large language models (llms) have brought innovation to many fields @xcite @xcite . one of the most important abilities of llms, in-context learning (icl), is a paradigm that allows language models to learn downstream tasks given only a few demonstrative examples @xcite . in this paper, we delve into the essential task of example selection within icl. since icl assumes that "llms can deduce many things from a few cases", like chain-of-thought that mimics the reasoning process of humans @xcite , we propose to rethink the selection of examples in * equal contribution. for the second question, we propose a chainof-quizzes (coq) framework inspired by two educational theories: bruner's spiral learning theory and mastery learning theory 3 . bruner's spiral learning theory proposes that learning should be built upon what learners have previously learned, and should gradually increase in complexity of examples @xcite . the mastery learning theory suggests that the learners should master the current examples before answering the new question @xcite . specifically, the coq framework engages the llm in answering the quiz (question in the example) to sift 'good' examples. it incrementally increases the complexity of the quiz based on the model's performance when combining the examples into a chain, and employs a final exam to evaluate the effectiveness of each example chain. our framework is further refined using majority voting to account for the inherent randomness in example selection. extensive experiments conducted across various datasets demonstrate that our approach yields superior results compared to baseline models. these initial findings highlight the framework's significant potential for future research. ## method in this section, we introduce details of the framework, whose illustration is shown in figure 2 . ## conclusion in this work, inspired by pedagogy theories, we propose a novel framework chain-of-quizzes (coq) to select examples for the in-context learning of llm. extensive results demonstrate the effectiveness of our approach. in the future, we will: 1) experiment with other question complexity measurements and 2) expand the coq to more tasks.