Instruction Tuning with Human Curriculum (2024)

Abstract: In this work, we (1) introduce Curriculum Instruction Tuning, (2) explore the potential advantages of employing diverse curriculum strategies, and (3) delineate a synthetic instruction-response generation framework that complements our theoretical approach. Distinct from the existing instruction tuning dataset, our generation pipeline is systematically structured to emulate the sequential and orderly characteristic of human learning. Additionally, we describe a methodology for generating instruction-response datasets that extensively span the various stages of human education, from middle school through the graduate level, utilizing educational subject catalogs.Before training, we meticulously organize the instruction data to ensure that questions escalate in difficulty regarding (A) the subject matter and (B) the intricacy of the instructions. The findings of our study reveal that substantial improvements in performance can be achieved through the mere application of curriculum ordering to instruction data—achieving gains of +4.76 on TruthfulQA, +2.98 on MMLU, +2.8 on OpenbookQA, and +1.28 on ARC-hard—compared to random shuffling. This enhancement is achieved without incurring additional computational expenses. Through comprehensive experimentation, we observe that the advantages of our proposed method are consistently evident across nine benchmarks.

## introduction in contemporary times, state-of-the-art instructionfollowing models like chatgpt and gpt-4 (ope-nai, 2023) have drawn attention owing to their unparalleled proficiency and versatility. a notable advancement over previous generation large language models (llms), like gpt-3 @xcite 2020), is their impressive capability to adeptly comprehend and act upon human instructions, where this alignment is attributed to the additional instruction tuning process @xcite . as these models continue to display progress, numerous research studies have offered many intriguing insights on instruction tuning through their endeavors to make models follow more complex instructions and enhance performance across a broad spectrum of tasks. for instance, various studies emphasize the significant influence of instruction data quality @xcite and the incorporation of diverse instruction formats @xcite on overall performance. furthermore, including step-by-step reasoning @xcite within the responses has been demonstrated to improve performance and elevate the reasoning ability of the language model @xcite . while recent research has offered valuable insights into optimizing data formats to a better form, exploring how to efficiently order and collect such data in a more grounded, trackable manner remains elusive, often relying on randomized or undirected diversity as the prevailing norm. ensuring efficiency in the instruction tuning process is important as extended instruction tuning undermines the inherent capability of llm. figure 1 : overview of our educational framework. we create a dataset based on a continuum from secondary school to grad school, extracting multiple concepts from each course. for every concept, we formulate 19 questions of varied cognitive levels using bloom's taxonomy. meanwhile, since the architectures of neural network innately emulates the human brain @xcite , adopting a learning process analogous to human education -a highly organized approach, progressively refined and empirically proven effective over centuries -constitutes a logically coherent and methodologically robust learning strategy for the machine as well @xcite . while many studies within the realm of curriculum learning have demonstrated the efficacy of this hypothesis in reaching faster convergence and finding better local minima, these investigations have predominantly offered a nuanced micro view, mostly confined to a specific task. to draw an educational analogy, such studies are akin to observing how students behave when learning a particular subject within the vast curricula. venturing beyond the niche perspective, our study aims to explore a comprehensive, holistic viewpoint on curriculum learning in the knowledge domain. specifically, we conceptualize the language model as a middle school student about to progressively acquire intellectual knowledge from educational institutions such as high schools and universities over the coming decades. and attempt to guide the student by the fundamental principle of learning from simple to complex @xcite based on two primary distinct dimensions: (1) educational stage: sequentially mastering elementary to intricate concepts and (2) cognitive hierarchy: gradually deepening the understanding of each concept. for instance, in mathematics, humans initiate the learning process with the fundamental concept of addition, gradually pro-gressing to more complex concepts like subtraction and multiplication by exploiting previously learned concepts to ease the learning @xcite . furthermore, when humans learn multiplication, the initial stage usually involves rote memorization of the times tables, progressively deepening the comprehension of the concept to the extent where we expand its application to real-world situations. this cognitive process enables the human intellect to traverse diverse fields, aligning massively multidomain knowledge. to systematically explore the potential merits of the interplay between educational curriculum and human cognitive process, we curated a massive synthetic knowledge instruction dataset and its training method called corgi (cognitively rigorous instructions). as illustrated in figure 1 , we initially establish a continuous progression across educational stages by integrating concrete educational frameworks provided by international secondary education curricula (i.e., cambridge igcse) and a combination of several university catalogs. subsequently, using a teacher model like chatgpt, we extracted various topics covered in every course at each educational level. based on the learning objectives in bloom's taxonomy @xcite , we crafted a comprehensive set of questions for each topic, with varying degrees of cognitive level. a standout feature of our dataset is its rich metainformation for each data point, facilitating the generation of coherent and contextually meaningful training data sequences. as shown in step 1 step 2 step 3 choose learning goals: extract relevant, real-world educational concepts from existing human curricula. develop learning materials: analogous to the human remote learning setup through pre-made instructions revise learning materials: go through instruction materials to filter out unclear ones utilize existing school education curricula to determine what should be trained to create a generally-performant agent. phys008: physics for architects i -an introduction to the classical laws of mechanics, including static equilibrium, elasticity, and oscillations,... secondary school university grad. school "progressive deepening of knowledge allows human intellect to traverse diverse fields, aligning massively multi-domain knowledge. this can apply to llms." remember understand apply bloom's taxonomy school curriculum utilize a teacher model to synthetically extract concepts from course descriptions. static equilibrium refers to the state of an object when all the forces acting on it are ... static equilibrium energy conservation hooke's law utilize teacher model and corgi question template (19 cognitively rigorous formats) to synthetically generate instructions from extracted concepts. also collect answers to these instructions from the same teacher model, using long-form explanation-inducing system messages. ## corgi corgi is a structured educational model that mimics the educational journey of a student. in this section, we delve into the detailed process of constructing our dataset and efficient training method inspired by the human knowledge acquisition process. ## background cognitively understanding human learning processes. "where do we begin to improve human thinking?" @xcite . among diverse learning theories, bloom's taxonomy @xcite ) is a well-cited approach, categorizing learning processes into six hierarchical stages, ranging from simple to complex and concrete to abstract: remembering, understanding, applying, analyzing, evaluating, and creating @xcite . its effectiveness spans diverse subjects, from math to political sciences @xcite @xcite . cognitive load theory underscores the significance of managing mental exertion during learning. the theory served as a major theory for classroom instructional design @xcite . with the rise of e-learning in the 2000s, the theory was again widely applied to designing effective instructional strategies @xcite @xcite . a major effort was devoted to finding strategies for a remote setup where learners communicate with teachers through pre-made instructions. benefiting neural networks with human learning processes. machine learning can benefit from adopting human-centric approaches. curriculum learning, for instance, stands as a research area that arranges training data in a meaningful sequence, showcasing its potential to expedite convergence while enhancing generalization @xcite @xcite @xcite @xcite struction tuning methods. some instruction tuning methods rely on what can be considered an "unstructured curriculum." this means they have less control over the progression and complexity of the instruction data presented to the language model during training. for example, the method behind wizardlm creates instructions of varying difficulty using an evolutionary algorithm but does not strictly follow a predefined structure in the complexity or domain of the instructions. wizardlm's approach, characterized by its innovative use of an evolutionary algorithm, evol-instruct, generates increasingly complex instructions to challenge and refine the model's abilities. on the other hand, citing @xcite takes a different path by utilizing a teacher-student dynamic to craft a curriculum that emphasizes the revision and refinement of responses based on predefined criteria, aiming for gradual improvement in handling instructions. what achieved our performance improvement is that corgi integrates structured progressions both in the (1) content, akin to a traditional school curriculum, and in the (2) difficulty of instructions, guided by bloom's taxonomy. this dual-layered progression ensures that the model not only covers a wide range of knowledge areas but also develops the ability to process and respond to instructions of varying cognitive demands systematically. unlike wizardlm, which primarily focuses on generating complex instructions without a specific educational framework, or citing, which centers on the refinement of responses, corgi's methodical approach ensures a balanced and structured exposure to both knowledge domains and cognitive skills. this structured progression is absent in the more dynamically generated instruction sets of wizardlm and the feedback-oriented refinement process of citing, marking a distinction in how corgi approaches instruction tuning with human curriculum. ## conclusion in this work, we introduced corgi, a novel methodology for instruction tuning in large language models that employ a structured pedagogyinspired dataset. our methodology not only surpasses existing benchmarks in both reasoning and knowledge-based tasks but also achieves this efficiency without escalating computational demands. moreover, the observed efficacy of interleaved sorting and two-tier filtering underlines the crucial role of structured, high-quality data in model performance. collectively, these findings illuminate the potential of leveraging educational paradigms to elevate the capabilities of machine learning models.