Toward In-Context Teaching: Adapting Examples to Students’ Misconceptions (2024)

Abstract: When a teacher provides examples for a student to study, these examples must be informative, enabling a student to progress from their current state toward a target concept or skill. Good teachers must therefore simultaneously infer what students already know and adapt their teaching to students’ changing state of knowledge. There is increasing interest in using computational models, particularly large language models, as pedagogical tools. As students, language models in particular have shown a remarkable ability to adapt to new tasks given small numbers of examples. But how effectively can these models adapt as teachers to students of different types? To study this question, we introduce a suite of models and evaluation methods we call AdapT. AdapT has two components: (1) a collection of simulated Bayesian student models that can be used for evaluation of automated teaching methods; (2) a platform for evaluation with human students, to characterize the real-world effectiveness of these methods. We additionally introduce (3) AToM, a new probabilistic method for adaptive teaching that jointly infers students’ past beliefs and optimizes for the correctness of future beliefs. In evaluations of simulated students across three learning domains (fraction arithmetic, English morphology, function learning), AToM systematically outperforms LLM-based and standard Bayesian teaching methods. In human experiments, both AToM and LLMs outperform non-adaptive random example selection. Our results highlight both the difficulty of the adaptive teaching task and the potential of learned adaptive methods for solving it.

## introduction good teaching is adaptive to students' specific beliefs and preconceptions @xcite . imagine, for example, that you have been tasked with tutoring students fraction arithmetic. you may start by first probing a student's understanding, asking if addition: else: if addition: else: ? 𝟥 𝟨 teacher what is ? 𝟣 𝟤 × 𝟣 𝟨 what is ? 𝟣 𝟧 × 𝟤 𝟧 incorrect. it's . 𝟣 𝟣𝟤 1) infer student misconceptions 2) choose most informative example prog + if addition: else: prog + if addition: else: multiplication learner addition learner over-generalize addition if denominators are equal make common denoms multiply nums make common denoms add nums over-generalize addition over-generalize mult. 𝟣 𝟧 × 𝟤 𝟧 = 𝟣 𝟤 + 𝟤 𝟦 = 𝟤 𝟧 𝟦 𝟦 add nums & denoms multiply nums & denoms make common denoms add nums 𝟣 𝟤 × 𝟣 𝟨 = 𝟥 𝟨 𝟣 𝟣𝟤 𝟥 𝟨 𝟤 𝟧 incorrect. it's 𝟤 𝟤𝟧 . make common denoms add nums make common denoms multiply nums if equal denoms: multiply nums else: multiply nums & denoms 𝟤 𝟤𝟧 multiply nums & denoms target concept target concept ## 13283 as this example highlights, interactions with students reveal insights about their misconceptions, and these misconceptions in turn influence the course of effective teaching. a good teacher should provide different problems for these students to target their specific misconceptions: the additiongeneralizer would benefit from multiplication examples, especially those with common denominators, while the multiplication-generalizer would benefit from addition examples. what does this mean for nlp? computational models-particularly language models (lms)-are increasingly used as pedagogical tools @xcite . but it is unclear how effectively any of today's models can tailor instruction to perform "incontext teaching" ( §2) for students with differing degrees of skill and prior knowledge. in this work, we draw on a long line of literature on rational models of pedagogy @xcite to study this question. to do so, we introduce adapt (adaptive teaching), an evaluation suite targeted at teaching students with varied prior misconceptions ( §3). in adapt, a teacher is tasked with selecting examples to teach a particular target concept. as shown in figure 1 , the teacher selects examples one by one and can observe predictions made by the student. importantly, the student has prior misconceptions that are unknown to the teacher. adapt is designed such that correct inferences about student misconceptions can enable more efficient learning. adapt has two components: simulated and human experiments allow us to characterize the pedagogical capabilities of teachers along several dimensions: their inferences about student beliefs ( §5.3, §6.3), adaptivity of chosen examples ( §5.4), and differences in teaching mathematical concepts ( §5.5). in addition to these evaluations, we introduce: 3. a new probabilistic method, atom (adaptive teaching towards misconceptions), which performs online inference of student priors, then uses these inferences to select informative teaching examples. atom provides proof-of-viability for adaptive teaching methods in simulated and human students ( §4.1). using adapt evaluations, we characterize atom, gpt-4, and a range of other methods. in simulated students, we find that while gpt-4 exhibits some adaptation to student misconceptions, there is room for improvement with adaptive approaches; it substantially underperforms atom, suggesting promise in using adaptive methods ( §5). in human experiments, however, both atom and gpt-4 outperform random example selection, highlighting the potential of learned models (of various kinds) for adaptive teaching ( §6). 1 ## preliminaries we formulate our problem setting as one in which a teacher interactively provides feedback to a student to communicate a new concept (a mapping from inputs x to outputs y). a teacher observes a sequence of (x, ŷ) pairs guessed by a student, and must infer what additional training example (x, y) pair will most improve the student's understanding of the concept. we term this problem in-context teaching to draw an analogy to the widely studied phenomenon of "in-context learning" @xcite @xcite . formally: if : undefined else: a x+ if addition: else: if addition: else: if addition: else: if : undefined else: a x+ if addition: else: mult prog concepts which determine their predictive distributions if : undefined else: ax+ b greater_2(x): concept space: 9 programs odd even divisible_20 divisible_3 . . . greater_20 greater_1 . . . prime positive 7 f(x) . +ed +d 0.08 0.6 concept space: possible settings of 10,528 parameters of generative model from the set of concepts in the concept space teacher aims to teach a specific target concept students concept space: 4,158 programs +ed learner +d learner +consonant+ed learner f learner b learner target concept: program for fraction arithmetic target concept: generative model of verbs target concept: wug(x) function that takes in an input x and returns an output have different priors over the concept space high prior belief in programs that over-generalize addition procedure to multiplication, eg: … … … high prior belief in programs with incorrect f/correct b a 1 prog + if addition: else: prog + if addition: else: multiplication learner addition learner 3 sub-programs each +ied learner high prior belief in incorrect parameters for +ied class +ied y fy zz gamify 0.27 0.05 0 1.0 0.6 hop cope accept … … … +consonant+ed +consonant+ed greater_4(x): 7 3 high prior belief in programs with correct f/incorrect b if : undefined else: a x+ greater_2(x): 3 3 human & simulated [uniform belief over values of a] … … high prior belief in programs that over-generalize multiplication procedure to addition, eg: 𝗉 [ 𝟣 𝟧 × 𝟤 𝟧 → 𝟤 𝟧 ] = 𝟢 . 𝟫 𝗉 [ 𝟣 𝟧 + 𝟤 𝟧 → 𝟥 𝟣𝟢 ] = 𝟢 . 𝟫 … 0.08 0.27 0.6 0.5 0.5 continuous distribution … … 𝗉 [ 𝗐𝗎𝗀(𝟥) → 𝟫 ] = 𝟢 . 𝟪 wug(3) 𝗉 [ 𝗀𝖺𝗆𝗂𝖿𝗒 → ] = 𝟢 . 𝟪 gamify +ed fractions add & multiply functions verbs categorize english past tense verbs functions learn what wug(x) does make common denoms add nums make common denoms multiply nums add nums & denoms multiply nums & denoms make common denoms add nums multiply nums & denoms different student types adapt has three tasks: fractions, verbs, and functions. for the fraction and function tasks, a student's concept space consists of programs; for the verbs task, a student's concept space is the space of generative models corresponding to english past tense verb classes. in each round of teaching, the teacher first presents the student with an input x i , the student produces a guess ŷi , and then the teacher reveals the true y i . foot_2 given this setup, a teaching strategy is a policy p t (x, y | x, y) that chooses examples to maximize the probability that the student assigns to h * . in the optimal teaching (ot) framework of @xcite , for example, p s (h | x, y) is assumed known, and teachers choose examples: in the running example, this criterion is more likely to select examples of addition for a student who has already mastered multiplication, and vice-versa. in addition to this greedy approach, it is possible to plan sequences of informative examples for students @xcite . as discussed in §1, however, the assumption that teachers have exact knowledge of p s (h | x, y) is often unrealistic-real-world teaching involves students of many different types, whose beliefs and misconceptions may not be known (or easily discerned) a priori. thus, we study teaching when students' priors are themselves unknown. we assume that students are drawn from a distribution over student types p(α), each associated with a concept prior p(h | α). in the running example, these priors may correspond to different beliefs about the algorithms for fraction addition and multiplication, with "addition generalizers" assigning high probability to a spurious multiplication algorithm that manipulates only numerators. in this setting, teachers must still implement an effective example selection policy p t (x, y | x, y); however, choosing informative examples now requires inferring student priors in order to estimate the effect of these examples on p s (h * | x, y). in the next section, we describe our proposed framework for evaluating adaptive teaching policies. in §4, we describe a set of candidate adaptive teaching policies (including our new atom method), and in §5 and §6 use adapt to evaluate these teaching policies with simulated and human students. ## the adapt evaluation framework adapt has two parts: an offline evaluation framework with simulated students ( §3.2), and a platform for doing experiments with human students ( §3.3). we first describe the tasks in adapt ( §3.1). an overview of adapt is shown in figure 2 . ## discussion both our human and simulated results show that atom and gpt-4 exhibit pedagogical ability over random example selection. atom's performance with human students suggests that the bayesian assumptions made by atom are accurate models of some aspects of human learning. across human and simulated experiments, we find evidence of some adaptivity in gpt-4, though less than in atom, both in the examples selected ( §5.4) and inferences about student type ( §5.3, §6.3). see §a for additional analyses about how adaptive selected teaching examples are to student beliefs. we also observe other qualitative limitations of gpt-4 as a teacher: selecting the same teaching examples multiple times or terminating teaching early due to an incorrect belief that all teaching examples have been exhausted. it is important to highlight that because we use ground truth outputs (e.g., ground truth function evaluations) with gpt-4, the gpt-4 results represent an upper bound on gpt-4's performance. despite these limitations, however, gpt-4 performs comparably to atom with human students, suggesting pedagogical benefits beyond adaptivity. in particular, the analysis in §5.5 suggests that lms may encode information about human learning that is hard to represent in more structured approaches like atom-e.g., that it is easier for humans to learn the weights of a line from inputs closer to the origin. together, our results point to complementary advantages of lm teachers like gpt-4 and more structured, probabilistic methods like atom. they suggest that there is substantial headroom to improve real-world teaching by augmenting the in-ferences of structured models with richer information about the priors that humans bring to learning, whether by combining such structured methods with information encoded in lms or by developing other rich models of student priors-e.g., by learning more complex "student types" from naturally occurring data. we perform an initial experiment with human students in this direction by combining gpt-4 and atom but do not find improvements compared to either teacher alone; see §f.3 for details. other directions for future work include modeling more complex student phenomenaaccounting for students who ask questions, reason pedagogically about teacher intentions, and provide feedback to teachers (chen et al., 2022)-and creating methods for adaptive teaching with natural language explanations in real-world teaching domains. ## related work this work builds on a long line of work in rational models of pedagogy, including bayesian models like those described by @xcite and @xcite , as well as improved planning and inference procedures like the one described by @xcite . past work generally assumes students' initial belief states are known. in parallel, @xcite use an inverse planning model to infer students' prior beliefs from their actions, and @xcite show that human teachers adapt examples to these prior beliefs. this work is also closely related to other bodies of work that aim to infer student knowledge. item response theory (irt) infers a scalar measure of student's skill based on their responses to questions @xcite . knowledge tracing (kt) models students' evolving knowledge states over time separately for a fixed set of skills @xcite ; previous work has used both bayesian methods for modeling individual students' prior knowledge @xcite and neural models for modeling @xcite and targeting (srivastava and goodman, 2021) students' evolving learning states. in contrast to irt and kt, our work aims to infer the student's entire prior and posterior over the concept space; these inferences in turn enable more fine-grained design of individual teaching examples. inferring student misconceptions from errors also uses tools from a broader literature on computational models of theory of mind. prominent work includes general-purpose bayesian models of other agents' beliefs and desires @xcite and models of pragmatic inference grounded in recursive reasoning about speakers and listeners @xcite . more recent work has studied theory of mind capabilities in lms; they find largely negative results in unaugmented llms @xcite but positive results from lms augmented with structured belief representations @xcite . recent work has also explored llms' theory of mind abilities in teaching smaller lms @xcite . there is also a large body of work on how to optimally provide and interpret human supervision for ml models. general frameworks for this problem include machine teaching @xcite and cooperative irl @xcite ; related ideas appear in program synthesis @xcite , robot learning @xcite , and natural language processing @xcite as well. there has been increased interest in using llms to assist or supplement human teachers. see kas ## conclusion we introduce adapt, an evaluation suite measuring how effectively automated teaching methods can teach students with different prior misconceptions. we also introduce atom, a two-part probabilistic approach to adaptive teaching that maintains explicit inferences about student priors. our evaluations of atom, llms, and other probabilistic baselines with both simulated and human students highlight the potential of learned adaptive models for solving the adaptive teaching task.