KARL : Knowledge-Aware Retrieval and Representations aid Retention and Learning in Students (2024)

Abstract: Flashcard schedulers rely on 1) *student models* to predict the flashcards a student knows; and 2) *teaching policies* to pick which cards to show next via these predictions.Prior student models, however, just use study data like the student’s past responses, ignoring the text on cards. We propose **content-aware scheduling**, the first schedulers exploiting flashcard content.To give the first evidence that such schedulers enhance student learning, we build KARL, a simple but effective content-aware student model employing deep knowledge tracing (DKT), retrieval, and BERT to predict student recall.We train KARL by collecting a new dataset of 123,143 study logs on diverse trivia questions.KARL bests existing student models in AUC and calibration error.To ensure our improved predictions lead to better student learning, we create a novel delta-based teaching policy to deploy KARL online.Based on 32 study paths from 27 users, KARL improves learning efficiency over SOTA, showing KARL’s strength and encouraging researchers to look beyond historical study data to fully capture student abilities.

## introduction flashcards help students learn answers to questions across subjects like trivia, vocabulary, and law @xcite . in education, a flashcard scheduler dictates when students review old flashcards and when they learn new ones @xcite . many schedulers use student models to predict the probability a student can recall a flashcard @xcite . a teaching policy then chooses the flashcards to show next based on recall predictions @xcite . teaching policies ensure personalized learning experiences by understanding a student's learning progress. thus, every student model must use informative features to capture a student's knowledge state @xcite . to predict recall, existing student models @xcite just use data from the student's study history, like their past answers and time since last review. this data models student behavior, but it ignores a key aspect of the card: its textual content. modeling the relations across flashcard content can enable student models to predict recall even on cards with no study data. for instance, if a student studies the question "who was the first u.s. president" for the first time, existing student models cannot discriminatively predict if the student knows the answer, as the card lacks study data. however, if the student already studied "who was the second u.s. president" and always answered it correctly, we can infer that the student knows u.s. presidents and likely can recall the first u.s. president. existing schedulers cannot make these semantic inferences, limiting their ability to predict and schedule flashcards with no study data. we propose content-aware flashcard schedulers-the first schedulers that exploit flashcard content. we connect this paradigm to deep knowledge tracing @xcite , which embeds student study histories @xcite to predict study correctness. some dkt models, like lm-kt @xcite , enhance these embeddings with language models, providing a strong basis for semantic inferences. while dkt models show promise on offline benchmarks, their adoption in real-world flashcard scheduling systems is absent, as there lacks concrete evidence that such models can improve learning ( §3). thus, our goal is to design a baseline content-aware scheduler using dkt and be the first to show that this model can successfully enhance student learning, motivating the benefits and potential of the paradigm. towards content-aware scheduling, we develop kar 3 l, a dkt student model using knowledge-aware retrieval and representations for retention student study history … @xmath0.8 semantically similar history predicted recall bert what's the capital of spain? madrid who's the 20th u.s. president? james garfield who's the 1st u.k. pm? robert walpole who's the 2nd u.s. president? john adams who's the 2nd u.s. president? john adams @xmath1.s. president? george wash. ## kar 3 l student model design our student model kar 3 l builds on deep knowledge tracing @xcite and uses as inputs: 1) the flashcard f t shown to the student at time t; and 2) a history of all past flashcards studied by the student @xmath2, f 2 , ..., f t-1 }. we also assume a flashcard f can be mapped to study data x (f ), like the student's total correct responses to f and its time since last review. using these inputs, kar 3 l predicts correctness a @xmath3, 1} denoting whether the student will answer card f t correctly. kar 3 l predicts a t in two steps (figure 1 ). first, kar 3 l uses a bert retriever p(f t | f i ) to find the top-k flashcards f ′ ⊆ f most semantically relevant to f t . next, kar 3 l feeds f t and f ′ as inputs to a classifier p(a t | f, f ′ ), which represents the current flashcard f and each retrieved flashcard in f ′ with bert embeddings and the features from x (z). we describe both of these steps next. ## training kar 3 l we now train the kar 3 l student model. we describe our data collection platform ( §5.1), create flashcards ( §5.2), collect student study data ( §5.3), and outline our training procedure ( §5.4). ## offline evaluation we evaluate kar 3 l on our dataset to highlight the offline strength of content-aware student models. this result motivates our design of a content-aware scheduler ( §7), eventually allowing us to prove that such schedulers improve student learning ( §7.3). ## online evaluation our offline evaluation ( §6) shows kar 3 l predicts recall accurately, but this does not capture the main goal of any educational tool: enhancing learning. thus, while researchers often stop at offline evaluation to claim scheduler superiority, we now provide evidence that content-aware schedulers can improve student learning over fsrs, the sota scheduler. we propose a teaching policy that equips dkt student models like kar 3 l for scheduling ( §7.1), design a test mode user study in our app to measure two facets of medium-term learning ( §7.2), and assess student learning outcomes to compare the learning benefits of kar 3 l versus fsrs ( §7.3). ## conclusion we introduce and successfully implement the first content-aware scheduler with kar 3 l + ∆, a simple but effective model using dkt, bert, retrieval, and a novel delta-based teaching policy. our offline evaluation on a newly-collected dataset shows kar 3 l provides accurate and well-calibrated recall predictions, while our online evaluation reveals kar 3 l improves learning efficiency over sota. thus, we give the first evidence that content-aware schedulers can be used to improve student learning. content-aware scheduling enhances personalization in learning tools, as models can infer student knowledge gaps through semantic inferences. given this strength, we hope future works extend content to modalities beyond text, such as images or audio. in online evaluation, kar 3 l surpasses fsrs in learning efficiency, but there are still many facets of learning that can be measured with online studies; these learning metrics could not only be used for evaluation but also training signals. in all, we show the strength of content-aware scheduling, which we hope will motivate works to look beyond single study items and also model relations between study items to fully capture student abilities.