Not All Options Are Created Equal: Textual Option Weighting for Token-Efficient LLM -Based Knowledge Tracing (2025)

Abstract: Large Language Models (LLMs) have recently emerged as promising tools for knowledge tracing due to their strong reasoning and generalization abilities. While recent LLM-based KT methods have introduced new prompt formats, they struggle to reflect the histories of example learners within a single prompt during in-context learning (ICL), leading to limited scalability and high computational cost under token constraints. In this work, we present LLM-based Option weighted Knowledge Tracing (LOKT) , a simple yet effective LLM-based knowledge tracing framework that encodes the interaction histories of example learners in context as textual categorical option weights (TCOW) . These are semantic labels (e.g., “inadequate”) assigned to the options selected by learners when answering questions helping understand LLM. Experiments on multiple-choice datasets show that LOKT outperforms existing LLM-based KT models in both warm-start and few-shot settings. Moreover, LOKT enables scalable and cost-efficient inference, performing strongly even under strict token constraints. Our code is available at https://anonymous.4open.science/r/LOKT_model-3233

## introduction knowledge tracing (kt) is a core method in learning analytics that aims to estimate and track a learner's evolving understanding of specific knowledge components (kcs) over time. it models how a learner's knowledge state changes with each learning interaction, such as answering a question, and uses this to predict future performance @xcite . however, its effectiveness drops in cold-start scenarios where interaction data is limited @xcite . to tackle the bottleneck, there have been increasing efforts to leverage large language models (llms), which possess broad prior knowledge and strong reasoning capabilities @xcite . most existing approaches rely on either fine-tuning (ft) llms using learner interaction data @xcite or in-context learning (icl) @xcite . particularly, the icl-based approach has recently gained attentions owing to its flexibility and practicality by providing prompts without updating model parameters. as shown in figure 1-(a ), this method predicts how the test student will answer the next question by looking at a few example sequences from other students, which are provided in the prompt as references. while icl-based kt has shown promising performance in few-shot prompt settings, it raises a key question: is this approach truly effective for kt tasks, where each example consists of long interaction sequences? to investigate this, we examined the cost, scalability, and performance of icl-based kt across various few-shot configurations (figure 2 ). including more examples in the prompt increases token usage and inference cost, while reducing them due to token budget constraints degrades performance. these patterns, consistent with findings in other domains @xcite @xcite , highlight the structural limitations of icl for long-context kt scenarios. to address these limitations, we propose a novel method called lokt(llm-based option weighted knowledge tracing). as shown in figure 1 (b), lokt generates an encapsulated representation of learner behavior by using option weights, instead of inserting full interaction histories into prompts as in prior icl-based work. this approach allows lokt to maintain a fixed input length regardless of the number of learner interactions. option weights quantify learners' current knowledge states represented in the selected option in multiple-choice question (mcq) responses. these weights support systematic analysis of learner understanding or misconception based on option-level selections. however, their continuous values (ranging from -1 to 1 @xcite ) are difficult for llms to interpret @xcite . to address this, we propose textual categorical option weights (tcow) which transforms the continuous weights into textual categories such as inadequate, limited, and proficient (see figure 1 b ). this facilitates llms' comprehension of the information lying in selected options. we conduct experiments on five multiple-choice knowledge tracing datasets. to assess the effectiveness of our approach, we compare lokt with both llm-based baselines and conventional kt models under cold-start condition. our findings show that lokt improves llms' knowledge tracing perfor-mance with shorter prompts which save costs. our contributions are as follows: ## experiment this section presents the experimental setup and results that validate the effectiveness of the proposed methodology. we describe the datasets used, model implementation details, and baseline configurations. to systematically evaluate our method, we design experiments to answer the following three research questions: rq1. does option weights improve llms' knowledge tracing performance in cold-start settings? we evaluate whether lokt outperforms both traditional kt and llm-based kt methods when cold-start settings. additionally, we conduct experiments under warm-start settings to verify the robustness of lokt. rq2. what are the scalability and efficiency benefits of using option-weight encapsulated prompts in llm-based knowledge tracing? we conduct experiments comparing inference costs across various few-shot settings and evaluate performance under fixed token budgets to assess the impact of encapsulating option weights. ## conclusion in this paper, we analyze the fundamental limitations of existing methods in llm-based knowledge tracing models developed to tackle the cold-start problem, focusing on inefficiency and scalability issues caused by long interaction histories. to address these challenges, we propose lokt, a simple yet effective framework that compresses learner interactions into textual categorical option weights within prompts, enabling large language models to efficiently understand learners' problem-solving histories. extensive experiments on multiple public datasets demonstrate that lokt surpasses conventional kt models and llm-based in-context learning methods in accuracy, scalability, and cost efficiency. our study highlights that leveraging effective compression techniques in in-context learning is key to scalable and practical knowledge tracing with llms.