Paper: Predictive, scalable and interpretable knowledge tracing on structured domains (ICLR-2024)

Reviewer comment (Experiments):
The experimental setup lacks persuasiveness. As demonstrated in Table 1, two datasets contain over 10,000 learners, yet the authors chose to use only 100-1,000 learners as training data. Conducting experiments with a small dataset may unfairly disadvantage deep learning baselines, which can effectively leverage the abundance of available data. The reasoning provided, "to simulate real-world data constraints in education," may not hold in the context of the vast amount of student learning data generated today.

Author response:
> Why did the authors choose to experiment with only a limited portion of the datasets? The explanation provided, "to simulate real-world data constraints in education," may benefit from additional clarification.

We thank the reviewer for spotting that “real-world data constraints” is imprecise and leads to confusion. We now (a) clarify our focus on small-to-moderate cohort sizes and (b) extend performance evaluations to all learners. In detail:

1. We reformulated the start of Section 4.1:  
“In our evaluations, we mainly focus on prediction and generalization when training on 10 interactions from up to 1000 learners. Good KT performance with little data is key in practical ITS to minimize the number of learners on an experimental treatment (principle of equipoise, similar to medical research [1]), to mitigate the cold-start problem, and extend the usefulness of the model to classroom-size groups. To provide ITS with a basis for adaptive guidance and long-term learner assessment, we always predict the 10 next interactions.”
2. We have run additional experiments and now report performance for the maximum number of learners available in each dataset after preprocessing (Figure 2, Table 11).  We discuss them in Section 4.1:    
“Figure 2 shows that PSI-KT's *within-learner prediction* performance is robustly above baselines for all but the largest cohort size (>60k learners, Junyi15), where all deep learning models perform similarly.   
The *between-learner generalization* accuracy of these models when tested on 100 out-of-sample learners is shown in Table 2, where fine tuning indicates that parameters were updated using (10-point) learning histories from the unseen learners. PSI-KTshows overall superior generalization except on Junyi15 (when fine-tuning).”

We are grateful for the opportunity to clarify our focus on the < 1000 learners setting, and for encouraging us to look beyond it into large cohorts. The differences in behavior between models warrant further research, which we have reflected in the discussion:  
“We further find that PSI-KT has remarkable predictive performance when trained on small cohorts whereas baselines require training data from at least 60k learners to reach similar performance. An open question for future KT research is how to combine PSI-KT's unique continual learning and interpretability properties with performance that grows beyond this extreme regime.”

Reviewer comment (Experiments):
The introduction of interpretable KT methods is not comprehensive. For instance, recent approaches like IKT, ICKT, and QIKT [1, 2, 3] incorporate interpretable psychological and cognitive modules into their methods. These relevant methods are not referenced in this paper, let alone included as baselines in the experiments.

Author response:
> The introduction of interpretable KT methods is not comprehensive. For instance, recent approaches like IKT, ICKT, and QIKT incorporate interpretable psychological and cognitive modules into their methods. 
> Could the authors consider using more recent interpretable deep learning methods like QIKT as their baseline comparisons? Doing so could enhance the credibility of the study.

We thank the reviewer for the valuable feedback and pointing us towards additional baselines. We found that QIKT is an excellent baseline to add (together with GKT [5], proposed by reviewer *J4qa*) and are pleased that our more comprehensive assessment (Figure 2-3, 12 and Table 2, 9-13, 15) provides even stronger support for our contributions.

We have improved the related work (Sec. 2), now covering interpretable KT approaches IKT [2], IEKT [3] and QIKT [4]. 

Adding even more baselines might still be valuable, but we were unable to include more because (1) the official IKT code lacks a key component (tree structure) making a fair comparison challenging; (2) while we were unfortunately unable to find a reference for ICKT, we found the similarly named IEKT and now reference it in the related work. We did not add it to evaluations as we believe QIKT provides a superior baseline. Should ICKT differ from IEKT, we welcome a reference for potential future inclusion.

Reviewer comment (Evaluation):
Perhaps conducting case studies of PSI-KT could offer a more intuitive understanding of its interpretability, such as visualizing trends in students' knowledge mastery, as shown in Figure 1(a).

Author response:
> Perhaps conducting case studies of PSI-KT could offer a more intuitive understanding of its interpretability, such as visualizing trends in students' knowledge mastery, as shown in Figure 1(a).

Thank you for this suggestion. We have added a visual example (Fig. 11 in A.6.5) of how latent knowledge states evolve in response to learning interactions, similar to our schematic Figure 1. We note that our model can estimate knowledge states at all times and not just interaction times, which allows us to use natural time in the x-axis and display knowledge states with curves instead of using the discrete color maps common in the KT literature.

Reviewer comment (Experiments):
Why did the authors choose to experiment with only a limited portion of the datasets? The explanation provided, "to simulate real-world data constraints in education," may benefit from additional clarification.

Author response:
> Why did the authors choose to experiment with only a limited portion of the datasets? The explanation provided, "to simulate real-world data constraints in education," may benefit from additional clarification.

We thank the reviewer for spotting that “real-world data constraints” is imprecise and leads to confusion. We now (a) clarify our focus on small-to-moderate cohort sizes and (b) extend performance evaluations to all learners. In detail:

1. We reformulated the start of Section 4.1:  
“In our evaluations, we mainly focus on prediction and generalization when training on 10 interactions from up to 1000 learners. Good KT performance with little data is key in practical ITS to minimize the number of learners on an experimental treatment (principle of equipoise, similar to medical research [1]), to mitigate the cold-start problem, and extend the usefulness of the model to classroom-size groups. To provide ITS with a basis for adaptive guidance and long-term learner assessment, we always predict the 10 next interactions.”
2. We have run additional experiments and now report performance for the maximum number of learners available in each dataset after preprocessing (Figure 2, Table 11).  We discuss them in Section 4.1:    
“Figure 2 shows that PSI-KT's *within-learner prediction* performance is robustly above baselines for all but the largest cohort size (>60k learners, Junyi15), where all deep learning models perform similarly.   
The *between-learner generalization* accuracy of these models when tested on 100 out-of-sample learners is shown in Table 2, where fine tuning indicates that parameters were updated using (10-point) learning histories from the unseen learners. PSI-KTshows overall superior generalization except on Junyi15 (when fine-tuning).”

We are grateful for the opportunity to clarify our focus on the < 1000 learners setting, and for encouraging us to look beyond it into large cohorts. The differences in behavior between models warrant further research, which we have reflected in the discussion:  
“We further find that PSI-KT has remarkable predictive performance when trained on small cohorts whereas baselines require training data from at least 60k learners to reach similar performance. An open question for future KT research is how to combine PSI-KT's unique continual learning and interpretability properties with performance that grows beyond this extreme regime.”

Reviewer comment (Experiments):
Could the authors consider using more recent interpretable deep learning methods like QIKT as their baseline comparisons? Doing so could enhance the credibility of the study.

Author response:
> The introduction of interpretable KT methods is not comprehensive. For instance, recent approaches like IKT, ICKT, and QIKT incorporate interpretable psychological and cognitive modules into their methods. 
> Could the authors consider using more recent interpretable deep learning methods like QIKT as their baseline comparisons? Doing so could enhance the credibility of the study.

We thank the reviewer for the valuable feedback and pointing us towards additional baselines. We found that QIKT is an excellent baseline to add (together with GKT [5], proposed by reviewer *J4qa*) and are pleased that our more comprehensive assessment (Figure 2-3, 12 and Table 2, 9-13, 15) provides even stronger support for our contributions.

We have improved the related work (Sec. 2), now covering interpretable KT approaches IKT [2], IEKT [3] and QIKT [4]. 

Adding even more baselines might still be valuable, but we were unable to include more because (1) the official IKT code lacks a key component (tree structure) making a fair comparison challenging; (2) while we were unfortunately unable to find a reference for ICKT, we found the similarly named IEKT and now reference it in the related work. We did not add it to evaluations as we believe QIKT provides a superior baseline. Should ICKT differ from IEKT, we welcome a reference for potential future inclusion.

Reviewer comment (Experiments):
Is there a specific reason why the authors did not provide case studies to visually demonstrate the model's interpretability, as has been done in previous KT research?

Author response:
> Is there a specific reason why the authors did not provide case studies to visually demonstrate the model's interpretability, as has been done in previous KT research?

Thank you for this suggestion. We have added a visual example (Fig. 11 in A.6.5) of how latent knowledge states evolve in response to learning interactions, similar to our schematic Figure 1. We note that our model can estimate knowledge states at all times and not just interaction times, which allows us to use natural time in the x-axis and display knowledge states with curves instead of using the discrete color maps common in the KT literature.

Reviewer comment (Evaluation):
Could the authors elaborate on the detailed rationale behind using mutual information between PSI-KT's learned parameters and the hidden vectors of baselines to measure interpretability? Further explanation would enhance the understanding of the experiments.

Author response:
> - Could the authors elaborate on the detailed rationale behind using mutual information between PSI-KT's learned parameters and the hidden vectors of baselines to measure interpretability? Further explanation would enhance the understanding of the experiments.

Thank you for bringing up these two points, which are indeed related: the rationale for expressing specificity, consistency, and disentanglement in terms of mutual information (Sec. 4.3.1) is precisely to avoid applying our preconceived notions of interpretable cognitive traits to the evaluation of baseline models. As an information-theoretical quantity, mutual information only quantifies the *amount* of information that, e.g., a learner representation contains about the learner’s identity (for the specificity metric). It is agnostic to the form in which this information is encoded in the learner representation. If a baseline model finds different underlying learner features that do not match our psychologically-motivated cognitive traits then the model can still have a high specificity score as long as the identified learner features contain information to identify the learner in some way. The same holds for consistency and disentanglement. We will add a shortened form of this clarification to the paper.

Reviewer comment (Experiments):
Experiments are somewhat insufficient. Although the authors have conducted an extensive array of analytical and validation experiments, there is a notable absence of ablation study to demonstrate the effectiveness of the two proposed motivations in the paper, namely cognitive traits and the prerequisite relationship graph, on PSI-KT. Furthermore, given the mention of the use of the prerequisite graph in the paper, it seems somewhat inadequate not to include some explicit baseline models that utilize knowledge concept graphs for comparison.

Author response:
We are grateful for the suggestion of an ablation analysis (also mentioned by reviewer *rAUs*) and the recommendation of additional baselines. The suggestion of performing ablation analyses was incredibly helpful, and allowed us to discover that all components of the model are crucial for performance. Additionally, the addition of a strong structure-aware baseline (GKT) has strengthened the credibility and thoroughness of support for our model.

In a new ablation analysis (Table 16 and Figure 13, Appendix A.8), we ablate the graph structure, the individual traits, and learner dynamics, observing significant drops in accuracy in each case. No single ablation reliably corresponds to the largest drop in accuracy across datasets. Thus, we believe that all components make important contributions to the robustness and generalizability of our model.

We have also added GKT to our set of baselines (along with an additional model QIKT [5] mentioned by other reviewer *k3Pn*). GKT represents one of the strongest baselines and expands our comprehensive assessment of alternative models (Figure 2-3, 12 and Table 2, 9-13, 15), which altogether provides even stronger support for our model.

While SKT has a number of innovative methods for modeling both similarity and prerequisite structures, we could not find how to perform data splitting to calculate the graph (either in the code base or paper). Specifically, the graph is calculated from performance statistics (i.e., correct/incorrect counts), which cannot be directly used to predict the same performance data (to avoid circularity). This same circularity is not an issue when the graph inference is part of an end-to-end learning pipeline, as in ours and other models. For this reason a fair comparison seems challenging.

Reviewer comment (Experiments):
Have the authors considered supplementing with essential ablation study and adding baseline models that explicitly take into account the knowledge graph structure, such as GKT[1] or SKT[2], the latter of which also considers prerequisite relationship between concepts?

Author response:
We are grateful for the suggestion of an ablation analysis (also mentioned by reviewer *rAUs*) and the recommendation of additional baselines. The suggestion of performing ablation analyses was incredibly helpful, and allowed us to discover that all components of the model are crucial for performance. Additionally, the addition of a strong structure-aware baseline (GKT) has strengthened the credibility and thoroughness of support for our model.

In a new ablation analysis (Table 16 and Figure 13, Appendix A.8), we ablate the graph structure, the individual traits, and learner dynamics, observing significant drops in accuracy in each case. No single ablation reliably corresponds to the largest drop in accuracy across datasets. Thus, we believe that all components make important contributions to the robustness and generalizability of our model.

We have also added GKT to our set of baselines (along with an additional model QIKT [5] mentioned by other reviewer *k3Pn*). GKT represents one of the strongest baselines and expands our comprehensive assessment of alternative models (Figure 2-3, 12 and Table 2, 9-13, 15), which altogether provides even stronger support for our model.

While SKT has a number of innovative methods for modeling both similarity and prerequisite structures, we could not find how to perform data splitting to calculate the graph (either in the code base or paper). Specifically, the graph is calculated from performance statistics (i.e., correct/incorrect counts), which cannot be directly used to predict the same performance data (to avoid circularity). This same circularity is not an issue when the graph inference is part of an end-to-end learning pipeline, as in ours and other models. For this reason a fair comparison seems challenging.

Reviewer comment (Experiments):
The evaluations focus on three specific educational datasets. Testing on a more diverse range of datasets could better reveal the model's capabilities and limitations. The authors could discuss what other domains or data characteristics pose challenges.

Author response:
> Could you provide insights into the dataset limitations and discuss potential challenges in applying the model to other educational domains or datasets? 

We thank the reviewer for the thorough feedback on our paper. We agree that the representativeness of empirical evaluations should be a central concern for ours and other models, and now highlight it in the discussion:  
​​“Although we designed PSI-KT with general structured domains in mind, our empirical evaluations were limited to mathematics learning by dataset availability. We highlight the need for more diverse datasets for structured KT research to strengthen representativeness in ecologically valid contexts.”

We expand on limitations in Appendix A.3.2 where we now describe in detail the datasets and the criteria we used to select them (also in response to reviewer *da66*). In brief, our choice of datasets was constrained by the following requirements: we require data sets in structured domains with (1) KC labels and (2) high temporal resolution. These conditions rule out Statics2011 (1), Assist09 and Assist15 (2), and Junyi20 (2).

Reviewer comment (Experiments):
While superior overall, some accuracy metrics are comparable to certain baselines. Further ablation studies could provide insight into which model components contribute most to accuracy gains.

Author response:
> Could you perform ablation studies to dissect the contributions of different model components to predictive accuracy, providing insights into the model's strengths?

We are very grateful to the reviewer for the suggestion of performing ablation studies, which was suggested by the reviewer *J4qa* as well. When we ablate any of the graph structure, the individual traits, and learner dynamics, we observe significant drops in accuracy across all three datasets. No single ablation reliably corresponds to the largest drop in accuracy across datasets. Thus, we believe that all components together allow our model to adapt to a variety of educational contexts and datasets, thereby enhancing its robustness and generalizability.

We present the results and their interpretation in Appendix A.8 (Table 16 and Figure 13) and summarize them in Section 4.1:  
"The advantage of PSI-KT comes from its combined modeling of KC prerequisite relations and individual learner traits that evolve in time."