Paper: Revisiting Knowledge Tracing: A Simple and Powerful Model (ICLR-2024)

Reviewer comment (Writing):
Can you further explain the design and workflow of the FRU framework? For example, what is the connection between FRU and human cognitive development models, what are inputs, what are learnable parameters and what are outputs? Besides, what does $I_\alpha$ mean in Section 3.4? I cannot find it on Figure 3.

Author response:
**A3**: Thank you for your more in-depth questions. We have provided a detailed response to the question about FRU in the "General Response about FRU." It includes the motivation behind designing FRU, inspiration for its design, the process of FRU, and the differences and advantages of FRU compared to MLP, RNN, LSTM, and GRU.

Furthermore, for FRU, its input consists of the student's historical interaction sequence (corresponding to $X_t$ in Figure 3). The learning parameters of FRU are the parameters within the Update and Forget components in Figure 3 (parameters $W_2$ and $b_2$ in Update, and parameters $W_1$ and $b_1$ in Forget). Its output is $Response_t$.

Thank you very much for your thorough review. $I_\alpha$ represents the embedded representation of time interval $\alpha$ (For example, the time interval from the last time $\alpha=2$, assuming there is a time embedding matrix $I \in \mathbb{R}^{T \times d}$, $T$ is the total time step, $d$ is the embedding dimension, then $I_\alpha=I_2$ (that is, the $\alpha$-th row of $I$)).

We have revised this detail in figure 3.($\alpha -> I_\alpha$)

Reviewer comment (Novelty):
In terms of sequence modeling, RNN and LSTM are also simple but effective. What are the advantages of FRU compared to them, especially in the context of knowledge tracing? Does your proposed FRU have the potential to be applied to other areas except for knowledge tracing (e.g., sequential recommendation)?

Author response:
**A4**: Thank you for your insightful inquiry. We have provided a detailed response to the question about FRU in the "General Response about FRU." It includes the motivation behind designing FRU, inspiration for its design, the process of FRU, and the differences and advantages of FRU compared to MLP, RNN, LSTM, and GRU.

Currently, we have not yet explored the application of FRU in other domains. However, given that FRU is inspired by models of human cognitive development, we believe there is theoretical potential for its application in domains related to modeling implicit states in humans, such as interests, health status, and more. The field you mentioned, sequential recommendation, is one that we consider to be a promising direction for potential application. Once again, we appreciate your insightful questions.

Reviewer comment (Writing):
There are some syntax and spelling errors need to be solved, such as the index “I” in the formula of loss function, $Loss_{KT}$. I guess it should be replaced with $t$.

Author response:
**A5**: Thank you very much for your careful review. We have corrected this error in the paper. 

Thanks again for reviewing and your valuable suggestions!

Reviewer comment (Novelty):
The paper employs similar approaches to previous DKT methods, such as RAKT [1], AKT [2], and [3] except for the FRU unit. All of three papers also implemented the FRU unit with exponential time decay as part of the attention mechanism in the transformer architecture. The authors have used MLP units as FRU and concatenated the hidden state to the final representations.

Author response:
**A1**: Because the goal of this paper is to make the model both simple and effective. Among the KT models based on deep learning, the DKT model as the founder can be said to be very simple. The similarity between ReKT (the model proposed in this paper) and DKT demonstrates the simplicity of ReKT. Additionally, ReKT's three different perspectives (question, concept, domain) in modeling ensure its performance advantage. 

As you mentioned, the exponential decay method used in considering forgetting in the three models is indeed very effective, but this computational forgetting method is more complex [1]. This paper wants to simplify the operation as much as possible, so FRU's forgetting is not designed to have an exponential time decay effect. Of course, we believe that considering exponential decay would definitely make FRU more effective. The fact that FRU consists of two linear regression units and connects the hidden state to the final representation is to make the model as simple as possible.

We also have provided a detailed response to the question about FRU in the "General Response about FRU". It includes the motivation behind designing FRU, inspiration for its design, the process of FRU, and the differences and advantages of FRU compared to MLP, RNN, LSTM, and GRU.

Reviewer comment (Reproducibility):
The Rasch difficulty is determined from students' question and response binary matrix.

Author response:
**A3**: Thank you for your in-depth question. Since your two questions are similar, I will answer them both together.

We did not calculate Rasch difficulty from three different interaction matrices. In our design, Rasch difficulty is specific to each question (i.e. each question has its own difficulty). Therefore, we fixed the representation of the question interactions, but the range seen in different perspectives is different, this allows the question interactions have different representations.

The Rasch difficulty you pointed out should be for all questions. Then the difficulty of Rasch is different from different perspectives. This is a very interesting idea, and we have not yet considered this situation.

But I have some ideas for your reference. Because the sequences obtained from different perspectives are different, their corresponding Rasch difficulty is also different. Then we can calculate the Rasch difficulty of the corresponding sequence from the question、concept、domain perspective respectively.

If the inputs from the three perspectives are made consistent, then the interaction representation method needs to be modified (for example, the interaction representation can be changed to question + concept + question sequence Rasch difficulty + concept sequence Rasch difficulty + domain sequence Rasch difficulty).

If the inputs from the three perspectives are inconsistent, then the input from the question perspective can be (question + question sequence Rasch difficulty), the input from the concept perspective can be (concept + concept sequence Rasch difficulty), and the input from the domain perspective can be (question + concept + domain sequence Rasch Difficulty).

We hope our response has addressed your concerns. If you have any further questions, please feel free to let us know.

Thanks again for reviewing and your valuable suggestions!

Reviewer comment (Novelty):
I am not convinced that the FRU gate is "simple". It appears to me as a variant of the gated recurrent unit (GRU) without the reset gate. Compared to the GRU, FRU has a similar forget gate and the hyperbolic tangent function in the end (without the affine combination of the update gate). I think the FRU architecture design, though interesting, does not qualify it as "very lightweight, as it consists of only two linear regression units". Otherwise, I can make the same "very lightweight" statement for GRU, which only consists of three linear regression units. Why not just use GRU? GRU takes into account not just forgetting (as in FRU), but also remembering/resetting, which might make more sense and have more modeling power? What exactly in the reference article "Toward a theory of instruction" do the authors get the inspiration to build FRU? This is an important question that the author should answer because they claim FRU as one of their core contributions, whereas I think FRU is not much different from GRU, which diminishes the value of this contribution. The authors should also cite GRU as important alternative modeling choices to compare to FRU (in addition to LSTM).

Author response:
**A1**: FRU is lightweight compared to most methods in the KT field (i.e. the lightweight nature of FRU is relative). FRU's design is also intended as a core framework for KT tasks. As mentioned in this paper, the current mainstream in KT research is building more complex models (such as stacked Transformers) to improve performance. In terms of being the core architecture for KT tasks, both FRU, GRU, and LSTM can be considered lightweight.

**A2**: Thank you for your in-depth questions about FRU. The purpose of designing FRU is to create a simpler (compared to LSTM, GRU, Transformer) architecture and suitable for KT, so we have not considered using GRU. 

Through experimentation, we found that we can achieve comparable or even better performance without the need for more complicated operations like memory/reset. Therefore, we simplified such operations.

We have provided a detailed response to the question about FRU in the "General Response about FRU". It includes the motivation behind designing FRU, inspiration for its design, the process of FRU, and the differences and advantages of FRU compared to MLP, RNN, LSTM, and GRU.

**A3**: Thank you very much for your suggestion. We have added a comparison of GRU as the core architecture of KT in the paper (refer to Table 6, 7, 8, 9).

Reviewer comment (Novelty):
The proposed approach to represent knowledge at question, concept, and history level is not entirely new; methods such as learning factor analysis (https://link.springer.com/chapter/10.1007/11774303_17), performance factor analysis (https://files.eric.ed.gov/fulltext/ED506305.pdf), additive factor models (http://www.cs.cmu.edu/~ggordon/chi-etal-ifa.pdf), knowledge factoring machines (https://arxiv.org/pdf/1811.03388.pdf) also take into account of modeling students' knowledge at concept (sometimes called skills in these literature), question, or entire history levels. In the spirit of "revisiting", the authors neither mentioned nor compared to these classic knowledge tracing methods.

Author response:
**A4**: Firstly, it needs to be clarified that classic knowledge tracing models based on machine learning, such as BKT, LFA, PFA, AFM, etc., only model student knowledge at the concept level. Knowledge tracing models based on machine learning, including KTM, only model student knowledge at the question level. They all struggle to utilize the entire interaction history of students (i.e., domain level). In contrast, mainstream approaches of deep learning-based knowledge tracing models utilize the entire interaction history of students (i.e., only domain level) to construct student knowledge (as more data helps improve the performance of deep learning models). However, very few methods exist that use deep learning to construct student knowledge from concept or question levels, which is evident as these perspectives imply a reduction in data volume (which is not beneficial to the performance of deep learning models). To the best of our knowledge, there is currently no knowledge tracing method that combines the three different perspectives of question, concept, and domain to construct student knowledge. Therefore, the proposed method in this paper is novel.

Reviewer comment (Evaluation):
Some of the results need more clarifications. For example, AKT-R (https://arxiv.org/pdf/2007.12324.pdf) can achieve an AUC of __0.8346__ on ASSIST09 (see Table 5 in the AKT paper), beating the AUC of 0.7917 by the proposed method. Several other baselines in the AKT paper also achieve AUC > 0.8.

Author response:
**A6**: This is mainly due to the way the data is processed. Suppose there is a student answer record $(q, (c_1, c_2, c_3))$: corresponding to a question $q$ and three concepts $(c_1, c_2, c_3)$ associated with it. According to [1], the processing method of this paper is to merge these three concepts $(c_1, c_2, c_3)$ into a single concept $c$, then in question-based KT, this record becomes $(q, c)$. What AKT does is to separate these three concepts $(c_1, c_2, c_3)$ into $(c_1)$, $(c_2)$, $(c_3)$. Then in question-based KT, this record becomes $(q, c_1), (q, c_2), (q, c_3)$ three records. In a real learning environment, the concept related to a specific question should remain unchanged. The processed approach in AKT cannot guarantee this, but the processed approach used in this paper can.

Furthermore, the AKT performance (and other baselines) reported in recent papers at top-level conferences (such as simpleKT[2], LBKT[3], SFKT[4]) all indicate that the performance of AKT is not as high as reported in their paper.

Thanks again for reviewing and your valuable suggestions!

[1] Xiaolu Xiong, Siyuan Zhao, Eric G Van Inwegen, and Joseph E Beck. Going deeper with deep knowledge tracing. In Proceedings of the 9th International Conference on Educational Data Mining, pp. 545–550. International Educational Data Mining Society, 2016.

[2] Zitao Liu, Qiongqiong Liu, Jiahao Chen, Shuyan Huang, and Weiqi Luo. simplekt: a simple but tough-to-beat baseline for knowledge tracing. In International Conference on Learning Representations, 2023b.

[3] Bihan Xu, Zhenya Huang, Jiayu Liu, Shuanghong Shen, Qi Liu, Enhong Chen, Jinze Wu, and Shijin Wang. Learning behavior-oriented knowledge tracing. In Proceedings of the 29th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2789–2800. Association for Computing Machinery, 2023.

[4] Zhang M, Zhu X, Zhang C, et al. No Length Left Behind: Enhancing Knowledge Tracing for Modeling Sequences of Excessive or Insufficient Lengths. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 2023.

Reviewer comment (Novelty):
How is FRU different from GRU? What motivate the differences?

Author response:
**A2**: Thank you for your in-depth questions about FRU. The purpose of designing FRU is to create a simpler (compared to LSTM, GRU, Transformer) architecture and suitable for KT, so we have not considered using GRU. 

Through experimentation, we found that we can achieve comparable or even better performance without the need for more complicated operations like memory/reset. Therefore, we simplified such operations.

We have provided a detailed response to the question about FRU in the "General Response about FRU". It includes the motivation behind designing FRU, inspiration for its design, the process of FRU, and the differences and advantages of FRU compared to MLP, RNN, LSTM, and GRU.

Reviewer comment (Novelty):
How is the proposed method contexualized within, and compared to, some classic literature such as LFA, PFA, AFM, KFM, and others?

Author response:
**A5**: Thank you very much for your reminder. We have added an review to the classic knowledge tracing method (BKT、LFA、PFA、AFM、KTM) in RELATED WORK. In addition, we added performance comparisons with classical knowledge tracing methods BKT and KTM (refer to Table 2, 3).

Reviewer comment (Evaluation):
Why are some results different (sometimes by a large margin) to existing published results?

Author response:
**A6**: This is mainly due to the way the data is processed. Suppose there is a student answer record $(q, (c_1, c_2, c_3))$: corresponding to a question $q$ and three concepts $(c_1, c_2, c_3)$ associated with it. According to [1], the processing method of this paper is to merge these three concepts $(c_1, c_2, c_3)$ into a single concept $c$, then in question-based KT, this record becomes $(q, c)$. What AKT does is to separate these three concepts $(c_1, c_2, c_3)$ into $(c_1)$, $(c_2)$, $(c_3)$. Then in question-based KT, this record becomes $(q, c_1), (q, c_2), (q, c_3)$ three records. In a real learning environment, the concept related to a specific question should remain unchanged. The processed approach in AKT cannot guarantee this, but the processed approach used in this paper can.

Furthermore, the AKT performance (and other baselines) reported in recent papers at top-level conferences (such as simpleKT[2], LBKT[3], SFKT[4]) all indicate that the performance of AKT is not as high as reported in their paper.

Thanks again for reviewing and your valuable suggestions!

[1] Xiaolu Xiong, Siyuan Zhao, Eric G Van Inwegen, and Joseph E Beck. Going deeper with deep knowledge tracing. In Proceedings of the 9th International Conference on Educational Data Mining, pp. 545–550. International Educational Data Mining Society, 2016.

[2] Zitao Liu, Qiongqiong Liu, Jiahao Chen, Shuyan Huang, and Weiqi Luo. simplekt: a simple but tough-to-beat baseline for knowledge tracing. In International Conference on Learning Representations, 2023b.

[3] Bihan Xu, Zhenya Huang, Jiayu Liu, Shuanghong Shen, Qi Liu, Enhong Chen, Jinze Wu, and Shijin Wang. Learning behavior-oriented knowledge tracing. In Proceedings of the 29th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2789–2800. Association for Computing Machinery, 2023.

[4] Zhang M, Zhu X, Zhang C, et al. No Length Left Behind: Enhancing Knowledge Tracing for Modeling Sequences of Excessive or Insufficient Lengths. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 2023.