From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: This work introduces a simple and strong knowledge tracking baseline. As reviewers pointed out, this approach is simple, computationally friendly, and performs better than many existing approaches on a wide range of tasks across different domains.  The paper itself is well-written. The concern was around whether the technical contribution is novel enough. After the discussion among reviewers and reviewing authors’ responses, I’m leaning towards acceptance as the simplicity of a method (e.g., utilizing transformers for this task) is not a weakness and strong baselines help us understand the task itself in many ways. 


justification_for_why_not_higher_score: This work would benefit a lot from poster discussion. 
justification_for_why_not_lower_score: N/A
summary_of_AC-reviewer_meeting: N/A
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper3300/Reviewer_EgtS
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper builds a simple yet strong baseline, named SimpleKT, for knowledge tracing (KT). The newly introduced baseline is motivated by the Rasch model in psychometrics to capture the individual differences in questions of the same KC. It only contains a few simple operations, i.e., element-wise product, addition, and the ordinary dot-product attention, which is simple to implement. Surprisingly, the proposed simple method can beat most of the 12 representative baselines on 7 public datasets.
strength_and_weaknesses: Strengths:
A simple-to-implement and strong baseline is constructed for the knowledge tracing (TK) task, which can facilitate future KT research.

Weaknesses:
I am not familiar with the knowledge tracing task. Thus, I cannot provide useful comments and suggestions to improve this paper.

clarity,_quality,_novelty_and_reproducibility: This paper is easy to understand even for a reader without much background on the knowledge tracing task.

summary_of_the_review: A simple-to-implement and well-motivated baseline model is presented for the Knowledge tracing task, which wins most of 12 existing methods on 7 public datasets. The newly constructed baseline can facilitate future research on KT.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3300/Reviewer_nkiF
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This work aims to address inconclusive findings in the literature about which models perform best for Knowledge Tracing (KT). In this paper, the authors present a baseline which convincingly outperforms most published work despite being simpler in nature. 
strength_and_weaknesses: STRENGTHS
- This work gives a very comprehensive review of dozens of work in the KT space and how they work.
- By comparing against 12 previous models, this weighs in on disagreements in the literature about model comparisons.
- Demonstrating that such a simple model can outperform many deep learning-based approaches provides meaningful insight into the nature of the task, which can inspire future work and model designs.

WEAKNESSES
- I could not tell whether the 12 comparison models had publicly available code or were reimplemented by the authors (or something else). Clarification would be helpful.

clarity,_quality,_novelty_and_reproducibility: Clarity: The clarity for this work is fair. It could improve with some rewrites with an eye towards the reader's experience:
    - Define "KC" in the abstract before referencing it. It is currently not defined until the Introduction.
    - The first paragraph of page 2 has a heavy mix of citations and prose. Whereas there is not much prose in the citation-heavy of paragraph of page 1 (hence being able to get away with it the first time), it is much harder to read the page 2 paragraph. It could be helpful to restate/recap the important claims to ensure they do not get get missed by the reader.
    - Although Figure 1 is very helpful for generally understanding the KT task, it is hard to follow the distinction between D1 (datasets containing info about both q's and KCs) and D2 (datasets containing info of *either* q's or KCs). One example of an instance from each type would be helpful to see.

Quality: This work is high-quality. By comprehensively surveying the literature, the authors identify a gap in approaches, which they fill using a simple model. This simple model employs tactics which are well-suited for the task (eg attention, question-difficulty info, etc) to demonstrate that many previous approaches were not as effective as once understood, despite their bells and whistles. This is a very useful finding, and it is done by measuring across 7 datasets.

Novelty: The novelty of this approach is, on the one hand, low, because it uses a simple approach. However, it demonstrates why a "simple" model is nonetheless able to perform better than nearly a dozen "more novel" models.

Reproducibility: This work is highly-reproducible. It provides a link to an anonymized repo with code and one of the datasets.
summary_of_the_review: In this paper, the authors present a baseline which convincingly outperforms most published work despite being simpler in nature. The baseline compares against 12 other models -- such a comprehensive comparison is itself valuable to the field. Additionally, the works conducts an ablation study, which provides important insight into understanding what makes the simple model nonetheless effective (e.g. how much is from the question-difficulty modeling?).
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper3300/Reviewer_vuA2
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes SIMPLEKT, a simple but tough-to-beat KT baseline that is simple to implement, computationally friendly and robust to a wide range of KT datasets across different domains. 
strength_and_weaknesses: Strength

1. The paper is well written and well presented.

2. The source code is publicly avaiblale.

3. The experimental results are good, comparing to existing baselines.


Weakness
1. The technical contributions of this work are very limited, simply adopting the tranformer model for their tasks. Overall contributions are below the bar of ICLR in my mind.
clarity,_quality,_novelty_and_reproducibility: Clarity: the motivations and ideas are clear in this work.

Quality: The techniques of this work is limited, directly using Transformer to solve their tasks.

Novelty: The novelty of this work is not high,

Reproducibility: The reproducibility of this work is good, since it is simple and source code is available at this time.
summary_of_the_review: In summary, the topic in this work is interesting, however the overall contributions of this work are not very enough for ICLR.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough