Paper:
Automatic Discovery of Cognitive Skills to Improve the Prediction of Student Learning
To master a discipline such as algebra or physics, students must acquire a set of cognitive skills. Traditionally, educators and domain experts use intuition to determine what these skills are and then select practice exercises to hone a particular skill. We propose a technique that uses student performance data to automatically discover the skills needed in a discipline. The technique assigns a latent skill to each exercise such that a student's expected accuracy on a sequence of same-skill exercises improves monotonically with practice. Rather than discarding the skills identified by experts, our technique incorporates a nonparametric prior over the exerciseskill assignments that is based on the expert-provided skills and a weighted Chinese restaurant process. We test our technique on datasets from five different intelligent tutoring systems designed for students ranging in age from middle school through college. We obtain two surprising results. First, in three of the five datasets, the skills inferred by our technique support significantly improved predictions of student performance over the expertprovided skills. Second, the expert-provided skills have little value: our technique predicts student performance nearly as well when it ignores the domain expertise as when it attempts to leverage it. We discuss explanations for these surprising results and also the relationship of our skilldiscovery technique to alternative approaches.

Review:
This paper proposes a probabilistic approach for learning the assignment of exercises to skills from student data, where student knowledge changes while exercises are being solved; the model also estimates the student knowledge while estimating the skill assignments. The paper uses a weighted CRP to model the assignment, incorporating expert labelings through the weighting. In simulation, the method recovers skill labelings with high accuracy, with little dependence on the expert labels, and across several datasets, the paper finds that skill labelings from this method result in higher prediction accuracy than other approaches. Overall, I found the paper to be clear and the proposed model is a relatively novel extension of existing methods. The evaluation methods demonstrate that the approach is likely to be effective for relevant applications, and that its inferences are accurate. I appreciated the inclusion of both synthetic data and a variety of real datasets. One interesting issue I found in the evaluation is that while the single skill assumption seems partially justified by the fact that these exercises were created to tap a single skill, the expert skills are in many cases a poor fit for prediction, suggesting that the experts may not be clear about the connection between each item and skill. Some discussion of the poor fit of the expert model is included in the discussion, but I think this remains an important point to consider, both in the motivation of the model and in trying to determine what effect using these new skills might have for control policies for tutors. One point that is not touched on is whether ordering/selection effects in terms of what items each student performs and in what order may skew the results; it would be great if a small amount of space in the paper could be devoted to addressing this issue, given that in typical datasets from tutoring systems one cannot count on items being selected randomly or administered in random order. The main concern I had with the paper was the presentation of alternative methods, specifically in the paragraphs around lines 57-94 of the intro. The paper first claims that automated techniques either "rely on an expert labeling which they then refine or treat the student knowledge as static." However, a later paragraph notes the Gonzalez-Brenes and Mostow work which does not make this assumption (note: I am assuming this is what is meant, rather than the listed reference numbers of 8-9 which are BKT). Additionally, the matrix factorization methods that are mentioned seem to be cases of "treating the student knowledge as static" and thus might be cited in this sentence as well (e.g., Desmarais' work or the SPARFA work). I noticed that the Lan et al. factorization work was included in the references, but could not find it in the main paper; it also seems relevant here and is an example of a model with time-varying student knowledge that does not rely on an expert labeling.Update: Thanks for the clarification in the rebuttal about the references - I'm leaving the notes here in case it's helpful to make sure things are fixed and because I think the apparent contradiction between the claims about existing methods and the Gonzalez-Brenes and Mostow work should be addressed, but the rebuttal addresses my concerns.  Also in the introduction, the paper notes that the "best models produced by matrix factorization have had low-dimensional skill spaces, specifically N_skill = 2"; it would be helpful to provide a citation here to clarify the context, given that this is not true for a number of results reported in papers about creating Q-matrices (e.g., Tatsuoka's original fraction work, Desmarais's refinement of these Q-matrices).  Minor note:On line 425, the same issue with reference numbers occurs as in the intro (8-9 are substituted when I suspect 10-11 are intended).