Paper:
Optimizing Instructional Policies
Psychologists are interested in developing instructional policies that boost student learning. An instructional policy specifies the manner and content of instruction. For example, in the domain of concept learning, a policy might specify the nature of exemplars chosen over a training sequence. Traditional psychological studies compare several hand-selected policies, e.g., contrasting a policy that selects only difficult-to-classify exemplars with a policy that gradually progresses over the training sequence from easy exemplars to more difficult (known as fading). We propose an alternative to the traditional methodology in which we define a parameterized space of policies and search this space to identify the optimal policy. For example, in concept learning, policies might be described by a fading function that specifies exemplar difficulty over time. We propose an experimental technique for searching policy spaces using Gaussian process surrogate-based optimization and a generative model of student performance. Instead of evaluating a few experimental conditions each with many human subjects, as the traditional methodology does, our technique evaluates many experimental conditions each with a few subjects. Even though individual subjects provide only a noisy estimate of the population mean, the optimization method allows us to determine the shape of the policy space and to identify the global optimum, and is as efficient in its subject budget as a traditional A-B comparison. We evaluate the method via two behavioral studies, and suggest that the method has broad applicability to optimization problems involving humans outside the educational arena.

Review:
Paper 1279 – Optimizing Instructional Policies In 
      this paper, the authors adapt an optimization technique based on Gaussian 
      process regression to select the parameters of experiments or teaching 
      regime, which will optimize human performance. They evaluate their method 
      in two behavioral experiments (one on the presentation rate of studied 
      items and another on the ordering of examples while learning a novel 
      concept), demonstrating that it is vastly more efficient than traditional 
      methods in psychology for exploring a continuous space of conditions. 
      Note: I have revised my score to reflect the author feedback's 
      assurance that the starting point of Experiment 1 wasn't the optimum. 
      However, I still don't fully understand how what they wrote in the 
      feedback connects to what they wrote in the paper. I implore the authors 
      to make sure it is clear in the final version of their paper. 
      There is a lot to like in this paper. They use modern machine 
      learning tools to solve a hot problem in cognitive science, optimal 
      experimental/instructional design, and validate their method by 
      replicating previous memory and concept learning experiments. Their 
      adaptation of a previously developed optimization technique is natural and 
      clever, albeit somewhat incremental. Besides for one potentially 
      catastrophic issue issue with the first experiment (that I will describe 
      below), the experiments do a good job of showcasing their method. 
      Additionally, the paper is well written and the framing and discussion is 
      very appropriate for a NIPS audience (though I have some concerns about 
      the framing). However, there are two serious issues with the paper 
      that leave me very concerned. 1. Although Experiment 1 may seem at 
      first to provide support for their method (as it shows quick convergence 
      to the optimal presentation rate), after a closer reading the starting 
      condition for the first subject is basically already at the optimum! 
      “Subject 1’s training policy, $x_1,$ was set to the middle of the range.” 
      The range of the policy space is [ln(.25), ln(5)]. This would make the 
      midpoint 0.1116, which corresponds to a duration of 1.118 seconds. The 
      reported optimum is 1.15 sec, which is extremely close to the starting 
      value for the first subject (1.118 sec). If this is true (and I hope the 
      authors can explain why this is not the case!), the new interpretation 
      would at best be that the method does not move away from the optimum when 
      it already is at it. This basically would invalidate the results of 
      Experiment 1, and make the article difficult to support without its 
      removal. 2. Although the paper is framed as being for constructing 
      “instructional policies” and I am somewhat convinced that POMDP/RL methods 
      are prohibitive to adapt for their situation, I really would have liked to 
      have seen their method compared against at least one other computational 
      method. It seems like there are several methods from optimal experimental 
      design that would not be too difficult to adapt to this setting. In fact, 
      I am not completely convinced that optimal experimental design is that 
      different from finding optimal instructional policies (where instead the 
      criterion is maximizing a function rather than discriminating between 
      models viewed as functions). At this point there is a long tradition of 
      optimal experimental design techniques that work over continuous spaces, 
      some of which have already been applied to cognitive psychology. For 
      example, Myung & Pitt (2009) Psych Rev. and other work from their 
      group (including their 2009 NIPS paper), Kujala (2011) has a review on 
      Bayesian adaptive estimation that discusses psychological applications, 
      and Vul et al. (2009) in the journal Seeing and Perceiving (available for 
      free from Vul’s website). The paper would have been much stronger if there 
      was a direct comparison with at least some of these methods (even if they 
      had to be adapted). In fact, although the framing of the paper is about 
      instructional policies, the experiments and presented results do not seem 
      to quite fit it (in the intelligent tutoring systems sense, which would be 
      more of a discrete optimization problem). It seemed more about finding 
      parameters/conditions to optimize human performance. However, I agree that 
      Experiment 2 does a decent job of fitting the framing of the paper. 
      Minor comments: Abstract: “… search this space to identify the 
      optimum policy.” -> “… search this space to identify the optimal 
      policy.” The last paragraph of the introduction seems out of place. 
      Perhaps an outline paragraph instead? Second paragraph of Section 2: 
      “The approach we will propose likewise requires specification of a 
      constrained policy space, but does not make assumptions about the internal 
      state of the learner or the temporal dynamics of learning.” No assumptions 
      about the internal state of the learner seems a bit too strong. There are 
      at least some minimal assumptions inherent in the generative model of the 
      student. I agree they are not nearly as strong as some other methods 
      though. Third paragraph of Section 2: I would have liked to see a 
      little more detail about why numerical approximation methods that account 
      for noisy function evaluations would not work. End of Section 2 - 
      Section 2.1: Why outline the entire function if you are only interested in 
      the maximum (and do not want any assumptions about the internal state of 
      the learner)? Section 2.2: you start to use $f_x$ before it is 
      defined. “In this paper, we give subjects questions with two equally 
      probable choices…” How do you know the choices are precisely 0.5 in the 
      minds of the subjects? Discussion of limitations: what about discrete 
      parameter spaces? Section 3: If there are 30/d trials and d is 
      continuous, what does a fractional trial mean? Figure 3: Label units 
      on x-axis (ms). Experiment 2: Why did you have to discard objects with 
      [2.75-3.25] graspability from the set? Figure 4: I did not understand 
      this plot (I printed it out on B&W, but even looking at the color, it 
      does not really make sense to me). Figure 5a: It could be nice to have 
      large arrows across the subplots that provide an interpretation of what a 
      small vs. medium vs. large fading/blocking policy means for the ordering 
      of stimuli (small fading means close to far or random with respect to the 
      boundary?)