Paper:
Learning Trajectory Preferences for Manipulators via Iterative Improvement
We consider the problem of learning good trajectories for manipulation tasks. This is challenging because the criterion defining a good trajectory varies with users, tasks and environments. In this paper, we propose a co-active online learning framework for teaching robots the preferences of its users for object manipulation tasks. The key novelty of our approach lies in the type of feedback expected from the user: the human user does not need to demonstrate optimal trajectories as training data, but merely needs to iteratively provide trajectories that slightly improve over the trajectory currently proposed by the system. We argue that this co-active preference feedback can be more easily elicited from the user than demonstrations of optimal trajectories, which are often challenging and non-intuitive to provide on high degrees of freedom manipulators. Nevertheless, theoretical regret bounds of our algorithm match the asymptotic rates of optimal trajectory algorithms. We demonstrate the generalizability of our algorithm on a variety of grocery checkout tasks, for whom, the preferences were not only influenced by the object being manipulated but also by the surrounding environment.

Review:
The paper present an co-active learning approach to 
      generate user- or task-preferred trajectories for manipulation accounting 
      for task-, object-, environment- and human-related context features. 
      The paper addresses a timely topic with an interesting approach 
      based on the rather novel idea of incorporating task context into the 
      problem of trajectory learning. It is understandable and easy to read, and 
      presents an evaluation with quantitative and subjective measures, the 
      former carried out in simulation with a 7-dof PR2 arm However, 
      there are weaknesses. Related literature is poorly discussed with 
      several omissions of related works and a rather shallow level of 
      discussion. Highly related are the lines of work by Sisbot and Alami et 
      al. (e.g. [2]) which is complementary to yours -- they take a model-based 
      approach, no learning -- but are, to the knowledge of the reviewer, the 
      first to address the issue of user preferences in robot motion planning 
      and the work by Cakmak et al. [1] on complex manipulation tasks in an 
      active learning framework. The line of work by Ijspeert and Schaal on DMPs 
      seem to be underrepresented in particular given the many variations and 
      related extensions of their original idea. The works by Calinon and 
      Billard et al. (e.g. [3]) on probabilistic motion generalization are not 
      at all cited and discussed. Also, it appears unjustified to dismiss the 
      cited and highly related work [32] so easily ("since they learn in 
      discretized action space their approach is not directly applicable to 
      manipulators with large action spaces"). Their work was evaluated on four 
      (continuous-state) domains similar to yours and the question of the 
      applicability of their method should at least be elaborated in more 
      detail. The paper's contribution is limited by the fact that the 
      core theoretical properties such as the TPP algorithm and the regret 
      bounds are from previous work [28] and that the amount of manual effort in 
      designing the features is very high (admittedly, certain context 
      information such as object properties are hard to get automatically). This 
      makes that the main contribution of this paper lies in the feature 
      definition, the implementation and evaluation. Also, the paper assumes 
      perfect knowledge of the environment and all static and dynamic objects. 
      Some questions remain unclear regarding interactive feedback: It 
      is not clear what you mean by correcting waypoints, can the user just 
      confirm/dismiss them or edit their 3D position? How many waypoints did a 
      typical trajectory have in the experiments? How much time did the training 
      phase take (for both kinds of feedback) and how many times bad waypoints 
      needed to be corrected during training? In the evaluation, the 
      nDCG metric is not sufficiently explained neither is Table 1 (values in 
      the cells?). The user study lacks a hypothesis and is of limited 
      significance with N = 2. References: [1] M. Cakmak et al., 
      "Designing Robot Learners that Ask Good Questions", HRI 2012 [2] E.A. 
      Sisbot et al., "A human aware mobile robot motion planner", TRO, 2007 
      [3] S. Calinon et al., "Active Teaching in Robot Programming by 
      Demonstration", RO-MAN 2007