From: ICLR.cc/2019/Workshop/LLD/Program_Chairs
title: Acceptance Decision
decision: Accept

From: ICLR.cc/2019/Workshop/LLD/Paper7/AnonReviewer1
title: Review of "Learning from Samples of Variable Quality"
review: Summary of the paper:
This paper proposes a technique to tackle the case where the labels of a supervised problem come from different sources and have varying quality. To this end, a three steps approach called FWL (Fidelity Weighted Learning) is proposed. In particular, FWL consists in constructing a “student” and a “teacher” interacting to refine the quality of the prediction.

Some numerical experiments supporting the approach are proposed. In particular, the FWL algorithm is tested on the problem of document ranking. 
An investigation of the sensitivity of the approach to the quality of the weak annotator is also proposed.


A few comments and questions:
-the fact that the algorithm uses a Neural Net should be emphasized more in the introduction
-Can the proposed approach be extended to other learning algorithm (i.e. not only a NN)?
-Why specifically use a Gaussian process? Can another approach be used?
-concerning the investigation on the “sensitivity of the FWL to the quality of the weak annotator” (sic),
does a “phase transition” occur? i.e. is there a threshold on the quality of the weak annotator below which FWL does not yield any improvement?

Reviewer’s assessment:
I found the paper to be fairly well written. The ideas are exposed clearly and the numerical results support the approach. Since the topic of this work clearly falls within the scope of the workshop, I recommend to accept this paper.


rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: ICLR.cc/2019/Workshop/LLD/Paper7/AnonReviewer2
title: interesting setup, but methods not well motivated and explained
review: The authors propose a distillation approach to learn from weakly annotated data aside from a strongly annotated data. The general setup and approach of distillation seems interesting, but it is unclear what the resulted model will learn to do. Specifically, I have the following concerns/questions:

- the proposed method is kind of hacky, in the sense that it is not well motivated and grounded on any specific intuition, theoretical or not. 
- it is not clear to me what the student will converge to after training in the step 3. will it just learn to predict whatever the teacher outputs? if that's the case why do we distill it, aside from the obvious fact that now it is parametric rather than a GP? 
- following the previous point, have you compared the performance of the student with the performance of the teacher? 


rating: 2: Marginally below acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct