Paper:
Improving Simple Models with Confidence Profiles
In this paper, we propose a new method called ProfWeight for transferring information from a pre-trained deep neural network that has a high test accuracy to a simpler interpretable model or a very shallow network of low complexity and a priori low test accuracy. We are motivated by applications in interpretability and model deployment in severely memory constrained environments (like sensors). Our method uses linear probes to generate confidence scores through flattened intermediate representations. Our transfer method involves a theoretically justified weighting of samples during the training of the simple model using confidence scores of these intermediate layers. The value of our method is first demonstrated on CIFAR-10, where our weighting method significantly improves (3-4%) networks with only a fraction of the number of Resnet blocks of a complex Resnet model. We further demonstrate operationally significant results on a real manufacturing problem, where we dramatically increase the test accuracy of a CART model (the domain standard) by roughly 13%.

Review:
The authors introduce ProfWeight - a method for transferring knowledge from a teacher model to a student model. A "confidence profile" (taken from classification layers placed throughout the network) is used to determine which training samples are easy and which are hard. The loss function for the student model is weighted to favor learning the easier samples. The authors test this method on CIFAR10 and a real-world dataset.  Quality:  The idea presented by this paper is interesting and well-motivated. The method and results could be presented with more clarity, and the paper could benefit from some additional empirical analysis. However, overall the quality of the experiments are good.  Clarity:  Most of the section are straightforward. However, the method section is confusing and is not strategically organized. Additionally, there are some notational issues (see small notes).  Specifically, the paper is missing a description of ProfWeight in the text. The components of the algorithm (weights, AUC/neural net) are described in Section 3, but there is no overview of the exact steps of the algorithm (learn a simple model, use error of simple model and the AUC/neural net to learn weights, retrain simple model with weights). A reader can only learn this overview from Algorithm 1, which is hard to parse because of all the notation. Similarly, Algorithm 3 should contain more description in the text.  Originality:  The idea of student/teacher transfer with sample weighting is novel and interesting.  Significance:  Model compression and interpretability is an important problem, and this paper presents a promising new method. It will be of interest to the community at large.  Overall:  I vote to accept this paper, since the method proposed is novel and seems promising. I do believe the methods section of this paper should be clarified, and more empirical results on other datasets would strengthen the paper.  Detailed comments:  Section 3.2 is a bit terse. The left-hand side of equation 2 is difficult to parse (what are the parameters \theta referring to? the parameters of the simple model (in which case, how does this differ from M'), or the complex model parameters?) which makes this section confusing.  Additionally, is there a guarantee that Algorithm 3 converges? It would be good to include such a guarantee since (1) frames ProfWeight as an optimization problem.  I wonder if your CIFAR results are influenced by the training set split? A model trained on 30,000 samples or 20,000 samples overfits quite a bit, as you note in the paper. It may be useful to try these experiments on a larger dataset like SVHN or ImageNet where a split dataset would still contain hundreds of thousands of samples.  Finally, I really appreciate the real-world experiment with semi-conductor manufacturing. It is good to see a method tested in real-world settings.  Smaller comments:  - 16-17: some example citations would be good here - "The MAIN criticism of these models has been their lack of transparency" - this is a bold claim. - 31: "Small data settings" - It's unclear how model transfer, as you've described in this paper, would work in such a scenario. - 59: "The first function we consider, area under the curve (AUC) traced by the confidence scores, shows much improved performance." - what is the performance you're talking about? - In the description of the method (and in Algorithm 1), it may be good to use the term "layer" instead of "unit", since "unit" can mean many things in the context of neural networks. - It would be good to re-define all notation in Algorithm 1 (i.e. \beta, the arguments to \mathcal L, etc.) - Labels in 127-129 should be capitalized (e.g. as seen in Algorithm 2) - There is inconsistent notation, which makes Section 3 a bit difficult in some places. For example, the weights are a set (W) in 97, but a vector in Algorithm 1. Additionally, the inputs/labels should probably be notated as vector (e.g. in 1 - as far as I can tell - y represents all labels, and therefore should be a vector (and bolded)?) - 135: there's a reference to Line 7 of Algorithm 3, but no line numbers in Algorithm 3. Also, it would be good to repeat this step in the text, rather than making the reader refer to the algorithm block. - Algorithm 3: should define what N is - Algorithm step 4: missing a \gamma in the update weights equation? - 193: "The set of units U (as defined in Section 2.1) whose representations are used to train the probes are the units in Table 1 of the trained complex model." - I'm not sure what this means? - 196: you should explain here that set 1 is used for the complex model and set 2 is used for the simple model. I was confused by this split until I read further. - Technically, [13] did not invent distillation - [13] is a refinement of the method proposed by [9]. - You should include the following citation on your list of distillation methods: Tan, Sarah, et al. "Detecting bias in black-box models using transparent model distillation." arXiv preprint arXiv:1710.06169 (2017).