From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper47/Authors
comment: Thank you for your comments, you have addressed some interesting points that we have discussed internally in relation to the value of hierarchical representations.

To you first point : Layers, channels, kernel size were empirically tested and chosen from common examples and made consistent throughout experiments except for deliberate comparisons e.g. 8-Layer BlendCNN vs 3-Layer BlendCNN.  The validation set used was the same size as the labelled training set (i.e. 100 examples per class).

In regards to the value of the hierarchical representations, here are the results that you asked for :

Testing of the 8-Layer BlendCNN with/without the hierarchical components gave the following scores for the distillation (using both labelled and unlabelled examples), which shows the uplift from incorporating the hierarchical representations :

3-Layer BlendCNN/TopOnlyCNN : 91.2/90.7    98.2/97.7    71.0/69.8
8-Layer BlendCNN/TopOnlyCNN : 91.2/90.7    98.5/97.1    70.8/68.7

Clearly, if we had more space we would have been able to incorporate these results, as well as those that address other directions suggested in other reviewer comments - all of which have been very helpful.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper47/AnonReviewer6
title: Good results with limited novelty
review: The paper described an application of distillation where a big teacher language model is used to train a small convolutional student model on downstream text-classification tasks. The main novelty seems to be the leveraging of hierarchical representations. This may be limited but sufficient for a workshop paper. The paper is clearly written although it could have made clearer what the novel aspects are.

Pros:
- good results
- clear enough

Cons:
- (missing information) how were the hyper-parameters (number of layers, number of channels, kernel-size, etc.) of the student networks chosen? please add this information. For example, have you used a validation set? And how big was it?
- (missing information) how much did the hierarchical representations improve the results with respect to using only representations from last convolutional layer?
rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper47/AnonReviewer3
title: Simple idea and good experiment results
review: This paper proposes to use Knowledge Distillation to help small student networks learn from big Language Model (teacher) on text classification task. For the student network, a novel CNN architecture named 'BlendCNN' is proposed to leverage hierarchical representations for text classification. Experiments on 3 standard datasets show the effectiveness of proposed Knowledge Distillation scheme and the proposed 'BlendCNN'.

Pros:
- simple idea, easy to follow
- significant experiment results

Cons:
- '2-Layer BiLSTM' (2016) and 'KimCNN' (2014) are quite old. What about the proposed 'BlendCNN' compared with more recent methods?

Questions:
-  What's the results of '3-Layer BlendCNN', '2-Layer BiLSTM' and 'KimCNN' without distillation for Table 2?
- What is the training cost increase when using distillation?

rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: ~Martin_Andrews1
comment: Thank you for your feedback.  

To address your questions about the distillation results, we can add to the last two lines of Table 2 so that they read :

3-Layer BlendCNN : 91.2/88.4    98.2/95.5    71.0/63.4
8-Layer BlendCNN : 91.2/89.9    98.5/96.0    70.8/63.4

where the first number in each pair is the score with labelled+unlabelled training data (as given in the current paper), 
and the second number in each pair is the score with distillation run on just the labelled data.  This quantifies the advantage of adding unlabelled data into the distillation phase of the student model training.

In terms of model training times, the rates of convergence are similar.  However, the wall-clock time is naturally longer when the unlabelled data is added, but the difference is less than being 11x as long overall : Only labelled data is used to train the OpenAI Transformer model; that model is then used (once) to pseudo-label both the labelled and unlabelled data; and the final combined dataset is then used to train the smaller student model.  We will add some text to this effect in a revised version.

title:

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper47/AnonReviewer1
comment: Rating: 3 of 5

Summary:
An Application of model distillation where small student networks are trained by distillation loss provided from the larger teacher network (OpenAI Transformer model). The main novelty is studying the model distillation under label sparse condition. The authors then apply some semi-supervised learning / pseudo-labeling technique to leverage a large amount of unlabeled data to improve the distillation in the downstream tasks.

Pros:
- Good results (small models, fast inference speed)
- Clarity is good

Cons:
- Experiments relatively few, but okay for a short workshop paper.
- What's the impact of #unlabel data for distillation? It would be nice to also show the distillation without any unlabeled data, which is the original setting of model distillation.
- What's the training time for distillation? Again, it is interesting to see how #unlabeled data has an impact on the distillation training.

Confidence:  3 of 3