From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: This paper carried out additional experiments regarding the dynamic parameter reallocation, which trains/searches a sparse network directly without a dense model beforehand. My major concern is that the major contribution of this paper is the experimental study, lacking novelty and originality. In particular, the paper is primarily based on the work in Anonymous [2019]. From the reviewer point of view, additional efforts in terms of novelty and methodology are required in this paper.

From: NIPS.cc/2018/Workshop/CDNNRIA/Program_Chairs
comment: Dear authors,

we fully agree. Other reviewers have been assigned, but unfortunately did not submit their reviews so far. At least one additional review should be available by tomorrow.

Best regards,
CDNNRIA chairs

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper31/Authors
comment: 
Dear chair,

Thank you for organizing this topical workshop. As regards to our paper, we feel that one review alone is not enough to yield a balanced evaluation of our work. While we got valuable comments from the reviewer, the primary finding of the paper was not particularly noted by him/her. We would thus greatly appreciate if our work can be evaluated by an additional independent party.

Kind regards,
Authors

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper31/Authors
comment: Thank you for your comments. We hope to be able to address them below:

C1:
The primary finding (that DPR was essential during training, even with prior knowledge of a high-performance sparse network structure) was described using several sentences in the abstract, starting from "We show that neither the structure..". To avoid confusion, we will preface this sentence with "Our primary finding is that we show that.."

C2:
In a 3-page workshop paper, a 'related work' section has to necessarily remain short in order to be able to present and discuss the main findings in a self-contained way. Several workshop papers eschew a 'related work' section altogether and simply mention the related references in passing in the main text. We will try to compress the text and figures as much as we can to find some additional space for extra sentences describing how previous DPR schemes work

C3: 
We apologize for not being sufficiently clear.  We empirically evaluated why static methods fail, and showed that DPR is necessary for effective direct training of compact sparse networks.  It is logically sufficient to present one successful dynamic method to show what static methods lack.  Our intention was not to compare/contrast different dynamic methods, and certainly not to argue that all DPR methods are successful.  This is because: (a) even if we give more than one successful DPR method, it still cannot prove _all_ DPR methods work, (b) it is easy to come up with a badly designed DPR method that fails.  

C4:
We are afraid we disagree with the evaluation criterion employed by the reviewer. Namely: that a related paper is in review, so the short workshop paper should be appended to it as it builds on the techniques of this related paper. The other paper's results stand on their own, so do the results of this workshop paper. The fact that the former is being reviewed elsewhere should not affect the evaluation of the later in our opinion. Rather, we ask that our submission be evaluated like any other paper, i.e,  based on whether the results are reproducible, whether they are significant enough and novel enough (considering all prior work, including the related paper in review) to merit a workshop publication.

Thank you,
Authors

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper31/AnonReviewer7
comment: Dear authors,

please find my comments below:

@C1 *** This summary did not state the primary finding of the paper.... ***  

The reviewer's summary contains the main points covered from reading/reviewing the paper. If your desired primary / most important outcome of the paper is not covered in the summary, it should be explicitly stated as the main outcome in the abstract to make this clear - even to avoid misinterpretations by other reviewers/readers.

@C2 *** We respectfully argue that the length alone is not a valid criticism without specifics on what in particular we missed in the rather short Related Work session.***

The "shortness" of the related work session is not caused by the pure number of words written. The entire section is "short" because of the spare explanations/short summary of the papers referenced.

@C3 ***As stated above, the Reviewer misunderstood the main point of this paper:... its aim was not ... rather, it elucidated WHY DPR methods in general are much more effective than static training.  Simply put, the question is not WHETHER dynamic method A is better than dynamic method B, but WHY dynamic methods in general are more effective than static.   ***

If you claim, that "dynamics methods in general are more effective than static" such a claim should be proven/validated by more than more than one example only. This is especially important, when the example-method used has been introduced/proposed by yourself.

@C4 *** It is not clear from the Reviwer's comment why this workshop paper, by its own merits, should be “part of an initial evaluation/discussion” of the original paper ***

In the reviewers opinion the findings/outcome of this paper are strongly related to the your DPR-approach which is currently under review. As stated above, without comparison to other DPR-methods stating similar behaviour, the findings in this paper have only been proven for this specific work and should thus be placed there as appendix or part of the discussion section.

@C5 *** This is factually untrue at the moment the Reviewer's comment was posted.... ***

Regarding this last comment you are right. I have to apologize and withdraw this last point. As checked today, in contrast to last week it is actually possible for me to access the original publication - even by the link mentioned in the reference section of your paper.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper31/Authors
comment: 
We thank the Reviewer for the comments.  However, the Reviewer's summary of our paper was inaccurate, his/her criticisms contained a factual error and lack factual support.  We describe these in the following as a point-by-point rebuttal, and respectfully ask for a correction of factual error, provision of factual support, and a re-evaluation of the merits of our manuscript.  

***Summary: The authors of the paper studied behaviour of their dynamic parameter re-allocation (DPR) approach (which has been already introduced in another publication) and claim to show, that their network pruning technique always improves the results even if applied only during the initial training epochs. The authors present experiments with a “wide resnet” trained on the CIFAR10 dataset to justify this claim.***

This summary did not state the primary finding of the paper.  
We asked and answered the following question: what makes training effective for compact sparse networks?  Our finding: not sparse structure, not initialization, not their combination, but rather the dynamics of parameter reallocation (DPR).  
We are interested in DPR as a general class of training methods, of which the specific method we used is a mere example.  Further, this work, for the first time, argued for the equivalence between memory-efficient DPR training and overparameterization, a major significance missed by the summary. 

***The related work session is rather short., and the main outcome/content of the referenced papers should be scratched in one or two sentences.***

We respectfully argue that the length alone is not a valid criticism without specifics on what in particular we missed in the rather short Related Work session.

***There is no comparison with other methods potentially showing similar behaviour.***

As stated above, the Reviewer misunderstood the main point of this paper: its aim was not the evaluation of a new DPR method and comparison against old ones; rather, it elucidated WHY DPR methods in general are much more effective than static training.  Simply put, the question is not WHETHER dynamic method A is better than dynamic method B, but WHY dynamic methods in general are more effective than static.  

***Although the outcome of the paper is interesting, it should be part of an initial evaluation/discussion in the paper the DPR approach is introduced.***

Thank you for acknowledgement of the positive outcome of our paper. 
This work is an entirely separate piece of work from the original paper, not only chronologically subsequent to it, but logically dependent on the findings presented in it.  It is not clear from the Reviwer's comment why this workshop paper, by its own merits, should be “part of an initial evaluation/discussion” of the original paper.  It is equally unclear why this makes it not suited for publication in this workshop.  

***More details about the algorithm-description are recommended, especially as the method is obviously published in another paper which cannot be checked due to anonymization issues.***

This is factually untrue at the moment the Reviewer's comment was posted.  The source code of the method published in the original paper is openly accessible to the public in its entirety despite the ongoing double-blind review ( https://openreview.net/forum?id=S1xBioR5KX&noteId=HyllVTC6sQ ). 
Moreover, the original paper itself has been selected as part of a reproducibility challenge, currently being validated by a third-party team ( https://github.com/reproducibility-challenge/iclr_2019/issues/31 ). 
We believe the exhaustive details of the methods presented in the original full-length paper and the complete openness of its supporting source code justify a simple citation in this short workshop paper, because any reader of this paper is able to follow the citation and conveniently obtain all detailed methods, its source code and the status of the third-party reproducibility study.

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: Summary: The authors of the paper studied behaviour of their dynamic parameter re-allocation (DPR) approach (which has been already introduced in another publication) and claim to show, that their network pruning technique always improves the results even if applied only during the initial training epochs. The authors present experiments with a “wide resnet” trained on the CIFAR10 dataset to justify this claim.

Review: 
o	The related work session is rather short., and the main outcome/content of the referenced papers should be scratched in one or two sentences.
o	There is no comparison with other methods potentially showing similar behaviour.
o	Although the outcome of the paper is interesting, it should be part of an initial evaluation/discussion in the paper the DPR approach is introduced.
o	More details about the algorithm-description are recommended, especially as the method is obviously published in another paper which cannot be checked due to anonymization issues.

Rating: Marginally below borderline in actual state.
Reviewers confidence: moderate confidence that the review is correct