Paper:
In the context of optimization, a gradient of a neural network indicates the amount a specific weight should change with respect to the loss. Therefore, small gradients indicate a good value of the weight that requires no change and can be kept frozen during training. This paper provides an experimental study on the importance of a neural network weights, and to which extent do they need to be updated. We wish to show that starting from the third epoch, freezing weights which have no informative gradient and are less likely to be changed during training, results in a very slight drop in the overall accuracy (and in sometimes better). We experiment on the MNIST, CIFAR10 and Flickr8k datasets using several architectures (VGG19, ResNet-110 and DenseNet-121). On CIFAR10, we show that freezing 80% of the VGG19 network parameters from the third epoch onwards results in 0.24% drop in accuracy, while freezing 50% of Resnet-110 parameters results in 0.9% drop in accuracy and finally freezing 70% of Densnet-121 parameters results in 0.57% drop in accuracy. Furthermore, to experiemnt with real-life applications, we train an image captioning model with attention mechanism on the Flickr8k dataset using LSTM networks, freezing 60% of the parameters from the third epoch onwards, resulting in a better BLEU-4 score than the fully-trained model. Our source code can be found in the appendix.

Review:
In this paper, the authors performed an empirical study on the importance of neural network weights and to which extent they need to be updated. Some observations are obtained such as from the third epoch on, a large proportion of weights do not need to be updated and the performance of the network is not significantly affected.
Overall speaking, the qualitative result in the paper has already been discovered in many previous work, although the quantitative results seem to be new. However, there is large room to improve regarding the experimental design and the comprehensiveness of the experiments. Just name a few as follows:
1)	For different models and different tasks, the quantitative results are different. There is no deep discussion on the intrinsic reason for this, and what is the most important factor that influences the redundancy of weight updates. The authors came to the conclusion that from the third epoch on, no need to update most of the weights. “3” seems to be a magic number to me. Why is it? No solid experiments were done regarding this, and no convincing analysis was made.
2)	The datasets used in the experiments are not diverse enough and are not of large scale. For example, the CIFA-10 and MNIST datasets are relatively of small scale. What if the datasets are much larger like ImageNet. In such more complicated case, will the weight updates still be unnecessary? Will the ratio and the epoch number change? What is the underlying factor determining these? For another example, there are many NLP datasets for language understanding and machine translation, which are of large scale. Why choosing an image captioning dataset (which I do not agree to be real-life experiments when compared with language understanding and machine translation)? Can the observations generalizable to more complicated tasks and datasets?
3)	The models studied in the paper are also a little simple, especially for the text task. Why just using a single-layer LSTM? Why not popularly used Transformer? 
As a summary, for an empirical study to be convincing, the tasks, datasets, scales, model structures, detailed settings, and discussions are the critical aspects. However, as explained above, this paper has not done a good job on these aspects. Significantly more work needs to be done in order to make it an impactful work. 
*I read the author rebuttal, but would like to keep my rating unchanged.