Paper: A multiobjective continuation method to compute the regularization path of deep neural networks (ICLR-2024)

Reviewer comment (Novelty):
Missing references. The authors claim in introduction that `Very recently, there was a first attempt to extend the concept of regularization paths to DNNs by means of treating the empirical loss and sparsity.` However, dating back to 2020, the problem has been studied in OBProxSG where the multi-objective could be considered as a weighted sum of each individual target objective. [1] Orthant Based Proximal Stochastic Gradient Method for $\ell_1$-Regularized Optimization, 2020.

Author response:
**You are correct that the mentioned paper goes in a similar direction, and we have added it to our discussion on related work. However, similar to other works, this paper does not address the computation of the entire Pareto front. In this regard, our paper in fact is the very first that truly solves a multiobjective optimization problem in such high dimensions, both in the machine learning and the optimization literature.**

Reviewer comment (Experiments):
Experiments are not sufficient. In particular, the paper studies multi-objective problems. However, in the experiment, only cross-entropy is considered as loss function. Where are other losses to form a real multi-objective problem? Meanwhile, the network and dataset are too simple to validate the efficacy.

Author response:
**A second, much more complex experiment using CIFAR10 and a more complex model has been included in the updated version of our paper. In terms of the problem not being multi-objective, we disagree in the sense that the problem truly does have a Pareto front for the two tasks we consider. On the other hand, we agree that multiple main tasks are just as interesting, in particular when this results in more than two objectives. We have added a discussion on this aspect in the outlook of the paper. As the goal of this work is to provide the Pareto front for such a large-dimensional problem for the first time, we would rather leave this additional challenge for future work, and then also address the question of steering on higher-dimensional sets.**

Reviewer comment (Theory):
The title and the paper over-claims to work on multi-objective problems, but actually the method is only for two objective problems. For more than two objectives, it seems to me that the method will not explore the entire Pareto front.

Author response:
**We are convinced that our proposed method works on more than two objective functions. From a computational perspective, both the predictor and corrector steps should not be any more expensive. However, given that in more than two objective functions, the Pareto front is no longer a line, the task of selecting a predictor direction becomes more challenging. We have added some references and a short discussion to address this concern. However, the task of our paper was to compute the entire front of very high-dimensional, non-convex fronts for the first time, such that we would like to leave the explicit treatment of more objectives for future work. We now say so in the conclusion.**

Reviewer comment (Experiments):
A comparison is missing with the standard approach to obtain the regularization path that starts with no penalty and gradually increases the L1 penalty lambda and optimize the penalized loss for each value of lambda.

Author response:
**The comparison of our method with standard approach as you have described can be seen in the comparison of our method with weighted sum where the weighting parameter lambda takes the values 1 for sparsity only $\ell^1$ norm, and lambda = 0 loss only. The results of the comparisons have been included as a table in the uploaded PDF version of our paper.**

Reviewer comment (Experiments):
Can the proposed method work on more than two objective functions?

Author response:
**We are convinced that our proposed method works on more than two objective functions. From a computational perspective, both the predictor and corrector steps should not be any more expensive. However, given that in more than two objective functions, the Pareto front is no longer a line, the task of selecting a predictor direction becomes more challenging. We have added some references and a short discussion to address this concern. However, the task of our paper was to compute the entire front of very high-dimensional, non-convex fronts for the first time, such that we would like to leave the explicit treatment of more objectives for future work. We now say so in the conclusion.**

Reviewer comment (Experiments):
How does the method compare in computation time and accuracy with the standard approach for obtaining the regularization path described above?

Author response:
**The comparison of our method with standard approach as you have described can be seen in the comparison of our method with weighted sum where the weighting parameter lambda takes the values 1 for sparsity only $\ell^1$ norm, and lambda = 0 loss only. The results of the comparisons have been included as a table in the uploaded PDF version of our paper.**

Reviewer comment (Theory):
Can the proposed method work with the SCAD/MCP penalties?

Author response:
**Thank you for bringing this up. We believe that including other penalties such as the SCAD (Smoothly Clipped Absolute Deviation) and MCP (Minimax Concave Penalty) which have non-convex shapes similar to the $\ell^1$ norm as conflicting objectives to the loss function would be possible for our method. This could be the case when exploring more than two objective functions.**

Reviewer comment (Evaluation):
You highlight the efficiency of your method in several aspects, such as reducing the computational expense by avoiding the computation of the full gradient in Section 4, and also provide your experimental settings. While your experimental settings are detailed, the inclusion of metrics on computational time would significantly enhance the evaluation of your method's efficiency.

Author response:
We have included a table detailing the metrics and experimental settings for our continuation method (CM) and weighted sum (WS) in the updated PDF version of our paper.

Reviewer comment (Experiments):
You claim that the algorithm shows a performance that is suitable for high-dimensional learning problems, may should be supported by more results extended to deeper DNNs including non-linear layers, and more complex datasets just as you mention in Section 5.

Author response:
We have implemented both our continuation method (CM) and the weighted sum (WS) method on CIFAR10 using a CNN model architecture containing two convolutional layers and two fully connected linear layers with a total number of 4,742,546 parameters. We demonstrate that our CM still generalizes better than the WS and provides a set of well distributed points on the front. Updated results of our new experiments can be seen in the updated PDF version of our paper and we will be adding an appendix section in the final paper submission.

Reviewer comment (Experiments):
Have you compared the performance of your method with existing regularization path generation methods?

Author response:
Yes, we implemented methods such as the early stopping and grid search (tuning of different hyperparameter values learning rate, weight decay). We observed that for both CM and WS, these methods tend to have similar effects. E.g., early stopping reduces the accuracy of both methods, and different values of the learning rate and weight decay tend to affect the rate of convergence to the Pareto front for both methods. We will be including a detailed discussion on these in the appendix section which will be added in the final submission.

Reviewer comment (Experiments):
One of your key merits is avoiding overfitting, and it would strengthen the claim to conduct additional experiments.

Author response:
By having added the CIFAR10 example, we believe that we can now make a much stronger case for our claim that we can avoid overfitting.

Reviewer comment (Experiments):
Have you compared the performance of your method with existing regularization path generation methods?

Author response:
Yes, we implemented methods such as the early stopping and grid search (tuning of different hyperparameter values learning rate, weight decay). We observed that for both CM and WS, these methods tend to have similar effects. E.g., early stopping reduces the accuracy of both methods, and different values of the learning rate and weight decay tend to affect the rate of convergence to the Pareto front for both methods. We will be including a detailed discussion on these in the appendix section which will be added in the final submission.

Reviewer comment (Experiments):
One of your key merits is avoiding overfitting, and it would strengthen the claim to conduct additional experiments.

Author response:
By having added the CIFAR10 example, we believe that we can now make a much stronger case for our claim that we can avoid overfitting.

Reviewer comment (Experiments):
With stochastic gradient, it is possible that the initial points are different between experiments. Will different initial points affect the performance or the iteration times of the algorithm?

Author response:
We have made extensive tests regarding multiple runs. Since our method does not aim for the smallest loss when finding the initial point on the front (cf. our response to question 5), we observe only very small variations in terms of repeated experiments. We will add more details to this in the appendix section of the final paper which will be added in the final submission.

Reviewer comment (Reproducibility):
What is the method to decide when should the training stop before when the slope of the Pareto front becomes too steep. In practical settings, it seems to complex and tricky to achieve the early stopping.

Author response:
The reviewer raises a good point here. The focus of our paper was first to demonstrate that we can obtain the entire Pareto front even for extremely high-dimensional problems. The question of early stopping is of vital importance, though, and we will add a brief discussion in the appendix section of the final paper submission, where we argue that a small validation set should suffice to detect the turn-around point in terms of the out-of-sample loss.

Reviewer comment (Novelty):
One missing related reference: DessiLBI: Exploring Structural Sparsity on Deep Network via Differential Inclusion Paths. ICML2020

Author response:
Thank you for the reference. We have added it to our paper and included it in the discussion on related work.