Paper: POPULATION DESCENT: A NATURAL-SELECTION BASED HYPER-PARAMETER TUNING FRAMEWORK (ICLR-2024)

Reviewer comment (Writing):
The language used in this paper is not good. I recommend the author to use large language models (e.g. ChatGPT) to go through your work.

Author response:
Extra proof-reading is a priority, and we will make sure to address as much of the poor language in the paper as possible.

Reviewer comment (Presentation):
Too many tables, algorithms, and subjective comments in the paper. You should use more rigorous statements.

Author response:
Thank you for the suggestion. Currently, we are improving the clarity and conciseness of our tables. As for the point on subjective comments, we will try our best to replace them. If you could point out specific areas in need of more rigor, it would be very helpful to us.

Reviewer comment (Writing):
Too many irrelated sentences in the introduction. Everyone knows the property of global opt... Should make it more compact.

Author response:
We are currently rewriting the introduction to be more concise and better explain our contributions.

Reviewer comment (Experiments):
Fmnist and cifar10 are just too simple. Since it is a algorithm-like paper. Just doing such weak experiments are not enough.

Author response:
We are expanding our experiments to include additional dataset benchmarks such as the Penn Treebank, and comparing against other state-of-the-art methods, including Sklearn’s Hyperopt and IRACE. We do use other tests to demonstrate PopDescent’s flexibility to not be extremely problem-specific; in Experiment 3.4, we show that PopDescent achieves quality results without parameter tuning, demonstrating the effectiveness of its 'active' tuning compared to other methods.

Reviewer comment (Novelty):
The authors propose a new memetic algorithm, but fail to sufficiently discuss existing state-of-the-art memetic algorithms in their literature review. ESGD is briefly mentioned in the “Benchmarks” section, but its workings are not described or critically compared to the proposed approach. Evolutionary/population-based algorithms are plenty, and without a critical discussion of existing methods, it is hard to evaluate the originality of the proposed method.

Author response:
Comment: “The authors propose a new memetic algorithm, but fail to sufficiently discuss existing state-of-the-art memetic algorithms in their literature review.”
Yes, our description of memetic algorithms is not as in-depth as our discussion on Tuning Frameworks. However, our decision is based on applying a memetic framework to hyper-parameter tuning, making PopDescent unique. We were initially “motivated by the difficulties in using a memetic framework for fine-tuning hyper-parameters.” Thus, we compared PopDescent to ESGD, but the method more directly competes against other state-of-the-art tuning frameworks like KerasTuner. There aren’t many other memetic algorithms that are designed for this task. However, we will compare our results with more existing memetic algorithms and more tuning frameworks which should strengthen our claims.

Reviewer comment (Experiments):
In the experiments, all methods employ Adam except for ESGD. This seems like an unfair comparison: perhaps the superiority of the proposed method is due to the superior performance of Adam as compared to SGD. Adam is known to converge faster than SGD, which might also explain why the proposed method converged quicker than ESGD.

Author response:
Comment: “In the experiments, all methods employ Adam except for ESGD. This seems like an unfair comparison: perhaps the superiority of the proposed method is due to the superior performance of Adam as compared to SGD? Adam is known to converge faster than SGD, which might also explain why the proposed method converged quicker than ESGD.”
Thank you for pointing this out. Our statement in Section 3.1 “All algorithms use the Adam optimizer for local search, except for ESGD, which uses SGD”, is confusing. The original work makes use of SGD as its local optimizer. However, in our benchmark evaluations, we used the optimizer that ESGD performs best with: SGD for FMNIST, and Adam for CIFAR-10. This choice is apparent in our code, but we failed to mention it in the paper, our revision will reflect this choice. We tried to make the comparison with ESGD as fair as possible. If there are any more suggestions on how we could do so, we would appreciate them greatly. We will also add additional experiments bolstering our claims, such as adding the Penn Treebank dataset and comparing PopDescent against other state-of-the-art methods like Sklearn’s Hyperopt, and IRACE.

Reviewer comment (Experiments):
Authors use cross-validation error to perform genetic evolution of the hyperparameters. This raises the question of whether this will leak information about the test set. It is unclear if the final errors reported are calculated on some hold-out set that is not seen during the evolutionary process.

Author response:
Comment: “The final errors reported - are they calculated on some hold-out set that is not seen during the evolutionary process?”
All final errors reported (except for ESGD, which we use as instructed in the provided open-source implementation), are calculated on a hold-out set that is not seen during the evolutionary process. For all methods (except for ESGD), we split the test-set into a “cross-validation set” and “test-set.” The evolutionary processes rely on the “cross-validation set” and the final errors are calculated on the “test-set.”

In the paper, we describe it as follows: “These individuals are maximized on batches from a test distribution that remains unseen during the procedure. Since the Test distribution must remain unseen, we are forced to make use of available proxy data in the form of a Training distribution and a CrossValidation distribution.” We have revised our statement, renaming this unseen dataset as a “hold-out” set in the paper for clarity.

Reviewer comment (Writing):
Citations are not enclosed in parentheses — for example, “…Large Language Models Cheng et al. (2023)” should rather be “…Large Language Models (Cheng et al., 2023)”.

Author response:
Formatting issues and further proofreading rounds will be applied for the final revision before the rebuttal period ends.

Reviewer comment (Writing):
Acronyms: there is no need to capitalize every word that is going to be abbreviated. For example, instead of “Neural Networks (NNs)” one can simply write “neural networks (NNs)”.

Author response:
Formatting issues and further proofreading rounds will be applied for the final revision before the rebuttal period ends.

Reviewer comment (Writing):
“when the magnitude is at 0, None” should be “when the magnitude is at 0, none”.

Author response:
Formatting issues and further proofreading rounds will be applied for the final revision before the rebuttal period ends.

Reviewer comment (Writing):
“One iteration defines one local and global update together. gradient updates the algorithm takes before performing a mutation.” — the two sentences seem malformed.

Author response:
Formatting issues and further proofreading rounds will be applied for the final revision before the rebuttal period ends.

Reviewer comment (Presentation):
The problem formulation in this paper is rather unconventional. To enhance the overall coherence of the paper, I recommend commencing with an overview of Black-box optimization and leveraging the context of evolutionary algorithms to guide the logical progression.

Author response:
We appreciate this thoughtful suggestion. We will enhance the logical progression of the introduction by providing an overview of Black-box optimization and leveraging the context of evolutionary algorithms for PopDescent.

Reviewer comment (Novelty):
The novelty of the proposed method appears to be limited in comparison to existing approaches.

Author response:
We will reframe our contributions focusing on the novelty in designing a memetic algorithm specifically for hyper-parameter tuning. We will convey that the trade-offs we make for the design choices forming the whole algorithm are novel, rather than the individual pieces themselves. While features like using “normalized fitness”, using m-elitist selection, or controlling the “randomness in mutation” are not our creations, we combine them uniquely to introduce an algorithm that has not been created before. To the best of our knowledge, we are the only algorithm to use the mentioned features in a combination that yields a much simpler tuning process: in the paper, we mention how other state-of-the-art memetic algorithms’ “genetic schemes are complex and thus difficult to implement.” PopDescent does not need fine-tuning of any parameters, however, as shown by Experiment 3.4. Also, PopDescent is, to the best of our knowledge, the most “active” hyper-parameter tuning framework, meaning it makes the most adjustments during training, instead of beforehand. As mentioned, other frameworks only partially test hyper-parameters over the first few iterations and lack precision during later phases of training, or ignore efficiency and train all possible outcomes. Similarly, PopDescent improves upon commonly used hyper-parameter “schedules,” which do not adjust rates based on training progress. We appreciate the feedback on presenting novelty, and we are currently rewording our contributions to highlight how PopDescent is 1) a new memetic algorithm given its unique combination of features, 2) uniquely applied to hyper-parameter tuning, and 3) a more “active” tuner and thus better performing/more efficient than existing tuning frameworks.

Reviewer comment (Experiments):
The experimental section is notably inadequate in terms of datasets and compared methods. It is essential to incorporate a comparison with advanced optimizers, such as the Sharpness-aware optimizer, to provide a more comprehensive evaluation and gauge the effectiveness of the proposed method against state-of-the-art techniques.

Author response:
We are working to add both tests on more dataset benchmarks like the Penn Treebank dataset, and against other state-of-the-art methods like Sklearn’s Hyperopt, and IRACE. The Sharpness-aware optimizer is designed as a competitor to optimizers like SGD, Adam, or Shampoo. However, as we state in the paper’s related works, “PopDescent relies on efficient local optimizers.” PopDescent is more directly comparable to hyper-parameter tuning algorithms instead. The Sharpness-Aware optimizer can replace Adam in any of our tests, as our paper does not rely on which specific local optimizer is used, and tunes hyper-parameters regardless.