Paper: On the Hyperparameter Loss Landscapes of Machine Learning Algorithms (ICLR-2024)

Reviewer comment (Experiments):
The workflow is inefficient in practice because it assumes that the entire (discretized) configuration was evaluated. It would be much more relevant if it could be directly applied to the output of any HPO approach.

Author response:
We appreciate the reviewer’s concern on this. However, we do need state that the goal of our developed FLA workflow is NOT to be competitive in efficiency as many other tools that work collaboratively with HPO methods (e.g., DeepCAVE, Optuna). Rather, our main motivation is to gain deep and comprehensive insights into the problem landscapes, which does not necessarily to be time-constrained. These two lines of work are often parallel to each other in landscape analysis literature. Therefore, we do not consider this should be a weakness of our proposed method. 

However, we do agree with the reviewer that it would be desirable by the community if our tool is able to serve as a post-optimization explanation method for existing HPO frameworks. We think this would be a promising direction to further improve it and thus make the advancements of landscape analysis accessible to a wider range of stakeholders.

Reviewer comment (Experiments):
In view of the renowned paper by Bergstra and Bengio on random search and the low effective dimensionality of HPO spaces, I don’t understand why the authors decided to use a grid sampling strategy nevertheless and thus do not efficiently sample all dimensions. The distance function could be a reason for that, but defining a distance function in a non-discretized space is not uncommon.

Author response:
While we appreciate that grid search is less efficient from the HPO perspective, we contend that it is more favorable to landscape analysis. To be specific, the “low efficiency” of grid search as pointed out in the random search paper [4] mainly criticized its redundant sampling on configurations with highly repeated HP values. This is indeed not desirable when we wish to cover a more diverse set of configurations as in the HPO, or when we want to build up accurate models for the predicting model loss. 

However, for landscape analysis, such redundancy can actually be exploited to enable higher interpretability. For example, it allows our constructed landscapes to carry important information regarding “*how would the performance change if I only slightly modify the value of 1 HP*” without needing to use techniques like partial dependence plot, which are built upon surrogate models.

Reviewer comment (Experiments):
The benchmarks heavily focus on tree-based algorithms. CNN is more of an outlier and the main paper discusses it also only very briefly (if at all). I would have hoped for a more diverse set of ML algorithms.

Author response:
To address the reviewer's concern, we include the analysis of HP loss landscapes of a feed-forward neural network (FCNet) on 4 UCI datasets, using part of the HPOBench data (**Appendix D, revised manuscript**). We also include supplementary results on NAS-Bench-101 and 201 to demonstrate the potential impact of our developed FLA framework for analyzing a wider range of AutoML tasks (**Appendix C, revised manuscript**).

Reviewer comment (Experiments):
HPO benchmark libraries such as HPOBench or YAHPO already provide ample data (also for multi-fidelity). I wonder why the authors decided to collect new HPO data and do not validate their findings on these established benchmarks.

Author response:
While we appreciate the considerable efforts that have made in developing an extensive set of HPO benchmarks, as incorporated in HPOBench, we choose to collect new data since we want a more *diverse* and *representative* set of scenarios that would better demonstrate the generalizability of our findings. In particular:

- HPOBench considers 20 tabular datasets to evaluate tree-based models like Random Forest and XGBoost. We contend that the tabular benchmark with 57 datasets as used in this paper can lead to more statistical meaningful results. 
- We also consider the basic Decision Tree, which is the based model of tree ensembles, and LightGBM, which has been widely used in real-world engineering due to its faster speed.
- For neural networks, HPOBench includes various NAS benchmarks, which we consider a different line of work from HPO, and simple models like MLP, which is not representative for more complicated DNNs.  We thereby seek to collect new data on CNNs.

Reviewer comment (Evaluation):
I wonder why the authors decided to use accuracy instead of balanced accuracy. I haven’t checked all the datasets from this paper, but unbalanced datasets are fairly common and thus balanced accuracy (or similar metrics) would be the better choice.

Author response:
We report results on accuracy score since it is still the most widely used metric for classification. However, we do consider another metric for the evaluation of classifiers, ROC-AUC, to cope with potential dataset imbalance. Additionally, we had also done inspections on the 57 tabular data that are studied in this paper, and these datasets are actually all perfectly balanced between classes. We understand that the reviewer raises this problem since we only metioned accuracy in our submitted manuscript, and we have corrected this in our manuscript (**Section 4, page 5**).

Reviewer comment (Theory):
I’m not convinced that the no free lunch theorem plays a role here since, as the authors argue themselves, the characteristics of HPO landscapes seem to be very similar s.t. in fact a single algorithm could be the best one.

Author response:
We do not think there is contradictions here. While the main conclusion of this paper is indeed that the general family of HP loss landscapes of ML models share various high-level common patterns, we do note that the detailed landscape chracteristics still largely depend on specific model, search space, dataset, fidelity, etc. Therefore, in general, we can expect many methods to work relatively robustly across different tasks due to their high-level commonalities. However, there are still many differences between tasks that one may pay particular attention to, and thus make more informed selections on problem solvers. Portfolio approaches can leverage such information and lead to prominent results, while by no means should be considered useless according to our results.  

Based on this, we argue that the No-Free-Lunch theorem does have a role to play in HPO. However, concerning the fact that this paper focuses more on highlighting the similarities rather than nuances between HP loss landscapes, we remove the related expressoin from **Section 1 (page 1)** of the manuscript.

Reviewer comment (Reproducibility):
The source code and the data are not part of the submission. The authors write in their paper that they will release the data. However, from my experience, it is of utmost importance to provide all of this for the reviews in an anonymized way.

Author response:
Our group is highly committed to the reproducible research cultural. We will release the source code along with the dataset used and generated in this paper after the acceptance of this paper. To mitigate the reviewer’s concern on the generalization ability of our findings, we have provided additional experiments on NAS-Bench-101 and 201 (**Appendix C**).

Reviewer comment (Experiments):
Portfolio approaches with complementary configurations (e.g., auto-sklearn 2.0 by Feurer at al.) are very famous and are constructed such that the configurations in the portfolio perform well on different datasets. If your conclusion is now that the area of well-performing configurations is consistent across datasets, I would also conclude that portfolio approaches should not be beneficial at all. One problem could be that the considered ML algorithms have this property, which might not be true for others; if that were true, it would partially invalidate your results. So, please comment on this contradiction between your results and previous results.

Author response:
We do not think there is contradictions here. While the main conclusion of this paper is indeed that the general family of HP loss landscapes of ML models share various high-level common patterns, we do note that the detailed landscape chracteristics still largely depend on specific model, search space, dataset, fidelity, etc. Therefore, in general, we can expect many methods to work relatively robustly across different tasks due to their high-level commonalities. However, there are still many differences between tasks that one may pay particular attention to, and thus make more informed selections on problem solvers. Portfolio approaches can leverage such information and lead to prominent results, while by no means should be considered useless according to our results.  

Based on this, we argue that the No-Free-Lunch theorem does have a role to play in HPO. However, concerning the fact that this paper focuses more on highlighting the similarities rather than nuances between HP loss landscapes, we remove the related expressoin from **Section 1 (page 1)** of the manuscript.

Reviewer comment (Presentation):
What is a unit of the color coding in Figure 1? If these are accuracy benchmarks, the values cannot go up to 6000 and beyond.

Author response:
The unit in all landscape visualizations are the ranks of the performance. Therefore, for the CNN result, the range will be $[0, 6479]$ as its search grid contains $6480$ configurations. We report this instead of raw metric values because model losses/accuracies can be on very different scales under different scenarios (e.g., across datasets). We thank the reviewer for raising this problem, and we have updated all the related figures with explicit colorbars & captions to indicate this. We additionally note that for ease of interpretability, for both minimization (e.g., loss) and maximization (e.g., accuracy) cases, we calculate the ranks in the way such that lower rank values (e.g., 1st, 2nd, 3rd...) are better.

Reviewer comment (Theory):
Why is the search space encoded as a ***directed*** graph? The neighborhood relationship goes in both directions.

Author response:
The edge directions and weights allows our constructed graph to model the propagation of $\Delta \mathcal{L}$ in the landscape, which can be exploited by graph mining techniques. For instance, PageRank centrality for each nodes typically show strong correlation ($0.5$ to $0.9$ for most cases) with configuration performance, despite it is calculated purely based on connectivity patterns of the network. Although we did not dive deep into this in the current paper, this already implies that the directed graph model of fitness landscapes can capture certain dynamic features of the landscape, and could be further exploited in the future. 

In addition, using this model, code-level implementation of landscape analysis methods can become more straight-forward. For example, since the edges in the graph always indicate the direction of improvement, first-improvement local search starting from a given node can be easily implemented by continuously following a random outgoing edge until a *sink* node (i.e., node with zero out-degree, local optimum) is encountered. Best-improvement local search, on the other hand, is almost the same, except that we need to select the outgoing edge with the largest improvement at each move.

Reviewer comment (Theory):
UMAP has several assumptions on the underlying space. Do all these assumptions hold true in this case?

Author response:
There are three most important assumptions for UMAP, and are expected to hold in our case, at least with a good approximation. Here we discuss them in more detail:

- *Data are uniformly distributed on Riemannian manifold*. As the distribution of HOPE embeddings essentially depends on the connectivity pattern of the graph, for HP loss landscapes, we can expect this assumption to hold. This is because the number of neighbors for each node in the graph would largely stay the same based on our neighborhood definition. Therefore, there will be no significance variation of density across the graph, and the HOPE embeddings in turn should approximately have a uniform distribution.
- *The Riemannian metric is locally constant.* The distances between HOPE embeddings correlate directly with the local connectivity patterns within the network. Given the stability in neighborhood structures as we discussed above, it is reasonable to presume that the local distance metrics would remain approximately constant in small regions.
- *The manifold is locally connected.* HOPE embeddings can actually preserve more than just the local structure between configurations, as it is able to capture high-order proximities in the graph. We thereby expect this assumption will also hold.

Note that since HOPE is able to preserve asymmetric transitivity patterns in directed graphs, the first two assumptions may not be fully satisfied. However, considering the fact that the variations of in- and out-degrees of each node is limited by its total degree, which is almost identical across the landscape, and the fact that UMAP is designed to be able to operate robustly even when these assumptions only partially hold true, we think the application of UMAP here does not carry much risk. 

We have updated our manuscript with these discussions on UMAP in **Appendix B**.

Reviewer comment (Experiments):
As you wrote, low effective dimensionality could have an impact on your evaluation. How would your results change if you first remove all unimportant hyperparameters?

Author response:
As we metioned in the paper (**Section 4 on page 7**), we have already been selective in the HPs to study according to their importance, and we introduce our methodology in more detail in **Appendix F of the revised manuscript**. To be brief, all the HPs considered in our study have certain impact on model performance and complexity, and are among the commonly-used ones in literature and practical scenarios. We contend that incorporating only $2$ to $3$ most ones can lead to landscapes that fail to capture long-range HP interactions. Therefore, our studied scenarios typically involve $5$ to $8$ HPs for each model. 

Despite these, if we were to consider less HPs (leaving others to deafult values) with higher importance:
- Neutrality of the landscape will decrease, as the remaining HPs are more likely to cause large performance shifts. 
- The number of local optima (if any) can also reduce due to the decrease in complex HP interactions (e.g., for XGBoost on dataset #45035, the number of local optima dropped from $64$ to $4$).
- For NDC metrics, we do not observe statistically meaningful changes across our studied scenarios. The lower-dimensional landscapes still tend to be planar around the optimum.
- However, for assortativity and autocorrelation, we observe a decrease in them for various landscapes when only studying the learning rate and maximum depth for XGBoost (e.g., for XGBoost on #45035, $ass: 0.7872 \to 0.7247, \rho_a: 0.6292 \to 0.4862 $). We postulate that one of the reasons for this is the fewer neutral regions in the landscape, which can form smooth plateaus and thus make the landscape more structured.

Reviewer comment (Novelty):
At the AutoML conference 2023, there was a paper on studying HPO landscapes for AutoRL by Mohan et al. How does your paper relate to that?

Author response:
This paper [1] investigates the RL hyperparameter landscapes, which is very different from the ML scenarios address in our work due to the dynamic nature of RL. In addition, it applies surrogate models based on RBF and GP to construct surrogate models of the fitness landscape, which has been a promising direction for studying expensive optimization problems (e.g., [2,3]).  This paper itself only considers aspects like the modality of the landscape, but as there have been increasing attention for studying HPs in RL (e.g., another paper in *ICML'23*, [4]), we believe it is a meaningful start to call for more in-depth landscape analysis on RL systems in future works. 

[1] Aditya Mohan *et al.*, “AutoRL Hyperparameter Landscapes”, *AutoML’23*, 2023.

[2] Philip A. Romero *et al.*, “Navigating the protein fitness landscape with Gaussian processes”, *Proc. Natl. Acad. Sci. (PNAS)*, 2013.

[3] Caroline J. Watson *et al.*, “The evolutionary dynamics and fitness landscape of clonal hematopoiesis”, *Nature*, 2020.

[4] Theresa Eimer *et al.*, “Hyperparameters in Reinforcement Learning and How To Tune Them”, *ICML’23*, 2023.

Reviewer comment (Experiments):
The study focuses on simple models such as random forest and support vector machines, while there is only a neural network architecture type involved (CNN). Given the relevance of neural networks in the current world, this study should include more detailed comparisons on this matter.

Author response:
**(W1): The paper focuses on simple models like RF and SVM, while more results on neural networks are desired.**

We see there might be some misundertanding here, since our paper does not consider SVM. The submitted manuscript studies 5 ML models, including *i)*: Random Forest (RF), *ii)*: XGBoost, *iii)*: LightGBM, *iv)*:Deicision Tree, and *v)*: CNN.

However, we agree with the reviwer that the community may be more desired to see results on neural networks. To this end, we have updated our manuscript with additional experiments on analyzing a feed-forward neural network setup from HPOBench using $4$ UCI datasets in **Section 4**. In addition, to demonstrate the potential of our proposed framework for analyzing more complicated DNN landscapes, we also conduct analysis on NAS-Bench-101 and 201 (**Appendix C**).

Reviewer comment (Experiments):
Another common scenario is to create a broad space where both RF and SVM (algorithm and hyperparameter selection) are included. It is a missed opportunity to discuss how the loss structure is affected when optimizing both HP and algorithms.

Author response:
**(W2): It is a missed opportunity to discuss how the loss structure is affected when optimizing both HP and algorithms.**

Our proposed FLA framework can be easily adopted to include AutoML pipelines like model selection and pre-processing, just as it is able to analyze NAS landscapes by adjusting the neighborhood definition. However, as the main focus of this paper is on HPO, which is an well-established subfield of AutoML, such analyses is beyond its scope, and we leave this to future works.

Reviewer comment (Novelty):
There should be a stronger discussion on the impact of the findings. From my perspective, although the findings are interesting, they are not surprising. Aspects like the correlation of the loss among fidelities and datasets are well-known and they are exploited by common HPO methods. In other words, it is not clear how the findings in this work can improve the current methods for optimizing HPs.

Author response:
**(W3): There should be a stronger discussion on the impact of the findings, as studied aspects like multi-fidelity and transfer learning have been widely used in the community.**

Despite transfer learning and multi-fidelity methods have been extensively used for HPO, they are mainly developed based on rough intuitions and their effectiveness are usually supported by algorithmic performance. To date, there is a lack of fundamental understanding regarding the validity of the underpinned assumptions made by these methods (see **Section 1, page 3, revised manuscript**). This has also pointed out by **Reviewer NkWq** in *Strengths*.

This work fills this gap by providing a *problem-level* illustration on the success these methods are effective. In particular, we show that:
- Multi-fidelity methods are reliable because the HP loss landscapes with lower fidelities greatly resemble high-fidelity landscapes w.r.t. both topological structure and performance ranks.
- Transfer learning methods are effective since HP loss landscapes induced on diverse datasets also share considerable commonalities.
- These findings are highly generalizable to a wide spectrum of tasks.

Also, for the first time, we revealed an unified portrait of HP loss landscapes of ML models, which is quite benign. Such properties can be leveraged by more tailored algorithms to more efficiently traverse the landscape. 

In addition, the FLA framework we developed to enable all the analyses in our study, can also be applied to advance the understanding of a wider range of AutoML tasks, e.g., NAS (see analyses on NAS-Bench-101 and 201 in **Appendix C, revised manuscript)**.

Reviewer comment (Experiments):
How does the loss landscape change with the dataset size? Is there any available insight from this?

Author response:
**(Q1): How does the loss landscape change with the dataset size?**

We thank the reviewer to raise this question as we also find this is an interesting point that worth further exploration. Through amended experiments on the $57$ tabular datasets (**Appendix E, revised manuscript**), we find that with a weak correlation:

- For all four tree-based models, the loss landscapes tend to be more similar on dataset with similar sizes.
- In general, all $4$ tree-based models are more likely to overfit on datasets with more instances and features.

However, here, we only applied most basic datasets meta-features like the number of instances and columns, while there are many more to explore (e.g., [1]). We think this should be investigated in more depth in future work.

[1] Matthias Feurer *et al.*, “Initializing Bayesian Hyperparameter Optimization via Meta-Learning”, *AAAI’15*, 2015.

Reviewer comment (Evaluation):
How does the loss landscape look when I consider only HPs that affect the expressiveness/complexity of the model? My intuition is that these HPs built up a convex search space, with not local optima. Is this somehow supported in this study?

Author response:
**(Q2): How does the loss landscape look when I consider only HPs that affect the expressiveness/complexity of the model?**

As we metioned in the paper (**Section 4 on page 7**), we have already been selective in the HPs to study according to their importance, and we introduce our methodology in more detail in **Appendix F of the revised manuscript**. To be brief, all the HPs considered in our study have certain impact on model performance and complexity, and are among the commonly-used ones in literature and practical scenarios. We contend that incorporating only 2 to 3 most ones can lead to landscapes that fail to capture long-range HP interactions. Therefore, our studied scenarios typically involve 5 to 8 HPs for each model. 

Despite these, if we were to consider less HPs (leaving others to deafult values) with higher importance:
- Neutrality of the landscape will decrease, as the remaining HPs are more likely to cause large performance shifts. 
- The number of local optima (if any) can also reduce due to the decrease in complex HP interactions (e.g., for XGBoost on dataset #45035, the number of local optima dropped from 64 to 4). However, with the pressence of 4 local optima, this still somehow deviates from the reviewer's hypothesis that with less HPs, the landscape can be unimodal. We note that this is because the pressence of HP interaction (also known as epistasis in landscape literature). This leads to the fact that even if two HPs reveal exactly unimodal response, their combined effect can be multimodal due to their interactions. Nevertheless, as we see from the paper, most HP loss landscapes seem quite benign in terms of modality. 
- For NDC metrics, we do not observe statistically meaningful changes across our studied scenarios. The lower-dimensional landscapes still tend to be planar around the optimum.
- However, for assortativity and autocorrelation, we observe a decrease in them for various landscapes when only studying the learning rate and maximum depth for XGBoost (e.g., for XGBoost on #45035, $ass: 0.7872 \to 0.7247, \rho_a: 0.6292 \to 0.4862 $). We postulate that one of the reasons for this is the fewer neutral regions in the landscape, which can form smooth plateaus and thus make the landscape more structured.

Reviewer comment (Experiments):
Page 4 / Landscape Visualization: In the last sentence, the authors state that they apply a linear interpolation to generate a smooth surface. At the same time, they argue at the end of page 5 that the loss landscapes shown in Figure 3 (a) are “relatively smooth”. It is unclear whether this is an effect of the smoothing applied or would be the case anyway. Considering that the FLA metrics concerning smoothness show similar results, it will probably also apply without the smoothing step. Nevertheless, this seems a little questionable to me.

Author response:
**(W2): It is unclear whether the smooth landscape as shown in visualization is an effect of linear interpolation.**

We understand the reviewer’s concern. We provide an updated diagram showing the workflow of our proposed visualization method in **Figure 7 (Appendix B)**, in which a scatter plot of the data points before interpolation is provided. It is clear to see that the smoothness conclusion does not directly hinges to the linear interpolation. We do this step only to refine the interpretability of the plots.

Reviewer comment (Experiments):
Page 7 / Highly neutral; planar around the optimum: The authors state that they have been “selective in the search space design”, which is not further explained. They do list the search spaces in the appendix, but considering this comment, it seems that the authors took great care in choosing these search spaces which naturally greatly influences the results presented in the paper. Thus, the authors should comment on their approach for choosing the search space and on the consequences such as potential limitations of their analysis. They should also relate this to the setups of related analyses they cite.

Author response:
**(W3): The authors should comment on their approach for choosing the search space and on the consequences such as potential limitations of their analysis. They should also relate this to the setups of related analyses they cite.**

We have updated our manuscript with details about the routines we take in search space design in **Appendix F**. In general, we design these search spaces to combine both knowledge from existing benchmarks/literature and empirical experience. However, in addition to HP importance, as we only have limited computational budget (though in total we spent tens of thousands of GPU/CPU hours), we cannot include all the HPs that have an observable impact on performance. On the other side, we also do not want our search spaces to be too simple, for example, a search space containing only 2 HPs can be more likely to reveal a unimodal structure due to less high-order HP interactions. Bearing these considerations in mind, our designed search spaces eventually contain 5 to 8 HPs for each model, spanning 6,480 to 24,200 configurations in total. 

In other related works, Pushak and Hoos [1] also considers XGBoost, where they evaluated a total of $413,343$ configurations for 11 HPs, but only on a single dataset. They also analyzed the same FCNet benchmark data as in our amended experiments (**Appendix D**). Rodrigues *et al*. [2] analyzed 5 HPs of CNN using merely $800$ configurations in total. Teixeira and Pappa [3], and also Pimenta et al. [4] studied AutoML search spaces, which are generated using derivation trees from a proposed context-free grammar, and are not directly comparable to HP search spaces. Schneider et al. [5] compared the landscapes of XGBoost and classic black-box optimization problems using XGBoost search spaces with $\{2,3,5\}$ HPs, using mostly logarithmic-valued grids.  


While we believe our designed spaces are representative of real-world scenarios, and are not manually biased towards certain conclusions, we do note that our FLA metrics can be affected by the selection of HPs and their grid spacing. 

For example, for the neutrality metric, using an extremely dense grid for HPs, e.g., $lr=[0.1, 0.10001, 0.10002, 0.10003, \dots]$ can lead to landscapes with extremely high neutrality,  whereas considering only the most important 1 or 2 HPs can shape landscapes with little neutrality. Landscape properties as measured by other metrics can also slightly vary with the search space design, and for detailed discussion on this, we kindly refer the reviewer to our response to **reviewer IUHx** (**Part 2, Q2**). Considering the extensive set of scenarios that we have studied, and the amended  experiments on existing benchmarks (HPOBench and NAS Bench-101/201, **Appendix C and D**), our result should be generalizable to the majority of practical cases.

[1] Yasha Pushak and Holger Hoos, “AutoML loss landscapes”, *ACM Trans. Evol. Learn.*, 2022. 

[2] Nuno M. Rodrigues, “Fitness landscape analysis of convolutional neural network architectures for image classification”, *Info. Sci.*, 2022.

[3] Matheus C. Teixeira and Gisele L. Pappa, “Understanding AutoML Search Spaces with Local Optima Networks”, *GECCO’22*, 2022.

[4] Cristiano G. Pimenta *et al.*, “Fitness Landscape Analysis of Automated Machine Learning Search Spaces” , *EvoCOP’20*, 2020.

[5] Lennart Schneider *et al.*, “HPO × ELA: Investigating Hyperparameter Optimization Landscapes by Means of Exploratory Landscape Analysis”, *PPSN’22*, 2022.

Reviewer comment (Reproducibility):
Limitations - Lack of elaboration on the limitations of the authors’ analysis. For example:  
* Can conditional dependencies between hyperparameters be modeled in the framework?  
* Do the embeddings account for the fact that for numerical hyperparameters (in principle) values between those given in the search grid can be chosen, but for categorical hyperparameters not? This is important as the analysis otherwise inherently assumes all hyperparameters to be categorical, (at least to a large extent).  
* What hyperparameters for UMAP were set? This greatly influences the outcome of the analysis (see e.g. https://pair-code.github.io/understanding-umap/), but the concrete hyperparameters or how they were chosen are never mentioned.

Author response:
**(W4): In general I miss an elaboration on the limitations of the authors’ analysis.**

We thank the reviewer to raise this problem. We have updated the final section of the body of the paper to include some limitation of our study. Here we also response to the 3 points that the reviewer specifically mentioned:

- *Can conditional dependencies between hyperparameters be modeled in the framework?* Yes. Conditional dependencies can be incorporated into landscapes by defining distances using grammar derivation tree (see, e.g., [1]). Actually, our framework is designed to be able to operate on the majority (if not all) of AutoML scenarios by properly defining a neighborhood structure. We show such an example on NAS-Bench-101 and 201 in the updated manuscript (**Appendix C**). 
- *Do the embeddings account for the fact that for numerical hyperparameters (in principle) values between those given in the search grid can be chosen, but for categorical hyperparameters not?*: Yes, HOPE generates embeddings of nodes based on the local connectivity of them. For numerical HPs, if intermediate values are added to the search space, the resulting new configurations would be closely connected to its neighbors that previous exist in the landscape. The generated embeddings would thus also stay between the existing configurations in the landscape in the embedding space, since HOPE is able to preserve local structure of the network. 
-  *What hyperparameters for UMAP were set?*: While the exact setting of UMAP should depend on the size of the landscape (e.g., total number of configurations), we have added a remark to provide some general guidelines for UMAP setup for landscape visualization purpose in **Appendix B**

Reviewer comment (Reproducibility):
It is very unfortunate that no source code is provided since this makes the claim of the authors in the conclusion that their framework could be used to analyze other benchmarks a bit meaningless.

Author response:
**(Additional Comment): Source code availability**

Our group is highly committed to the reproducible research cultural. We will release the source code along with the dataset used and generated in this paper after the acceptance of this paper. To mitigate the reviewer’s concern on the generalization ability of our findings, we have provided additional experiments on NAS-Bench-101 and 201 (**Appendix C**).

Reviewer comment (Novelty):
The authors could highlight more strongly that other papers have also performed dimensionality reduction techniques to show how loss landscapes look. While the approach of using HOPE + UMAP has its pros and novelty, easier approaches have been used before (see e.g. Q2 on page 5 in Sass et al., 2022).

Author response:
**(Additional Comment): The authors could highlight a little bit more that also other papers performed dimensionality reduction techniques.**

We thank the reviwer for this advice. We have added comments on works that directly applies dimensionality reduction techniques in the **Page 2 of Section 1**. To be brief, a direct use of dimensionality reduction methods can cannot well-preserve local connectivity pattern in the landscape, and would thus lead to results that are less like a “topographic map”. In addition to the reference that the reviwer provides, we also mention other works in this direction, e.g., [1, 2].

Reviewer comment (Novelty):
This work provides empirical evidence that many HPO practitioners already use, which limits the work’s impact and significance. For example, the finding that HP landscapes are highly similar across datasets is the basis for hyperparameter transfer learning. The similarity across fidelity verifies again that the assumption of multi-fidelity HPO methods.

Author response:
**(W1) This work focuses on aspects that many HPO practitioners already use (e.g., multi-fidelity, transfer learning)**

Despite transfer learning and multi-fidelity methods have been extensively used for HPO, they are mainly developed based on rough *intuitions* and their effectiveness are usually supported by algorithmic performance. To date, there is a lack of fundamental understanding regarding the validity of the underpinned assumptions made by these methods. This has also pointed out by **Reviewer NkWq** in Strengths. 

Our work fills this gap by providing a *problem-level* illustration on the reason that these methods are effective. In particular, we show that:
- Multi-fidelity methods are reliable because the HP loss landscapes with lower fidelities greatly resemble high-fidelity landscapes w.r.t. both topological structure and performance ranks.
- Transfer learning methods are effective since HP loss landscapes induced on diverse datasets also share considerable commonalities.
- These findings are highly generalizable to a wide spectrum of tasks.

Also, for the first time, we revealed an unified portrait of HP loss landscapes of ML models, which is quite benign. Such properties can be leveraged by more tailored algorithms to more efficiently traverse the landscape. 

In addition, the FLA framework we developed to enable all the analyses in our study, can also be applied to advance the understanding of a wider range of AutoML tasks, e.g., NAS (see analyses on NAS-Bench-101 and 201 in **Appendix C, revised manuscript)**.

Reviewer comment (Novelty):
What’s the motivation of the proposed method in Section 2? What’s the pros and cons compared to other related works? Why do the authors think the proposed method is better than the others such as Pushak & Hoos (2022)? Like mentioned in the section of “Nearly unimodal”, the conclusion may indeed differ based on what analysis method is used. If using the method from Pushak & Hoos (2022) in the same setting as in this paper, would the authors have the same conclusion? This may be a good experiment to cross verify the findings.

Author response:
**(Q1) What is the motivation, as well as pros & cons of the proposed landscape analysis method?**

From the highest level (see updated conceptual diagram in **Figure 7, revised manuscript**), our proposed methods are motivated by the fact that prior works only used an limited set of FLA metrics to assess partial HP loss landscape characteristics, and could not provide intuitive visualization of landscape topography, or to interrogate the similarities between different landscapes. These gaps are also mentioned in **Section 1 on page 2**. Consequently, they fail to provide an unified, comprehensive understanding of HPO search spaces. 

Our proposed method fills these gaps by (also see **Section 1 on page 2**):
1. Developing a novel landscape visualization method that can preserve both local and global landscape structure.
2. Synthesizing a diverse set of FLA metrics to characterize HP loss landscapes from several essential, complementary perspectives. 
3. Applying 3 dedicated landscape similarity measures tailored for HPO context (by leveraging configurations ranks) to enable informative comparison between landscapes. 

Next, we address the reviewer's concern by providing a more detailed discussion on their pros and cons:
1. *Landscape visualization:* Our proposed visualizatoin method is able to handle high-dimensionality landscapes while incorporating neighborhood information, and thus allows us to provide a highly interpretable illustration of landscape topography that none of the priors works is able to. However, we do note that we need to be cautious when interpreting the result, as any 2D visualization of high-dimensional data can inevitably loss certain details. This paper ensures the correctness of the observed patterns using FLA metrics that are directly calculated on the original HP space. 
2. *FLA metrics:* Our applied metrics together enables the first unified portrait of HP loss landscape topography. However, while we consider this feature set sufficient for the purpose of this work, it could be further enriched in the future with metrics characterizing more detailed properties of the landscape.
3. *Landscape similarity measure:* Our method enables the first comparison between different HP loss landscapes, and the results are directly concerned by HPO solvers. However, such methods require the landscapes under inspection to have exactly the same set of configurations. In future work, this could be addressed by using a portfolio of landscape features and graph-level embeddings, which can be calculated on landscapes on different scales.

Finally, for the modality part that the reviewer had specifically mentioned, we note that this is rather a factual problem instead of method-dependent one, as any FLA methods should only faithfully reflect the true underlying structure of the landscape. Therefore, our assessment of landscape modality is in essence the same as in Pushak & Hoos (2022). In fact, for the amended experiments on FCNet (**Appendix E, revised manuscript**), we observe very similar results as in their. However, we also note that other aspects that we studied, e.g., basin of attraction, neutrality, smothness, are not considered in their work.

Reviewer comment (Reproducibility):
I may have missed this. What is the space for quantifying the landscape characteristics? The low dimension embedding based on HOPE method, or the 2D after UMAP?

Author response:
**(Q2) What is the space for quantifying the landscape characteristics?**

All the FLA metrics we calculated in this study directly operate on the *original* high-dimensional landscape. The HOPE embeddings and UMAP method are only used for generating landscape visualizations (see an updated conceptual workflow with more details in **Figure 7, revised manuscript)**.

Reviewer comment (Evaluation):
If many observations in Section 4.1 are based on the 2D landscape, then my following question will be relevant. In Figure 3, the landscape looks indeed smooth. But is it a result of using graph embeddings and UMAP, which smooth the landscape? Can it be different in the original higher dimensional space? Also can other properties of the landscape also be due to the usage of graph embeddings and UMAP? I don’t know much about high dimensional statistics and if there are some methods to quantify characteristics in the original HP space, that would be great. Otherwise, successful application of the observations derived from the paper to some HPO application could also indirectly verify the findings are relevant. For example, if the optimal tends to have a plateau, then the local search step in BO should not bring much improvement.

Author response:
**(Q3) Is the smooth landscapes as can observed in the visualizations caused by the use of UMAP or HOPE?**

We verified all the findings observed from landscape visualizations using a set of dedicated FLA metrics (**Table 1, page 4**). Importantly, these metrics are exactly developed to operate on the original high-dimensional space, and do not rely on embeddings in low-dimensional space. Therefore, a good match between qualitative and quantitative results as in this paper would imply that our landscape visualizations can faithfully preserve patterns in the high-dimensional landscape, and our findings are thus not a result of HOPE, UMAP or interpolation methods. 

For example, to verify that there is a flat plateau around the global optimum, we calculate NDC metrics for our studied landscapes. High NDC (**Figure 3 (c), page 6**) values would imply that as we move closer to the global optimum, we are more likely to encounter local search moves with subtle improvements. Beyond our study, experiments on HPO methods in many existing literature also have observed diminishing improvement near the best-found configuration (e.g., [1,2]).

[1] Jasper Snoek *et al.*, “Practical Bayesian Optimization of Machine Learning Algorithms”, *NIPS’12*, 2012.

[2] Stefan Falkner *et al.*, “BOHB: Robust and Efficient Hyperparameter Optimization at Scale”, *ICML’18*, 2018.

Reviewer comment (Presentation):
Since the author mentioned “However, when zooming into the top-10% regions, we find that the majority of our studied scenarios reveal low γ-set similarities.”, I think this should be highlighted in the paper because the prominent regions are the areas that people care about.

Author response:
**(Q4) The authors should highlight the results on the top-10% region of the landscape**

We thank the reviewer for this helpful suggestion. We have updated the related expression in our revised manuscript.