From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: The paper proposes a new hyperparameter optimization method using tensor completion. The new algorithm consider a N dimensional tensor with its dimension representing tunable parameters and values representing feedbacks (e.g. validation loss). A Tucker decomposition is used.

Strengths:
- Using tensor completion to improve hyper parameter optimization is a new and innovative idea

Weaknesses:
- The low-rank assumption is not well justified
- Not all tensor completion algorithm can converge
- The numerical results seem weak
- The paper is not ready yet in terms of clarity and quality 

All reviewers agree that, while the paper suggests an interesting new idea, itis not ready for publication yet.
justification_for_why_not_higher_score: All reviewers agree that the paper is a clear reject.
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper3949/Authors
title: Any concerns before the discussion deadline we have not addressed 
comment: Thank you again for your helpful comments. As the discussion period is soon coming to an end, we would like to enquire whether there are any remaining concerns. If so, we would be happy to address them.

From: ICLR.cc/2023/Conference/Paper3949/Authors
title: Any concerns before the discussion deadline we have not addressed 
comment: Thank you again for your helpful comments. As the discussion period is soon coming to an end, we would like to enquire whether there are any remaining concerns. If so, we would be happy to address them.

From: ICLR.cc/2023/Conference/Paper3949/Authors
title: Any concerns before the discussion deadline we have not addressed
comment: Thank you again for your helpful comments. As the discussion period is soon coming to an end, we would like to enquire whether there are any remaining concerns. If so, we would be happy to address them.

From: ICLR.cc/2023/Conference/Paper3949/Authors
title: Any concerns before the discussion deadline we have not addressed
comment: Thank you again for your helpful comments. As the discussion period is soon coming to an end, we would like to enquire whether there are any remaining concerns. If so, we would be happy to address them.

From: ICLR.cc/2023/Conference/Paper3949/Authors
title: Response to Reviewer C2WJ part 3 
comment: -If the tensor decomposition used in the paper is “noisy tensor completion with Cross measure- ments (Algorithm 2, Zhang (2019))”, then I would expect the authors to explain at least the core ideas.

We have included additional details on the algorithm, which are important to our hyperparameter optimisation technique, in Section 3. We avoided replicating information for the sake of space, as all details can be found in the original paper.

-Is it a hard requirement for the uniformness of the integer and real hyperparameters (Section 4.1)?

For our Python implementation, yes, because we have so far only devised ways to represent uniformly distributed hyperparameters in the tensor. We have not yet devised a strategy to represent hyperparameters whose values follow a skewed distribution, unless this distribution can be converted somehow into a uniform one e.g. a 2^4, 2^5, 2^6 can be represented as the exponents 4,5,6. We would need to devise the strategy and verify through extensive experimentation that it works – this is not possible in the timeframe of submitting the paper, but is certainly an area of future work.

-After reading Section 4.3, I understand why the grid search is used in Algorithm 1. It is because when the search space is small enough, one could afford a grid search and not miss any important HPs. But it will be very helpful to explain that while describing the algorithm, or at least mention that it will be explained later on.

We have now included this in the description of the algorithm in the first paragraph of Section 5.

-The language needs to be improved for correctness and smoothness.

We have rewritten large parts of the manuscript in an effort to improve the language quality and clarity.

From: ICLR.cc/2023/Conference/Paper3949/Authors
title: Response to Reviewer C2WJ part 2 
comment: -In Section 5.2, the author wrote “For HOTC, T was achieved by varying the number of tensor completion cycles, resolution of the initial search space, maximum number of elements for grid-search and minimum resolution for real-valued hyperparameters.” This is problematic because the search spaces (min, max, resolution) should be exactly the same for all the methods.

In our experiments, min and max in the search spaces were kept the same across all experiments. For all techniques except HOTC, the resolution of integer search spaces was 1 and real-valued search spaces are continuous. For HOTC, a resolution interval has to always be defined, especially for real-valued hyperparameters, so that a tensor (which has discrete elements) can be constructed based on the search space. Due to the nature of the algorithm, if we define the initial search space resolution for an integer hyperparameter  in HOTC to be, say, 10, this resolution interval would decrease to 1 (same resolution for integer hyperparameter as other methods) in 4 tensor completion cycles due to search space narrowing performed by the algorithm. Therefore, HOTC is able to explore integer values at the same level of granularity as other techniques. A limitation of HOTC is that it cannot search a continuous range of a real value, it has to discretise it. The best that can be done in this case is to keep min and max the same.

-The narrowing down search space part to me seems a bit arbitrary. Do you have experiment results without it? I would highly recommend taking this component out and comparing the proposed algorithm to other methods.

This comparison is now possible for a reader to see in the paper. Firstly, we have added experiments in which we evaluate the accuracy of tensor completion on tensors generated from validation losses for different hyperparameter combinations. The search space narrowing is performed based on the results of this completion, therefore the experiments can provide some insight into what does on “behind the scenes” of the arbitrary narrowing down.
Furthermore, we have included graphs of the vest validation loss obtained over time for HOTC at different completion cycles. This is compared to the best validation losses obtained from the other hyperparameter optimisation techniques at each trial; refer to figure 2 of the revised paper. The first point on the HOTC graph corresponds to the end of the first completion cycle. Search space narrowing only occurs after the first completion cycle, thus the first point in the graph can be considered as the result of operating our algorithm without the search space narrowing. In some cases, even this first completion cycle can surpass the other hyperparameter optimisation techniques.
It is true that the heuristic used to narrow the search space is arbitrary. However, our choice of heuristic was based on our desire to implement a simple algorithm – halving the size of the search space around the optimal hyperparameter value seemed like a heuristic that was simple to implement and explain. It does not appear to us that the narrowing down is contrived to generate good performance in only the machine learning problems that we present, firstly because we have demonstrated that the tensor completion is accurate and secondly we performed two new experiments (the CNN and VGG) since the first version of the paper and HOTC was still able to perform strongly.

-It can be hard to know the low rank number “T” in advance. Did you try some other values in the experiments and some tip on how to set T?

We have now included experiments in which we try alternative values of T, and have provided a theoretical explanation as to why we recommend setting the Tucker rank to 1 for every dimension. There are many reasons for this, namely fast performance, reproducible tensor completion (i.e. the results do not vary if the sampled elements do not vary), accurate modelling of the tensor (as verified by experiments), the fact that the Tucker core does not scale as more dimensions are added etc. In the paper, we have made it clear that setting the Tucker rank to [1,1...1] is our recommendation.

From: ICLR.cc/2023/Conference/Paper3949/Authors
title: Response to Reviewer C2WJ part 1 
comment: -Why is the low rank assumption true? Any motivating examples? For example, on some small scale problem, evaluate all the HP configurations and show that the low rank approximation has good accuracy.

Please see general comments.

-How good is the predicted accuracy based on tensor decomposition, giving more and more hyperparameter evaluations? Tensor completion with only a few entries could not end up with a good prediction. How does it compare to GP that is used in BO? This is not studied but critical for a convincing story in HPO.

To sample Tensor elements, we follow the heuristic of the Cross technique (Zhang 2019) for efficient tensor completion, which specifically dictates which elements of the have to be sampled to estimate the complete tensor. The authors of (Zhang 2019) have proven that this heuristic samples the minimum number of tensor elements required generate the complete tensor, based on the Tucker rank assumed for this tensor. Thus, if we assume that the tensor has a particular Tucker rank, the number of elements we need to sample is deterministic. It is in fact important to sample as few tensor elements i.e. evaluate as few hyperparameter combinations as possible because each hyperparameter evaluation takes time. If we evaluate too many hyperparameter combinations, HOTC would not be competitive with state-of-the-art techniques such as BOHB, as we have demonstrated it to be. Furthermore, in the revised version, we have provided experiments where we quantify the accuracy of tensor completion based on the elements we sample under the heuristic – we find that the accuracy is conducive to predicting the validation losses of unevaluated hyperparameters, even though the number of entries we sample is small. 
In terms of comparison with the GP used in BO, we have compared the two techniques (BOGP and HOTC) in terms of performance and found HOTC to be consistently superior. However, apart from this, there are many theoretical differences between a Gaussian Process and Tensor Completion – a Gaussian Process is a probabilistic model that can be updated in an online fashion. Tensor completion, however, requires a batch of data because it is based on the relationships between the data samples. A Gaussian Process can identify which hyperparameter combination is most likely to be optimal/has not been explored, but it cannot explicitly predict the validation losses of hyperparameter combinations that have not been evaluated. Tensor completion can do this. The Gaussian Process and tensor completion are so highly different theoretically, it not clear on what basis a comparative study should be made between the two. Furthermore, it is not clear why comparing HOTC with BO-GP on a concept level is critical for a convincing story in HPO because the HOTC approach is radically different from BO-GP. Indeed, we believe HOTC illustrates that BO-GP is not the only convincing story in HPO.

-The current algorithm only does exploitation but no exploration. This is surprising and why did the author design it in this way?

This is true. In the first version of the paper, we have identified adding exploration of the space to the algorithm as an area of future work. We think it is a very promising direction, but unfortunately we did not have the time to run these experiments yet. 
Initially, we wanted to start with a simple algorithm before adding complex steps – therefore our first approach was to implement an exploitative algorithm as is described in the paper. It is based on a simple idea: to identify a potentially optimal region of the search space through tensor completion and then examine this region in more detail. On testing this approach, it was found to deliver competitive performance against state-of-the-art hyperparameter optimisation algorithms, therefore we felt these results were worth sharing with the machine learning community. The algorithm can, of course be enhanced from its current form, but this will require more experimentation and it is not guaranteed that any of these enhancements will work as well as the current algorithm does. We are not averse to performing this experimentation, but in the current time-frame, it is not possible.

From: ICLR.cc/2023/Conference/Paper3949/Authors
title: Response to Reviewer BwZ7
comment: -The step of narrowing the search space may result in finding a local optimum. In other words, the very first search is just crucial for the final result.

This is true. The result of the very first search would depend strongly on the accuracy of the initial tensor completion – if the completion is accurate, one is more likely to find the optimal region of the hyperparameter search space. In the revised version of the paper, we have included tensor completion experiments in which we complete a tensor consisting of validation loss values for different hyperparameter combinations, using the same tensor completion technique as in HOTC. The tensor completion is observed to accurately model the general trends in validation loss values, which would facilitate the crucial first search, although it would perhaps overlook local variations in the validation loss values that could create the global optimum. 

A possible area of future work that we have identified is, indeed, to add some exploration, maybe random exploration, to our decidedly greedy algorithm. This has not been possible within the timeframe of submission of the paper – it would require a scheme to be devised and extensive experiments to be performed to establish that it improves over the current technique in speed and performance. It is definitely a goal for the future; our initial algorithm implementation was deliberately kept as simple as possible to facilitate incremental development. Even in its current state, in our experiments it has competitive performance next to state-of-the-art hyperparameter optimisation techniques.

-As the first work of using tensor completion for hyper-parameter search, I think it is important to perform a more thorough study on the quality of the completion. For instance, one could simulate a full tensor as ground truth and show how well the proposed algorithm can find the global minimum.

Please see general comments.

-The presentation of the results: the results could have been more convincing if the authors could provide results with more than 2 choices of execution time. For instance, consider visualizing the loss values in y-axis against execution time in x-axis.

Please see general comments.

-I would also be interested in the memory consumption of the proposed approach in comparison with SOTA.

This is indeed an interesting angle; we have just not had the time to perform the necessary experiments. We do have a framework to profile the memory usage of each technique implementation, but the profiling greatly increases the runtimes of the programs. Therefore, it was decided to prioritise time performance over memory usage for this submission. The system that we ran the most memory intensive experiment I.e. VGG transfer learning on had a relatively small amount of RAM: 12.68GB, but all of the hyperparameter optimisation techniques could run without using all the RAM.

-Comparing different tensor decomposition methods could also be an interesting contribution.

In past experiments, we have used tensor completion algorithms based on the CP, Tucker and TT decompositions – TT rank was in fact superior. However, an important consideration is how many hyperparameter combinations we are evaluating to generate the estimate of the complete tensor – the Cross technique (Zhang, 2019) is the only heuristic we have found that guarantees accurate tensor completion based by sampling as few tensor elements as is theoretically possible. To ensure that HOTC is competitive with state-of-the-art algorithms in speed, it is thus very important to use the Cross technique to dictate which hyperparameter combinations to evaluate. Since this technique is based on the Tucker decomposition, we have no choice but to use this decomposition, although we agree it is not the only decomposition that is effective in facilitating low rank tensor completion.

-How does this work compare to Deng and Xiao: "A New Automatic Hyperparameter Recommendation Approach Under Low-Rank Tensor Completion Framework"?

We have included this related work in Section 2 and describe how it differs from ours. Thank you for mentioning this reference.

From: ICLR.cc/2023/Conference/Paper3949/Authors
title: Response to Reviewer 1YoG
comment: -One is whether the low-rank assumption is reasonable for the hyperparameter selection task. Although the authors give an example at the beginning to show the similarity of the loss with varing different hyperparameters, this fact does not directly imply the low-rank structure. I think it is closer to the targeted tensor's smoothness rather than low rankness. It thus makes me doubt if the low-rank structure exists in real. Since the success of the proposed method is based on the low-rank assumption, this point is deserved to be carefully discussed.

Please see general comments.

-It is unclear how to dimension explosion issues when the combinations of hyperparameters are on a large scale. I have such concern since the Tucker model is used in the paper, but we all know that the size of the core tensor of the Tucker model will grow exponentially with the tensor order. It means that one must deal with a very high-dimensional core when the tensor completion is applied to high-order tensors. In the experiment, the authors only evaluate the proposed method on several classic machine learning models and a very shallow neural network. The experimental settings can not say the usefulness of the proposed method in the real world, where a large amount of hyperparameters needs to be tuned.

In the paper, we highly recommend that the Tucker rank is set to 1 for all dimensions for numerous reasons (including performance) which, in the revised version of the paper, are both explained and experimentally validated. The Tucker rank does not have to be set to this value, but when it is, the core tensor is essentially a single value. In this case, you can increase the number of dimensions of the tensor as much as you want, but the core will still only have one element i.e. it will not scale in size. The number of factor matrices would, of course, increase and therefore the curse of dimensionality is not fully negated.

However, it is true that we can increase the number of hyperparameters being optimised at a time, and use more complex models to investigate the effect of high dimensionality (as well as  the fact that each hyperparameter combination takes longer to evaluate). The CNN experiment from our original paper is now an 11 hyperparameter optimisation problem (leading to an 11-dimensional tensor to be completed), and the CNN is trained for longer on each hyperparameter combination before finding the validation loss. The experiment shows very promising results, we see HOTC competing with the best techniques even in this case. We believe that being able to optimize at least 11 hyperparameters puts this method in the position to be applied to a very wide range of real world experiments. 
To experiment with a more complex model i.e. a deeper neural network we optimised 7 hyperparameters involved the transfer learning of the VGG16 CNN  (over 18 million trainable parameters) on the CIFAR10 data set. Here as well, HOTC is competitive with the state-of-the art. Unfortunately, it was not possible for us to run larger scale experiments since we are limited in our access to the required computational resources. However, we believe that the experiments we have now submitted could be a useful starting point for future investigation.

From: ICLR.cc/2023/Conference/Paper3949/Authors
title: Response to Reviewer PJTn
comment: -The motivation of assuming the optimized metrics satisfies the low rank properties seems weak. Although heuristically sound reasonable. Could the author help to provide more evidence on reasoning of the low rank assumption? Although in the introduction, the paper mentions close parameter provides similar performance, it does not seem to motivate the low rank assumption. Is it possible to have search space with all evaluated loss, and then using tucker factorization to quantify if the low rank assumption holds?

Please see general comments.

-Not all tensor completion algorithm can converge, which in general depends on the sampling pattern and sampling ratio from the tensor. In this paper, the sampling pattern is directly related to S (set of search spaces) in algorithm 1. Could the author help to clarify how to select the search spaces to ensure the success of tensor completion algorithm here?

These factors were indeed an issue when we were using TMac-TT as the tensor completion algorithm (in previous experiments, we do not report these in the paper). However, the Cross technique (Zhang, 2019) for efficient tensor completion has solved all these problems. Firstly, it provides a deterministic heuristic dictating which tensor elements to sample, and the authors of (Zhang, 2019) have proved that the number of tensor elements sampled under this heuristic is the minimum required to generate a complete tensor for a given assumed Tucker rank.  The tensor completion is guaranteed to converge, because the sampled elements are put through a series of matrix operations to construct the core tensor and factor matrices of a Tucker decomposition of the completed tensor. These operations always reach a result.

The sampling pattern is related to the set of search spaces in that the set of search spaces dictates the dimensions of the tensor. The Cross heuristic considers the tensor and dictates which parts of the tensor need to be sampled for efficient tensor completion. We have endeavored to explain all these considerations in the revised version of the paper – hopefully a reader would be able to gain a better understanding of the Cross heuristic from our explanations.

-Overall, the writing of the paper can be improved. The algorithm description in section 4.1 and 4.2 can be rephrased to increase the readability.

This is true, and we have rewritten the corresponding sections to improve clarity and readability. We have put more emphasis on the reasoning behind the decisions we made in writing the algorithm, so hopefully this will help the reader to understand not just the algorithm, but also the rationale behind it.

From: ICLR.cc/2023/Conference/Paper3949/Authors
title: General Comments 
comment: We thank the Reviewers for their insightful comments and suggestions. As similar main points were raised, we address the common concerns in this section. For Reviewer-specific comments, we address them in the section of each corresponding Reviewer. Finally, we have also released our full codebase in the attached zipped folder for enhanced reproducibility. Please find attached notebooks in the code that can be run to reproduce our experimental results.

Low-rank assumption: This was a common point of concern, namely if the low-rank assumption of the hyperparameter tensor holds. In view of this, we performed additional experiments to check for this assumption. We have included an additional section in the paper, Section 4 where we empirically show that the approximation of the hyperparameter tensor is a well-motivated assumption in 3 different setups. We also motivate the use of rank-1 Tucker decomposition in the same Section.

Experimental results: A common concern was the comparison of the methods for only 2 time thresholds. To address this, we have included a graph illustrating the obtained validation losses over time for 5 setups. As is clear in Fig.2  in the paper, the proposed method is at least competitive with the competing methods in all the experiments we tried. Note that we have changed the neural network experiments to include a much larger network than the original version of the paper.

Other changes in the paper include: 

•	The “Related Work” section has been updated to reflect the paper suggested by Reviewer 4, we were not aware of this paper and are grateful to Reviewer 4 for bringing it to our attention.

•	The “Tensor Preliminaries” Section has been updated to provide a more thorough explanation of the Cross technique we use for tensor completion and why we chose it. In the initial paper, we merely referred the reader to the publication regarding this technique.

•	The layout and distribution of content in the paper has changed due to the addition of the section in which we validate the assumption that a tensor of validation losses is of low rank.

•	Accounting for the concerns of the reviewers, the section describing the algorithm has been edited to greater reflect the thought processes of the authors in devising it. Effort is made to explain the flow of the algorithm rather than just explain what each function in the algorithm does.

•	The part where heuristics to use HOTC in practice is discussed has a new paragraph about how the time taken by HOTC can be predicted based on the initial search space defined. 

•	The notation for the machine learning problems in all experiments has changed i.e., instead of “Problem 1” we use abbreviations like “KNN-C-Wine": a short form for “KNN classification on the wine data set”, which are more associative to what is being discussed.

•	Finally, we have created appendices where we describe the search spaces for each of the hyperparameter optimisation problems, the configurations of HOTC and the budget configurations for HB and BOHB.

From: ICLR.cc/2023/Conference/Paper3949/Reviewer_PJTn
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper proposed a new hyper parameter optimization method using tensor completion algorithm. The new algorithm consider a N dimensional tensor with its dimension representing tunable parameters and values representing feedbacks (e.g. validation loss). A Tucker factorization is used to for tensor completion in this paper.

Evaluation with machine learning algorithm, including SVM, KNN, Random Forest, Neural Network shows a better selected parameters for optimize the validation losses in the given execution time, as compared to other Bayesian Optimization approach. 
strength_and_weaknesses: Strength
[+] Using tensor completion to improve hyper parameter optimization is a new and innovative idea.


Weakness
[-] The motivation of assuming the optimized metrics satisfies the low rank properties seems weak.
Although heuristically sound reasonable. Could the author help to provide more evidence on reasoning of the low rank assumption? Although in the introduction, the paper mentions close parameter provides similar performance, it does not seem to motivate the low rank assumption. Is it possible to have search space with all evaluated loss, and then using tucker factorization to quantify if the low rank assumption holds?

[-]  Not all tensor completion algorithm can converge, which in general depends on the sampling pattern and sampling ratio from the tensor. In this paper, the sampling pattern is directly related to S (set of search spaces) in algorithm 1. Could the author help to clarify how to select  the search spaces to ensure the success of tensor completion algorithm here?

[-] The numerical results seem weak. With only the final searching results, it is difficult to quantify the performance of the algorithm. Could the author provide the convergence of plot of the parameter optimization process to demonstrate either fast or better convergence?
clarity,_quality,_novelty_and_reproducibility: Overall, the writing of the paper can be improved. The algorithm description in section 4.1 and 4.2 can be rephrased to increase the readability. 
summary_of_the_review: Overall, I feel the idea of using tensor completion to improve hyper-parameter tuning is an interesting and innovative idea. 

However, due the concerns from 
[1] The motivation of the paper seems weak and little evidence has been provided to prove the low rank property holds, let along why Tucker rank is selected in the paper.
[2] There are some concerns on the main algorithm of the paper, particular from sampling patter and sampling ratio to ensure tensor completion converges. Some minor concerns are from the rank selection as well.
[3] In sufficient numerical results to demonstrate the advantage of the new algorithms. 

Thus, I feel the paper can be improved. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper3949/Reviewer_1YoG
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper applied tensor completion to search for the optimal hyperparameters for machine learning methods. Precisely, the combinations of parameter candidates are mapped on the tensor values with multiple modes. By the efficient sampling trick (Zhang, 2019) followed by tensor completion, the optimal parameters are selected from the minimum (or maximum) values of the completed tensor.
strength_and_weaknesses: **Strength:**

1. As a new application of tensor completion, this work may be the first to apply tensor completion to NAS (as claimed in the paper), which might inspire more studies on this problem in the tensor community.
2. Some tricks, such as the mappings from hyperparameters to tensor values and the ones for performance improvement, is interesting and might be applied to broader ranges of tensor applications.

**Weakness**

1. Several crucial problems are not convincingly discussed in the paper.
2. The experimental results cannot sufficiently support that the proposed completion method outperforms SOTAs in practice since only several simple architectures are used for evaluation in the paper.
clarity,_quality,_novelty_and_reproducibility: **Clarity and quality**

This paper is relatively well written. The main idea is clearly explained and easy to be followed. However, there are several questions which are crucial to be answered but lacking in the paper.

1. One is whether the low-rank assumption is reasonable for the hyperparameter selection task. Although the authors give an example at the beginning to show the similarity of the loss with varing different hyperparameters, this fact does not directly imply the low-rank structure. I think it is closer to the targeted tensor's smoothness rather than low rankness. It thus makes me doubt if the low-rank structure exists in real. Since the success of the proposed method is based on the low-rank assumption, this point is deserved to be carefully discussed.
2. It is unclear how to dimension explosion issues when the combinations of hyperparameters are on a large scale. I have such concern since the Tucker model is used in the paper, but we all know that the size of the core tensor of the Tucker model will grow exponentially with the tensor order. It means that one must deal with a very high-dimensional core when the tensor completion is applied to high-order tensors. In the experiment, the authors only evaluate the proposed method on several classic machine learning models and a very shallow neural network. The experimental settings can not say the usefulness of the proposed method in the real world, where a large amount of hyperparameters needs to be tuned.

**Novelty**

The application of tensor completion seems new in the NAS task.
summary_of_the_review: I agree that tensor completion methods might be useful for quickly predicting the optimal combinations of hyperparameters. But as mentioned, the usefulness is held if the low-rank assumption exists in practice in the incompleted tensor. However, this point is not clearly discussed and proved from either theoretical or empirical side.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3949/Reviewer_C2WJ
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposed a HPO method using tensor decomposition. It uses tensor decomposition to predict hyperparameter performance, which is a very novel idea. The best hyperparameter after tensor completion is recommended as the next hyperparameters to try and the algorithm continues until a predefined budget exhausts. Experiments compared to common HPO methods on tuning 4 algorithms showed some potential of the methods.
strength_and_weaknesses: On the strength, the work proposed a very novel route for HPO through tensor completion and I found it very interesting.


On the weakness, I have major concerns on the technical quality and clarity, as well as in the experiments set up. I think many questions should be answered to show tensor completion is a promising route for HPO. The questions follows:

On the technical quality.
* Why is the low rank assumption true? Any motivating examples? For example, on some small scale problem, evaluate all the HP configurations and show that the low rank approximation has good accuracy.
* How good is the predicted accuracy based on tensor decomposition, giving more and more hyperparameter evaluations? Tensor completion with only a few entries could not end up with a good prediction. How does it compare to GP that is used in BO? This is not studied but critical for a convincing story in HPO.
* The current algorithm only does exploitation but no exploration. This is surprising and why did the author design it in this way?

On the experiment set up.
* In Section 5.2, the author wrote “For HOTC, T was achieved by varying the number of
tensor completion cycles, resolution of the initial search space, maximum number of elements for grid-search and minimum resolution for real-valued hyperparameters.” This is problematic because the search spaces (min, max, resolution) should be exactly the same for all the methods.
* The narrowing down search space part to me seems a bit arbitrary. Do you have experiment results without it? I would highly recommend taking this component out and comparing the proposed algorithm  to other methods.
* It can be hard to know the low rank number “T” in advance. Did you try some other values in the experiments and some tip on how to set T?

On the clarity:
* If the tensor decomposition used in the paper is “noisy tensor completion with Cross measure-
ments (Algorithm 2, Zhang (2019))”, then I would expect the authors to explain at least the core ideas.
* Is it a hard requirement for the uniformness of the integer and real hyperparameters (Section 4.1)?
* After reading Section 4.3, I understand why the grid search is used in Algorithm 1. It is because when the search space is small enough, one could afford a grid search and not miss any important HPs. But it will be very helpful to explain that while describing the algorithm, or at least mention that it will be explained later on.
* The language needs to be improved for correctness and smoothness.

clarity,_quality,_novelty_and_reproducibility: The clarity and quality is not matching the ICLR bar in my opinion, as already mentioned in the weakness section. The paper contains limited detail of the experiments and no code is available (“removed for double-blind review purposes”). The novelty is good and I think the authors should make a major edit and give it another try in the future.
summary_of_the_review: The paper is not matching the ICLR standard due to the weakness that I have mentioned above.
correctness: 1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper3949/Reviewer_BwZ7
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper introduces a new way to perform hyper-parameter search based on tensor completion. The combinatorial space of hyper-parameters is formulated as a multi-way tensor, where the loss value of a performed trial is registered as tensor entries. By decomposing and reconstructing the highly sparse tensor, one could predict the loss value of unseen hyper-parameter settings. On a set of experiments, the proposed method has shown comparable results with SOTA algorithms. 
strength_and_weaknesses: This idea is quite intriguing and I’ve been expecting some work in this direction for some time. 
My major concerns are the following: 

Weaknesses: 
1. The step of narrowing the search space may result in finding a local optimum. In other words, the very first search is just crucial for the final result.
2. As the first work of using tensor completion for hyper-parameter search, I think it is important to perform a more thorough study on the quality of the completion. For instance, one could simulate a full tensor as ground truth and show how well the proposed algorithm can find the global minimum. 
3. The presentation of the results: the results could have been more convincing if the authors could provide results with more than 2 choices of execution time. For instance, consider visualizing the loss values in y-axis against execution time in x-axis. 
4. I would also be interested in the memory consumption of the proposed approach in comparison with SOTA. 
5. Comparing different tensor decomposition methods could also be an interesting contribution. 
6. How does this work compare to Deng and Xiao: "A New Automatic Hyperparameter Recommendation Approach Under Low-Rank Tensor Completion Framework"? 


clarity,_quality,_novelty_and_reproducibility: The paper is very well presented and easy to follow. 
The proposed idea is novel enough to me. 
The authors promise that the codes will be made available upon publication. 

summary_of_the_review: In general this is a very interesting idea but the presentation could have been better, especially with respect to points 2 and 3 under weakness. I would like to see the work published but after a revision. It would be much more valuable if the community could have a sound and thorough analysis of this idea, even if it turns out to underperform some SOTAs, than to have some discrete numbers that are supposed to demonstrate the superiority of the proposed approach. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold