From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: We apologize for failing to find a third reviewer despite many requests and reminders.

As the area chair, I am familiar with quantum annealing and with mining association rules. I agree with the overall sentiment of the two reviewers that this paper is interesting. It should be published, but not at ICLR now. 

In their responses, the authors provided clarifications but did not show that the reviews are incorrect. To help the authors resubmit elsewhere, here are a few questions or suggestions.

1. Clarify whether you used actual quantum hardware from D-Wave and/or Fujitsu. If yes, show that your method enables finding optimal solutions better or faster using the hardware. None of the current figures show this.

2. The FARM method does not have guarantees, and the rules discovered may be false in the solution to the annealing problem. So the contribution of the paper is not solid and strong enough for ICLR.

3. Be more incisive when discussing previous research. This area chair has been a professor and FAANG director in ML for over 30 years. I am not aware of a single real-world useful application of association rules, for market basket analysis or for anything else.

Also, fix the formatting of citations, which has words run together with missing spaces
justification_for_why_not_higher_score: Results are preliminary.
justification_for_why_not_lower_score: The idea is worth exploring.

From: ICLR.cc/2023/Conference/Paper4323/Authors
title: Revise the paper and add clarifications according to reviewers comments
comment: C1  
Some concepts introduced in the paper should be clarified. It seem an incremental work where many concepts could better described and the results should be improved adopting different strategies.

R1  
This work investigates the association between decision variables in annealing-based solvers. We do not see similar work in the literature. We do not see our work as an incremental one.  
Sampling is a core concept in a probabilistic optimisation method, where you sample a lot of times to find a promising solution. For those readers who are not familiar with annealing-based method, we add a brief introduction in the first section for readers to quickly pick up necessary information.  
Annealing, e.g., simulated annealing, is a family of probabilistic methods for optimising the variables of a system (e.g. minimising a function). In annealing, we heat the system to high temperature level and gradually bring down the temperature. Variables in the system gradually lose their energy and eventually sit in a low energy state. One round of heating and cooling, i.e., an annealing process, produces one sample, which is a configuration of all variables. Annealing process is a random process. The energy distribution of the samples follows a Boltzmann distribution, i.e., the system is more likely to be in a lower energy state. Quantum annealing works in a similar way, except that it makes use of quantum mechanics.  

C2  
The are same part of the paper that could be better explained. For instance how the rules have been adopted to reduce the size of the dataset is not reported.

R2  
As we briefly mentioned in section 2 and 3.3, we use rule to condition a QUBO problem, such that the number of variables is reduced. We have never mentioned about reducing the size of a dataset. The way of adopting rules is well documented in previous literature. To help readers better understand how it works, we add detailed explanation in the appendix.

C3 and C4  
poor result  
The experimental evaluation is poor and the validity of the proposed approach is non confirmed by the reported results.

R3 and R4   
Section 5.1 demonstrates our method outperforms SOTA on Quantum annealing (QA). The experiment on Digital Annealer (DA) in section 5.2 demonstrates the limited utility of "fix variables" for classical solvers. Section 5.1 and 5.2 indicate the critical difference between QA and DA on this topic, which missing in existing literature. 
We add clarification accordingly in section 1 and 5.

From: ICLR.cc/2023/Conference/Paper4323/Authors
title: Revise the paper and add clarifications according to reviewers comments
comment: C1  
As the authors mentioned, determining the number of variables that maximizes performance is difficult.

R1  
Determining the number of variables is challenging and remains an open question and a weakness in all existing literature. We demonstrate FARM as a visualisation tool that help user understand the distribution of the rules and narrow down the range of optimal number of variables to be fixed. In the history of this research topic, we are the first to investigate the distribution of frequent patterns in terms of rule length v.s. support rate. Our next step is to examine the corresponding determination methods.

We add this clarification to the conclusion and discussion section

C2  
No theoretical guarantee or justification for the approximation quality (e.g., risk to find wrong rule) is provided.

R2  
FARM is an approximated algorithm in the sense that it does not enumerate all possible rules, which is neither resource efficient nor necessary in our use case. We have demonstrated that FARM can find numerous amounts of rules from the demo problem set. From them, we are only selecting one or two promising rules for subsequent evaluations. The approximation has a limited impact on the results.
We add this clarification in the description of FARM algorithm.

C3  
The paper is sometimes difficult to follow for those who are not familiar with a QA.

R3  
We add a brief intro of annealing in the first section to help readers quickly pick up the information necessary for understand the background.

Annealing, e.g., simulated annealing, is a family of probabilistic methods for optimising the variables of a system (e.g. minimising a function). In annealing, we heat the system to high temperature level and gradually bring down the temperature. Variables in the system gradually lose their energy and eventually sit in a low energy state. One round of heating and cooling, i.e., an annealing process, produces one sample, which is a configuration of all variables. Annealing process is a random process. The energy distribution of the samples follows a Boltzmann distribution, i.e., the system is more likely to be in a lower energy state. Quantum annealing works in a similar way, except that it makes use of quantum mechanics.

From: ICLR.cc/2023/Conference/Paper4323/Reviewer_5aqV
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper proposes applying association rule mining for reducing the problem size of QUBO. The mining is performed for solution samples, and the authors proposes an efficient approximate association rule mining specifically for QUBO. Empirical results show that the proposed procedure can substantially reduce the problem size and find good solutions compared with an existing method.
strength_and_weaknesses: Strength:
- The basic idea, combining a quantum annealer (QA) and association rule mining, is interesting and seems novel.
- The empirical results suggest superiority of the proposed method.

Weaknesses:
- As the authors mentioned, determining the number of variables that maximizes performance is difficult.
- No theoretical guarantee or justification for the approximation quality (e.g., risk to find wrong rule) is provided.
- The paper is sometimes difficult to follow for those who are not familiar with a QA. 
clarity,_quality,_novelty_and_reproducibility: Clarity: OK, but sometimes a bit difficult to follow for those who are not familiar with a QA. 

Quality: Fine, though the proposed method is a simple heuristics.

Novelty: Combination of annealer and rule mining would be novel

Reproducibility: OK.

summary_of_the_review: The paper seems to try an interesting approach combining QA and rule mining. Although the proposed method is a simple heuristics that do not have a theoretical support, the empirical results suggest the practical usefulness of the proposed framework. However, to be honest, I am not familiar with the topic and my review contains educated guess. 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4323/Reviewer_ABPP
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes to use heuristic method to discover association rules between variables in an optimization problem.

strength_and_weaknesses: + An interesting idea compared to the state of the art works
- Poor results
clarity,_quality,_novelty_and_reproducibility: Some concepts introduced in the paper should be clarified. It seem an incremental work where many concepts could better described and the results should be improved adopting different strategies.
summary_of_the_review: The introduction of the problem is unclear. For instance in the introduction in not clear the the meaning of samples in an optimization problem. This should be clarified.

The are same part of the paper that could be better explained. For instance how the rules have been adopted to reduce the size of the dataset is not reported. 

The idea could be interesting but the experimental evaluation is poor and the validity of the proposed approach is non confirmed by the reported results.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough