Paper: Efficient Interactive Preference Learning in Evolutionary Algorithms: Active Dueling Bandits and Active Learning Integration (ICLR-2024)

Reviewer comment (Writing):
$\hat{P}$ denotes the predicted preference matrix, but there is no prediction step described. According to Eq. 4, it seems to be the observed preferences.

Author response:
The definition of $\hat P, P,\mathbf{w}$ are introduced in this paper, $P$ in basic definition, $\hat P, \mathbf{w}$ in **section 2.2.2**. The calculation of $\mathbf{w}$ is in **line 21** of **algorithm 1** which means we will increment the corresponding weight vector according to the comparison result. $w_{ij}$ denotes the total time arm i beat arm j. $P$ is the theoretical winning probability of incumbent solutions, the calculation of which is shown in **eq3**. $\hat P$ is the predicted value on $P$ defined by **eq4**. $\hat P$ rely on winning time matrix $\mathbf{w}$.

Reviewer comment (Writing):
Unclear how $P$, $\mathbf{w}$ and $\hat{P}$ are related.

Author response:
The definition of $\hat P, P,\mathbf{w}$ are introduced in this paper, $P$ in basic definition, $\hat P, \mathbf{w}$ in **section 2.2.2**. The calculation of $\mathbf{w}$ is in **line 21** of **algorithm 1** which means we will increment the corresponding weight vector according to the comparison result. $w_{ij}$ denotes the total time arm i beat arm j. $P$ is the theoretical winning probability of incumbent solutions, the calculation of which is shown in **eq3**. $\hat P$ is the predicted value on $P$ defined by **eq4**. $\hat P$ rely on winning time matrix $\mathbf{w}$.

Reviewer comment (Theory):
$P$ and $\hat{P}$ are formalized as probability matrix, but "user preferences are consistent and stationary", which is even used in the Algorithm 1 by reusing previous results. Resultingly $p_{ij}$ is an indictor function?

Author response:
We changed the reused symbols $L_t$ to $V_t$ (**section 2.4**) to avoid misdirection. The $V_t$ is a probability distribution of the whole population according to the recommended solutions while $\hat P$ denotes the winning probability of incumbent solutions (i.e., K arms).

Reviewer comment (Theory):
Eq. 6 depends on $P$, but it is not clear how this is obtained.

Author response:
The definition of $\hat P, P,\mathbf{w}$ are introduced in this paper, $P$ in basic definition, $\hat P, \mathbf{w}$ in **section 2.2.2**. The calculation of $\mathbf{w}$ is in **line 21** of **algorithm 1** which means we will increment the corresponding weight vector according to the comparison result. $w_{ij}$ denotes the total time arm i beat arm j. $P$ is the theoretical winning probability of incumbent solutions, the calculation of which is shown in **eq3**. $\hat P$ is the predicted value on $P$ defined by **eq4**. $\hat P$ rely on winning time matrix $\mathbf{w}$.

Reviewer comment (Writing):
$L_t$ denotes the predicted preference distribution. How is this different from $\hat{P}$ and $\mathbf{w}$?

Author response:
We changed the reused symbols $L_t$ to $V_t$ (**section 2.4**) to avoid misdirection. The $V_t$ is a probability distribution of the whole population according to the recommended solutions while $\hat P$ denotes the winning probability of incumbent solutions (i.e., K arms).

Reviewer comment (Writing):
$L$ is used as symbol for the loss as well as the preference distribution.

Author response:
We changed the reused symbols $L_t$ to $V_t$ (**section 2.4**) to avoid misdirection. The $V_t$ is a probability distribution of the whole population according to the recommended solutions while $\hat P$ denotes the winning probability of incumbent solutions (i.e., K arms).

Reviewer comment (Writing):
$p_\text{min}$ is never described and only defined within Algorithm 1.

Author response:
The definition of $p_{min}$ is introduced below **eq5** and in **line 5** in **Algorithm 1**.

Reviewer comment (Writing):
Fitness is never introduced, but is likely to be assumes as a function of $\mathbf{F}(x)$. However, it seems to be also assumed that this function is linear, as Section 2.3 mentions "weight vectors"?

Author response:
Firstly, the fitness function is not a function in $\mathbf{F}(x)$. It helps conduct non-dominated sort in MOEAs, especially dominance-based MOEAs. The fitness function can filter solutions generated by MOEAs and reserve non-dominated solutions. Decomposition-based and indicator-based MOEAs don’t need fitness function, they find solutions in every weight vector direction. So those weight vectors are not utilized to approximate fitness function. It is noteworthy we construct a virtual fitness function. State-of-the-art **preference-based evolutionary multi-objective (PBEMO)** algorithms are all model-based. They need to approximate the whole utility function which may not exist or be hard to express. Our virtual fitness function is only accurate concerning the optimal solution as **eq5** indicates.

Reviewer comment (Theory):
Unclear how preferences are generated, but it does not seem to be possible to employ arbitrary generation processes, as a scalar fitness function is not able to result in cyclic preferences.

Author response:
In our experiment session, preference information is produced by a **virtual user**, which in specific we will set a reference point as user preference. When comparing two incumbent solutions, we will compare their distance to the reference point.

Reviewer comment (Writing):
Virtual fitness function not formally introduced.

Author response:
The virtual fitness function is actually $V_t$ (**section 2.4**).

Reviewer comment (Reproducibility):
It seems the bandit algorithm is used to evaluate candidate solutions, but in how far is the EA algorithm still working with the multi-objective targets?

Author response:
MOEAs will work with the bandit until they reach the maximum evaluation generation. The **bandit algorithm** performs as an **AI assistant** to guide the MOEAs to converge to user-preferred regions. Without the guidance of the bandit, MOEAs will converge to a PF rather than a certain solution.

Reviewer comment (Writing):
What is meant with "runs as normal"?

Author response:
“runs as normal” means the MOEAs run as defined in their origin paper. We have refined this sentence.

Reviewer comment (Reproducibility):
What is optimized in between the query times?

Author response:
Between two queries, MOEAs will process the last recommendation. And the whole population will converge to the last recommendation iteratively.

Reviewer comment (Novelty):
Section 2.3 mentions "assigning a fitness value" and "storing weight vectors". This suggests the preferences are used to learn some feedback function that can be used to evaluate candidates. How does this differ from learning a fitness function?

Author response:
If I’m getting it right, the reviewer may misunderstand that our proposed method also needs to simulate a fitness function representing user preference.
        - Traditional **PBEMO** algorithms learn user preference by **approximating a utility function** with the help of mathematical tools like GP, RankNet, etc. The utility function in reality may not exist or be hard to express. However, our proposed method learns the **optimal solution** with the help of a bandit. The virtual fitness function is only an **assumption** about preference distribution which is only accurate at global optima.
        - “assigning a fitness value” means the fitness is assigned to solutions according to the virtual fitness function.
        - We mentioned “storing weight vectors” in **section 2.4**. Those weight vectors are not used to approximate a fitness function. They are the corresponding weight vector of recommended solutions concerning **search direction**.

Reviewer comment (Writing):
Unclear how "rounds" are related to the budget, timesteps and queries.

Author response:
In this paper, round shares the same meaning as timestep. In each round, the bandit algorithm will select two incumbent solutions to make the next query. Budget is the number of times the dueling bandit is allowed to present the query to the Oracle. The relationship between the round and budget is: $round > K\ge budget$.

Reviewer comment (Reproducibility):
Results are not reproducible, as several hyperparameters are not mentioned, especially concerning the EA algorithms.

Author response:
The results are reproducible and hyperparameter settings are introduced in **Appendix A.5.1**.

Reviewer comment (Experiments):
"Our proposed method achieves the minimum mean in 8 instances" - This only seems true if one is able to optimize the EA algorithm used in conjunction with RUCB-AL. Of course, in practice this is not really possible. Under a fair comparison (preselected EA algorithm), IEMO/D seems to be the best?

Author response:
Although from the perspective of $Loss$, IEMO/D seems to have the most time of beating other algorithms. However, empirically we find only calculating $Loss$ is not enough. Because IEMO/D has the poorest convergence among these peer algorithms. The final results of IEMO/D are scattered around the whole PF. So apart from using $Loss$ as a performance metric, we intend to take the total $Loss$ of the whole population into consideration.

Reviewer comment (Presentation):
Table 1 is hardly readable because of the number format. Maybe use relative values (e.g. relative difference to the optimum).

Author response:
We’ve rearranged **Table 1** and made it readable.

Reviewer comment (Reproducibility):
PSP trials do not mention used budget, rounds, etc.

Author response:
The most important parameter of PSP problems is the native energy of different protein structures. The native energy of the 4 protein are listed in **Table 2**. The other parameter settings are aligned with Zhang et al. (2023). We added this statement to the paper.

    Zhang, Zhiming, et al. "Pareto Dominance Archive and Coordinated Selection Strategy-Based Many-Objective Optimizer for Protein Structure Prediction." *IEEE/ACM Transactions on Computational Biology and Bioinformatics* (2023).

Reviewer comment (Novelty):
The work does not sufficiently consider related work, e.g., "Preference-based Multiobjective Evolutionary Algorithm for Power Network Reconfiguration" or "A Preference Based Interactive Evolutionary Algorithm for Multi-objective Optimization".

Author response:
As to the three recommended reference papers, we’ve thoroughly read them.
    - *“Preference-based Multiobjective Evolutionary Algorithm for Power Network Reconfiguration”* utilized a preference-based evolutionary multi-objective (PBEMO) algorithm, namely PD-NSGA-II, to solve power network reconfiguration problem. However, their proposed method is only an easily modified version of NSGA-II concerning the dominance relationship. It didn’t learn preference by approximating a utility function or through pairwise preference from human feedback. So we didn’t take this paper into account.
    - *“A Preference Based Interactive Evolutionary Algorithm for Multi-objective Optimization: PIE”* introduced an interactive PBEMO algorithm implementing NAUTILUS. However, it is not based on pairwise preference learning which is different from our proposed method and peer algorithms in this paper. So we didn’t take this paper’s method as a peer algorithm.
    - *“Preference-based Online Learning with Dueling Bandits: A Survey”* listed state-of-the-art dueling bandit algorithms. As far as we investigated, none of these dueling bandit algorithms combine active learning in their process. They referred to active ranking, but this method is not a dueling bandit algorithm.

Reviewer comment (Novelty):
It should also be mentioned that a vast number of query selection strategies already exist (e.g., see Bengs 2021), that aim at reducing the number of queries to be performed. Why should this not be considered "active learning"?

Author response:
As to the three recommended reference papers, we’ve thoroughly read them.
    - *“Preference-based Multiobjective Evolutionary Algorithm for Power Network Reconfiguration”* utilized a preference-based evolutionary multi-objective (PBEMO) algorithm, namely PD-NSGA-II, to solve power network reconfiguration problem. However, their proposed method is only an easily modified version of NSGA-II concerning the dominance relationship. It didn’t learn preference by approximating a utility function or through pairwise preference from human feedback. So we didn’t take this paper into account.
    - *“A Preference Based Interactive Evolutionary Algorithm for Multi-objective Optimization: PIE”* introduced an interactive PBEMO algorithm implementing NAUTILUS. However, it is not based on pairwise preference learning which is different from our proposed method and peer algorithms in this paper. So we didn’t take this paper’s method as a peer algorithm.
    - *“Preference-based Online Learning with Dueling Bandits: A Survey”* listed state-of-the-art dueling bandit algorithms. As far as we investigated, none of these dueling bandit algorithms combine active learning in their process. They referred to active ranking, but this method is not a dueling bandit algorithm.

Reviewer comment (Theory):
The approach does not seem to be fitness free, as Sec. 2.3 mentions something resembling learned functions.

Author response:
Our proposed method is fitness-free because we don’t learn a fitness function through the process of preference learning. Compare to those fitness-based PBEMO algorithms (e.g., I-EMO/D-PLVF, I-EMO/D-PPL) that **approximate a fitness (utility) function** in preference learning, our method only cares about the **global optima**. Then construct a **virtual utility function** that is only accurate at global optima. This process avoids approximating a fitness function that may not exist or is hard to express in reality.

Reviewer comment (Experiments):
The difference between the "AL" part of the algorithm and other query selection strategies is not sufficiently substantiated.

Author response:
Other query strategies balance between exploration and **exploitation from the perspective** of different kinds of **regret**. However, our proposed method balances between **random search and greedy search** from the perspective of **loss (uncertainty)**. With the help of active learning, our proposed method can prompt the dueling bandit’s ability to progressively find the global optima within a limited budget. This is very important in real problems, because users may get impatient and give inconsistent preference after querying too much.

Reviewer comment (Novelty):
My major concern is the novelty of the proposed framwork, as learning objective functions according to human feedback is not new[1][2][3].

Author response:
The reviewer may misunderstand our research field. If I’m getting it right, the reviewer thinks our proposed method is to conduct preference learning from human feedback and also learn an objective function representing user preference. However, we are implementing our proposed method in a **preference-based evolutionary multi-objective optimization (PBEMO)** scenario. 
    - As far as we know, we are the first to adopt fitness-free preference learning in MOEAs. State-of-the-art algorithms learn user preference by approximating a utility function, which in reality may not exist or is hard to express. This kind of method is called fitness-based or model-based because it depends on prior knowledge of the distribution of solutions (e.g., Bradley Terry model). However, our proposed method utilizes the property of dueling bandit to conduct preference learning in MOEAs. Given incumbent solutions, the dueling bandit will not try to approximate a utility function. Conversely, it only focuses on the optimal solution. To utilize the recommended solution $\hat{\mathbf{z}}^*$, we construct a virtual utility function. The virtual utility function $V_t$ (**section 2.4**) is only accurate regarding optimal solutions. 
    - As to the three recommended reference papers, we’ve thoroughly read them. The research scenarios are different. **Roijers et al. (2017)** studied preference learning in an online interactive multi-objective reinforcement learning (MORL) scenario. They mapped a policy to an arm and with the help of a dueling bandit the policy can converge to an optimal solution aggressively. **Ding & Zhou (2018)** targeted learning a linear combination loss function that outperforms other loss functions. The different loss ingredients are so-called multi-objectives. Their learned loss function will participate in the optimization process. The preference learning indirectly helps optimization. **Tucker et al. (2020)** proposed a Bayesian dueling bandit method for single-objective preference learning. These three research all converge to **a single solution or an aggregation** at last. However, in our research field, PBEMO, preference learning directly helps the optimization algorithm converge to **a set of favorite solutions**. Our final results are a population that helps our algorithm avoid trapping in local optima.

Reviewer comment (Theory):
The proposed active dueling bandits algorithm is confusing. In traditional active learning, there should be an uncertainty measure, according to which the learner decides whether to query; in active dueling bandits proposed in this paper, if I'm getting it right, whether to query if sorely focusing on if the pair is compared before, which is a noisy feedback that is not trustworthy.

Author response:
About the proposed active dueling bandit method: 
    - For our proposed method, we have uncertainty measures to decide whether to query. As can be seen in **eq5**, we decide the next query based on a balance between random and greedy search. Also, our proposed dueling bandit is not just based on whether it has been asked before. The two arms are selected according to the definition of $p_{a_c}$ in **eq5**. Then if the two have been queried before, they will not be presented to the user again. 
    - We understand your concern that user feedback may be noisy and not trustworthy. In our experiment, the user feedback follows the **Bernoulli distribution**. It doesn’t matter in this consultation session user gives the wrong answer. Our preference will be adjusted in several consultations. 
    - About the regret bond, we are confident that our regret bond can be polished up since the experiment results suggest our proposed method works well with multi-objective optimization problems. We will polish it up and reference CCB and SCB structure (Zoghi et al., 2015).

Reviewer comment (Theory):
Weak connections between the dueling bandit algorithm and the main goal of the paper. It seems only this regret is analyzed instead of how well it optimizes the objectives, which are not discussed in a theoretical manner.

Author response:
We added an elaboration of the optimization module (**section 2.2**) before the preference elicitation module to connect our proposed method with optimization. Since not all feasible solutions are visible to users, MOEAs perform as optimizers to generate incumbent solutions. 
    - In this paper, we haven’t considered MOEAs to utilize the regret. In our future work, we are going to utilize the regret in MOPs (e.g., take the recommendation point $\hat {\mathbf{z}}^*$ as a distribution rather than a converging target).

Reviewer comment (Theory):
The authors provide a regret for the proposed algorithm with order $O(T)$. This result seems too weak. Could the author further improve this bound? Also, could the authors compare this with some existing benchmarks? Since there are many algorithms that are available in dueling bandit area that can achieve $O(\sqrt{T})$ upper bound.

Author response:
We are confident that our regret bond can be polished up since the experiment results suggest our proposed method works well with multi-objective optimization problems. We are committed to considering CCB and SCB structure (Zoghi et al., 2015) and further refining the regret bond associated with our proposed method.

Reviewer comment (Theory):
When implementing the dueling bandit, it seems the greed algorithm was implemented. What if we consider an algorithm with an upper confidence interval? Would this give a better regret bound?

Author response:
As to why not use the UCB function. Traditional UCB is not active enough, empirically given $K=10$ we will need more than 300 rounds to receive an acceptable accuracy. Our proposed method is inspired by UCB. Traditional RUCB algorithms only balance the mean and uncertainty of winning probability. Our designed active RUCB-AL balances exploration and exploitation from another perspective. We've elaborated on the components of **eq 5**, emphasizing its role in achieving a balance between random and greedy search strategies.

Reviewer comment (Theory):
RUCB algorithm relies on the assumption that there exists a Condorcet winner. Relaxation of this assumption, i.e., find Copeland winner instead, requires different algorithm design, like the CCB and SCB algorithm in Zoghi et al. (2015). The authors dropped the Condorcet winner assumption in Assumption 1. I would appreciate it if the authors could elaborate on how this can be achieved with the modified RUCB algorithm in Algorithm 1.

Author response:
2. We have corrected the definition of the Copeland winner (**eq 2**). In order to actively query users, we designed our bandit by choosing arms from the perspective of uncertainty (i.e., MSE loss). We believe by improving the certainty of the winning probability matrix, the bandit algorithm will output a more precise Copeland winner within a limited budget. Empirically, this method proves to be efficient. We are also committed to considering CCB and SCB structure (Zoghi et al., 2015) and further refining the regret bond associated with our proposed method.

Reviewer comment (Theory):
Related to the previous comment, it is not clear to me why the bonus term of standard RUCB algorithm is modified to Eq 5. In addition, if I am not missing anything, the squared loss of $\hat{p}\_{i,j}$ in Eq 5 cannot be computed by the algorithm, since the value of $p_{i,j}$ is unknown?

Author response:
3. Traditional RUCB only balances the mean and uncertainty of winning probability. Our designed active RUCB-AL balances exploration and exploitation from another perspective. The first term in **eq5** refers to a decaying random search process, while the second term refers to a greedy search based on the loss function. We designed the **eq5** to balance between random search and greedy search. As to how to achieve $p_{i,j}$ which is unknown to the user. It is approximated with the following steps:
    - In the first round, $p_{i,j}=0.5,\ \forall\ i,j\in K$,
    - After the second round, we will receive a current optimal solution $\hat{\mathbf{z}}^*_t$,
    - According to the recommendation $\hat{\mathbf{z}}^*_t$, we can calculate a virtual utility function $V_t$ (**section 2.4**). Unlike existing PBEMOs that require mathematical tools like GP to approximate the whole utility function, our virtual utility function is not accurate except at the optimal solution.
    - calculate $p_{i,j}=\sigma(V_t(\mathbf{z}_i)-V_t(\mathbf{z}_j))$