Paper: Objectives Are All You Need: Solving Deceptive Problems Without Explicit Diversity Maintenance (ICLR-2024)

Reviewer comment (Experiments):
The range of problems where the proposed algorithm would be useful, as covered by the two example problems given in the paper, are too narrow.

Author response:
Thank you for this suggestion. We agree that we must detail the conditions under which this algorithm can work well. We believe that our method can work in problems regardless of their dimensionality. We have studied 1D (time-based subaggregation) and 2D (space-based subaggregation), but we do not see why this method cannot easily be applied to higher dimensions by simply changing the way we sub-aggregate to be sensitive to the new dimensions. Furthermore, we believe there are many interesting problems that exist on a 2D plane that would be interesting to study in future work. For example, Go, Chess, and all the Atari benchmarks have only two dimensions in state space, yet would each come with their own unique challenges and deceptive-ness. 

Furthermore, we have added a new domain to the paper, which involves finding parameter values for a robotic arm. This is another example of a problem that exists on a 2D plane, however we think it presents new insights due to the structure of the search landscape.

Reviewer comment (Novelty):
Both problems take place on a 2D plane where the space-based subaggregation method can be applied naturally.

Author response:
Thank you for this suggestion. We agree that we must detail the conditions under which this algorithm can work well. We believe that our method can work in problems regardless of their dimensionality. We have studied 1D (time-based subaggregation) and 2D (space-based subaggregation), but we do not see why this method cannot easily be applied to higher dimensions by simply changing the way we sub-aggregate to be sensitive to the new dimensions. Furthermore, we believe there are many interesting problems that exist on a 2D plane that would be interesting to study in future work. For example, Go, Chess, and all the Atari benchmarks have only two dimensions in state space, yet would each come with their own unique challenges and deceptive-ness. 

Furthermore, we have added a new domain to the paper, which involves finding parameter values for a robotic arm. This is another example of a problem that exists on a 2D plane, however we think it presents new insights due to the structure of the search landscape.

Reviewer comment (Experiments):
I would like to see the proposed method applied in broader domains.

Author response:
Thank you for this suggestion. We agree that we must detail the conditions under which this algorithm can work well. We believe that our method can work in problems regardless of their dimensionality. We have studied 1D (time-based subaggregation) and 2D (space-based subaggregation), but we do not see why this method cannot easily be applied to higher dimensions by simply changing the way we sub-aggregate to be sensitive to the new dimensions. Furthermore, we believe there are many interesting problems that exist on a 2D plane that would be interesting to study in future work. For example, Go, Chess, and all the Atari benchmarks have only two dimensions in state space, yet would each come with their own unique challenges and deceptive-ness. 

Furthermore, we have added a new domain to the paper, which involves finding parameter values for a robotic arm. This is another example of a problem that exists on a 2D plane, however we think it presents new insights due to the structure of the search landscape.

Reviewer comment (Writing):
Easy read for somebody in EA community but should be more motivated for why the broader ML community should care.

Author response:
Thank you for pointing this out. We have added a more in depth description of our evolutionary algorithm used here to make this work more broadly understandable to people unfamiliar with the field. Furthermore, we have added an algorithm outlining our entire proposed system to help a reader follow along in more depth and to aid in reproducibility. We also include an algorithm to our appendix outlining the algorithm we are comparing to, MAP-Elites.

Reviewer comment (Writing):
MAP-Elites is not introduced when first mentioned in the introduction.

Author response:
Thank you for the comment. We agree that we did not sufficiently explain MAP-Elites as a baseline. We briefly introduce the algorithm in the main text to help with readability and we also provide a detailed description in the appendix for a curious reader to learn more.

Reviewer comment (Writing):
This description should be extended to make it very clear for the reader "The idea of lexicase selection is that instead of compiling performance metrics over the training dataset, we can leverage individual performance to sift through a population via a sequence of randomly shuffled training cases.”

Author response:
Thank you for suggesting this. We have included an algorithm outlining our entire procedure (including the lexicase selection portion), and improved our in text explanation to better outline the lexicase selection procedure.

Reviewer comment (Experiments):
More complex domains should be tested and the approach should be compared to RL-based approaches as baselines to increase its impact on the broader ML community.

Author response:
We appreciate this comment. We agree and believe that this technique can be applied to more complex RL-based approaches to help benchmark it in a more rigorous manner. However, we believe that the conclusions from this paper are broadly applicable due to the variety of domains (discrete optimization for knight touring and RL for the maze) studied. To help further demonstrate this, we add a fourth domain, the robotic arm, to the appendix. We hope that this work can motivate future work to try our implicit diversity maintenance in more advanced RL domains such as Atari.

Reviewer comment (Writing):
I would suggest not another “all you need” title...

Author response:
Thanks to your suggestion, we have opted for a new title, that expressed the conclusions from this paper well: "Deceptive Problems Can Be Solved Without Explicit Diversity Maintenance"

Reviewer comment (Presentation):
It should be noted in Figure 2 that n is the number of objectives in lex_n. Is lex_1 equivalent to not having any lexicase, just a standard selection?

Author response:
Yes, lex_1 is equivalent to not performing lexicase selection. In fact, it is equivalent to best-of-n hill climbing, where we simply select the best individual in the entire population based on the aggregated fitness value. We have added an explanation of the naming convention to the figure caption.

Reviewer comment (Writing):
Add a description of lexicase to the abstract/introduction.

Author response:
We agree with this idea and have added a short description of lexicase selection to the introduction. We hope that this, in combination with our included algorithm, will sufficiently outline the lexicase selection algorithm

Reviewer comment (Novelty):
Can this approach be extended to RL methods, which would significantly increase its impact?

Author response:
Thanks for the great question. In fact, the maze domains that we use requires performing search over parameters for a neural network policy, which is an evolutionary approach to performing RL. We have added specific details including neural network policy parameters to the appendix. For this reason, we strongly believe that this approach is compatible and broadly applicable to higher dimensional RL methods. As such, we believe that scaling up our technique to higher dimensional RL domains is trivial once a sub-aggregation is determined.

Reviewer comment (Experiments):
Could deceptive domains from OpenAI Gym/Atari be included? Montezuma’s Revenge is a classic example! The authors claim that objectives are all we need for solving any deceptive domain. A performance comparison of the proposed approach to other SOTA algorithms in these domains would make the claim stronger.

Author response:
This is a great question! Yes, we believe that this paper can easily be extended to larger domains with more complicated search spaces. However, as an initial proof of concept, we believe that the relatively simple domains are good for the diagnostic experimentation that we performed, paving the way for future work to extend our claims to larger domains.

Reviewer comment (Experiments):
The author(s) also claim that this approach enhances exploration in deceptive domains. However, it is also important to evaluate the performance of RL agents (trained by the proposed algorithm) in non-deceptive MuJoCo environments. This would validate the algorithm’s sensitivity to the choice of the sub-aggregation schemes.

Author response:
Thank you for the great idea. We have decided to include an entirely non-deceptive domain in our work to help demonstrate the sensitivity of our algorithm to non-deception. We would love to extend this to MuJoCo or other advanced RL domains in future work. Although the illumination maze is not inherently deceptive, we include another commonly used QD domain, known as the Arm Repertoire, that is entirely non-deceptive. In order to maximize the objective, the optimization algorithm should minimize the variance between angles of a 2D robotic arm. Our results show that when the domains are non-deceptive, our approach does not outperform MAP-Elites on the diversity focused metrics, but does outperform it on the quality-based metrics. This makes sense as the domains that are not deceptive are easy to maximize by simply following an objective, whereas other QD algorithms might spend too many resources exploring low quality regions of the search space (boosting the QD-Score). This further illustrates why our method works better than ME on deceptive tasks: it focuses more towards quality and uses diversity to drive search if/when it is needed, while ME equally balances diversity and quality regardless of their instrumentality in solving the problem at hand.

Reviewer comment (Reproducibility):
An algorithm is missing! The entire approach should have been formalized as an algorithm and written clearly in the main paper.

Author response:
Thank you for pointing this out. We have added an algorithm detailing our approach in Section 3. We hope that this makes it easier to follow our proposal!