Paper: Divide and Conquer: Provably Unveiling the Pareto Front with Multi-Objective Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
There are some important baselines that need to be included, for example [1] and [2]. Furthermore, comparisons with GPI-LS did not demonstrate any improvements. I think that the authors should provide more discussion on this. It is hard to see why one would want to use IPRO over an existing method.

Author response:
With respect to the baselines, as mentioned in our general comment, GPI-LS is currently the state-of-the-art method for learning a convex hull. As the Minecart and MO-Reacher have (mostly) convex Pareto fronts, GPI-LS (which explicitly relies on this knowledge) presents a suitable “upper bound” on IPRO which does not make the simplifying assumption that the scalarisation function is linear.

To better convey why one would prefer IPRO over related work, we have introduced Table 1 in the introduction which summarises our contributions. In a nutshell, IPRO addresses a more general setting, discarding the assumption of linearity of the scalarisation function. This allows IPRO to also discover a richer front of Pareto optimal policies as well as tackle different policy classes. We have further attempted to improve the presentation of the paper by removing section 3.2.1 and adding the pseudocode for IPRO to the main text.

Reviewer comment (Presentation):
The presentation of the paper could use a major improvement. There are several places where definitions are either missing or unclear, making the paper hard to understand. The IPRO algorithm is described in words only in the main paper. I think Algorithms 1 and 2 in the Appendix should be included in the main paper to improve the clarity. Please see my comments in the next sections about other places for clarification.

Author response:
To better convey why one would prefer IPRO over related work, we have introduced Table 1 in the introduction which summarises our contributions. In a nutshell, IPRO addresses a more general setting, discarding the assumption of linearity of the scalarisation function. This allows IPRO to also discover a richer front of Pareto optimal policies as well as tackle different policy classes. We have further attempted to improve the presentation of the paper by removing section 3.2.1 and adding the pseudocode for IPRO to the main text.

Reviewer comment (Writing):
This may be a point I did not understand entirely due to the presentation issue. In Section 3.2, in describing the design of a Pareto oracle, the authors mentioned using GGF-DQN to optimize the objective function in Equation (1). My question is: what is the benefit of utilizing another multi-objective algorithm as a subroutine in IPRO compared to directly using it to optimize the multi-objective function?

Author response:
1. Utilising Another Multi-Objective Algorithm as a Subroutine: IPRO serves as a problem-agnostic algorithm, identifying target regions for undiscovered Pareto optimal points and delegating the task to a problem-specific subroutine. This approach allows for flexibility in handling various optimization tasks, extending beyond multi-objective reinforcement learning. The extension of the GGF-DQN algorithm allows for IPRO to be applied to the MORL setting, which we are particularly interested in, but it could indeed be applied more broadly to different domains.

Reviewer comment (Theory):
In Theorem 4.1, does one need to place some assumptions on the Pareto oracle to claim this result?

Author response:
2. Theorem 4.1 and Assumptions on Pareto Oracle: While additional assumptions are not explicitly placed on the Pareto oracle, we define the concept of a Pareto oracle ourselves, thus embedding certain desiderata into its definition. One example is that, we ensure that a Pareto oracle always returns an optimal point in its target region, but future work may be able to relax this definition.

Reviewer comment (Experiments):
IPRO needs to find the ideal and nadir vectors for initialization. In practice, even the reduced single-objective RL problem might not be solved to optimality. How does this fact impact the performance of IPRO?

Author response:
3. Impact of Single-Objective RL Solution Accuracy on IPRO: While this may indeed happen with certain single-objective methods, this need not be a problem for IPRO. For example, if it is known that the single-objective RL solver has some bounded error, we may simply adjust the nadir and ideal to account for this. On the other hand, we could also “assume” that the method used to generate the nadir and ideal always does so optimally. Then, IPRO will still obtain all points in the resulting bounding box but may fail to retrieve the points outside the bounding box.

Reviewer comment (Writing):
In the definition of Achievement scalarizing functions (Section 2), please explain how $s$ becomes $s_r$ depending on $r$.

Author response:
4. The definition of Achievement scalarising functions: We have updated the definition of $s_\mathbf{r}$ to better explain this. Intuitively, the ASF is instantiated with a referent $\\mathbf{r}$. We have also added a definition of the target region. Formally, the target region given a reference point $\\mathbf{r}$ contains all points $\\mathbf{v}$ such that $\\mathbf{v} \\succeq \\mathbf{r}$. We have added this to the preliminaries.

Reviewer comment (Writing):
In the Problem setup (Section 2), please define the expected return value $v^{\pi}$ rigorously.

Author response:
5. Definition $\\mathbf{v}^\\pi$: We have added a rigorous definition of the expected return value $\\mathbf{v}^\\pi$ in the Problem Setup (Section 2) for clarity.

Reviewer comment (Reproducibility):
How do you select the reference point \( l \) from the set of lower points \( L \) if there are many candidates?

Author response:
The question about selecting the reference point from the set of lower points is addressed in the appendix due to space constraints. In practice, we use a heuristic that chooses the point with the most "open space" (i.e. space not allocated to an excluded subset) dominating it. The intuition is that a reference point with more open space above it is more likely to lead to the discovery of a Pareto optimal point. It is essential to note that IPRO is not reliant on any specific selection method, and this heuristic is employed for practical efficiency.

Reviewer comment (Theory):
Since the convergence is only asymptotic, how do you show the advantage of your method theoretically? For example, one can optimize a weighted sum utility each time, and the weights are sampled uniformly. This method can also converge to the Pareto front asymptotically.

Author response:
We note that while convergence to the true Pareto front is asymptotic, we can guarantee convergence in a finite number of iterations to an arbitrary $\tau$-Pareto front with $\tau > 0$. The main benefit of IPRO is that we do not assume that utility is a weighted sum of the objectives, as opposed to Envelope, PGMORL, GPI-LS, and the vast majority of work in MORL. Rather, we allow for any nonlinear utility function. It has been proven that in the case of deterministic policies, the resulting Pareto front may be non-convex and hence methods which rely on linear scalarisation are inherently incapable of retrieving the full front. IPRO in contrast can find all points. We highlight this now in the introduction by adding a table which summarises the contributions of our work compared to relevant related work.

Reviewer comment (Theory):
Following Point 3, is it possible to characterize the convergence rate?

Author response:
While this is an interesting question, it is not straightforward as the convergence of IPRO is also dependent on the distribution of points on the Pareto front. It may be possible to characterise the behaviour of IPRO in the worst-case distribution of points along a Pareto front. We do not have any concrete results along this line but consider it an interesting topic for future work.

Reviewer comment (Theory):
What is the big difference between a weak Pareto oracle and an approximate Pareto oracle in practice? Is it only for theoretical rigorousness?

Author response:
The distinction between a weak Pareto oracle and an approximate Pareto oracle is primarily for theoretical rigour. However, both types of Pareto oracles may lead to different practical implementations. In the appendix, we describe the implementation of an approximate Pareto oracle using constrained MDPs, which would not be feasible with weak Pareto oracles.

Reviewer comment (Evaluation):
My main concern is that the proposed IPRO framework completely ignores one core problem of RL (and hence MORL) -- sample efficiency. Specifically, IPRO completely abstracts away the "learning from online interactions" aspect of RL through the use of Pareto oracles, which are assumed to be capable of directly returning a Pareto optimal policy given any reference point. However, this is actually the fundamental challenge of RL (and surely MORL as well). If the complexity of "learning from online interactions" is completely encapsulated into an oracle, then the problem would simply degenerate to a typical Multi-Objective Optimization (MOO) problem and then there is no need for reinforcement learning (That also manifests why the experimental results are all shown in terms of iterations, not in number of samples). With that said, the IPRO appears more like just an MOO algorithm rather than an MORL method (as it is agnostic to the specific problem structure of RL).

Author response:
We appreciate the detailed feedback from the reviewer and would like to address the raised points.

**Sample Complexity and Learning From Interactions**

We believe the reviewer strikes an important point. Indeed, given our current training setup, we train a new policy in each oracle call. However, it is likely that a single-network approach is also possible with IPRO. As our primary focus was on novel theoretical results in this challenging setting, we did not yet opt for this approach but consider it a promising direction for future work. 

As suggested, IPRO is indeed problem-agnostic and can thus be applied for general multi-objective problems. We are primarily interested in learning a Pareto front in MOMDPs where no explicit model is given, hence necessitating the use of RL as a subroutine. We plan to evaluate IPRO in related domains, such as optimisation and pathfinding, in the future.

Reviewer comment (Reproducibility):
Regarding the practical implementation of a Pareto oracle, a Pareto oracle could be achieved by taking the augmented Chebyshev scalarisation function as the objective under any off-the-shelf RL algorithm (e.g., DQN, A2C, etc). Notably, the augmented Chebyshev function depends not only on the reference point but also  $\lambda$ and $\rho$, which are the weight vector and the augmentation coefficient that determine the required improvement of each dimension. The choices of $\lambda$ and $\rho$ are critical in determining which Pareto optimal solution we are looking for (similar to the preference vector in linear scalarization). However, in IPRO, it is not explained (either in the main text of the pseudo code in Appendix) how these parameters are determined in each iteration.

Author response:
**$\lambda$ and $\rho$ parameters**

We note that $\lambda$ is a weight vector that is supplied by IPRO and is computed as $(\mathbf{v}^\text{i} - \mathbf{r})^{-1}$ where $\mathbf{v}^\text{i}$ is the ideal and $\mathbf{r}$ the referent. This normalises the improvement of a vector $\mathbf{v}$ relative to the referent $\mathbf{r}$ by the theoretical maximum improvement at the ideal $\mathbf{v}^\text{i}$. The purpose of this normalisation is to maintain a balanced scale across all objectives, preventing the dominance of one objective over another. $\rho$, on the other hand, is a hyperparameter which we tune.

Reviewer comment (Reproducibility):
In IPRO, it is not explained how to construct the sets $L$ and $U$.

Author response:
**$L$ and $U$ sets**

The construction of the sets $L$ and $U$ is now detailed more explicitly in Appendix B1. Intuitively, this is a sequential process that starts from the initial $L$ and $U$ sets. Then, every time a new point is added to the Pareto front, we can compute the added “corners” of the boundaries of the dominated and dominating sets. These new corners are then added to $L$ and $U$. For a visualisation, we refer to Figure 2. Pseudocode is also presented in Algorithm 2.

Reviewer comment (Reproducibility):
It is unclear what has been done in the Pruning phase and how Pruning uncovers additional Pareto optimal solutions.

Author response:
1. The use of pruning: Pruning does not uncover additional points, but rather rejects points which are not Pareto optimal but were added during IPRO’s execution. This step may be safely omitted without changing anything except possibly returning an unnecessarily large set.

Reviewer comment (Reproducibility):
It is not explained how the memory-based policy is implemented or whether there is any special neural network architecture involved.

Author response:
2. Memory-based policy implementation: We concatenate the memory (which is the accrued reward until the given timestep) to the observation and feed this into the neural network. This induces what is known in the MORL literature as an “augmented state space” and is a common approach.

Reviewer comment (Writing):
What are the definitions of “nadir” $v^n$ and “ideal” vector $v^i$ in Figures 1 and 2? (They seem to be some uniform lower bound and upper bound of all possible return vectors?)

Author response:
3. Definitions of “nadir” and “ideal” vector: The nadir $\mathbf{v}^\text{n}$ is defined as the “greatest lower bound” while the ideal $\mathbf{v}^\text{i}$ is defined as the “least upper bound”. To give an example, let $PF = \\{(1, 2), (2, 0)\\}$ then the nadir is $(1, 0)$ while the ideal is $(2, 2)$. These quantities are defined in the preliminaries.

Reviewer comment (Theory):
What is the reason behind the design of a* in eq (3)? Is this design theoretically grounded?

Author response:
4. The design of $a^\ast$ in eq (3): Intuitively, this definition “mimics” the optimisation objective of the agent where it aims to optimise its scalarisation function applied to the expected returns of the policy. While this update does not exactly represent this objective, it is common in the MORL literature and works well in practice.

Reviewer comment (Evaluation):
In Fig 3(e), the coverage of DQN as Pareto oracle is generally higher than the other two while the hypervolume of DQN as Pareto oracle is the lowest in Fig 3(b). Why is that the case?

Author response:
5. The discrepancy between coverage and hypervolume: The DQN agent fails to find a Pareto optimal point in some target region and then incorrectly excludes the search space in the region from consideration. As such, it has a higher coverage while still resulting in a lower hypervolume. For future work, we plan to extend IPRO to incorporate uncertainty over which sections of the search space may still contain optimal points rather than the current approach of including/excluding search space.

Reviewer comment (Reproducibility):
What is the main usage of Section 3.2.1? Is the stochastic stationary policy used in the implementation?

Author response:
6. Usage of Section 3.2.1: We intended to inform the reader that the concept of a Pareto oracle is feasible to implement and comes with theoretical grounding for specific policy classes. We did not use it in the remainder of the text. However, most reviewers agreed that this section was unnecessary and complicated the understanding of our paper. We have therefore moved this section to the appendix and added the pseudocode of IPRO to the main text instead.

Reviewer comment (Experiments):
Despite the novelty of the idea, the major concern is that the experimental evidence in the paper is not sufficient to demonstrate the advantages of the method compared to other MORL algorithms. First, the benchmarks used are relatively simple, and there are no experiments on complex multi-objective tasks, such as the classic MO-MuJoCo tasks. Additionally, there is a lack of comparisons with relevant MORL baselines, such as Envelope [1], PGMORL [2], and Q-Pensieve [3].

Author response:
Addressing the concern about experimental results, we conducted an extended hyperparameter search, resulting in significant improvements across all experiments. In addition, it is worth noting that the Reacher benchmark is a MuJoCo environment and that the remaining multi-objective MuJoCo environments in MO-Gymnasium have continuous actions. While IPRO is agnostic to the underlying problem and thus could handle continuous actions, this would require significant additional contributions as there has been limited work tackling such complex continuous control settings with nonlinear scalarisations. Finally, we did not compare against Envelope, PG-MORL and Q-Pensieve as GPI-LS has been reported to reach state-of-the-art results on the chosen benchmarks.

Reviewer comment (Presentation):
Further improvements are needed in the presentation of the paper. For instance, the definitions in Chapter 4 could be introduced in Chapters 2 and 3, rather than gradually unfolding them within later sections.

Author response:
As stated in our general comment, we have moved Section 3.2.1 to the appendix and added pseudocode for IPRO to the main text. We believe that this improves the presentation.

Reviewer comment (Reproducibility):
I strongly recommend the authors to include pseudocode in the main text.

Author response:
As stated in our general comment, we have moved Section 3.2.1 to the appendix and added pseudocode for IPRO to the main text. We believe that this improves the presentation.

Reviewer comment (Experiments):
Could the authors provide a performance comparison between IRPO and other MORL algorithms on MO-MuJoCo?

Author response:
Addressing the concern about experimental results, we conducted an extended hyperparameter search, resulting in significant improvements across all experiments. In addition, it is worth noting that the Reacher benchmark is a MuJoCo environment and that the remaining multi-objective MuJoCo environments in MO-Gymnasium have continuous actions. While IPRO is agnostic to the underlying problem and thus could handle continuous actions, this would require significant additional contributions as there has been limited work tackling such complex continuous control settings with nonlinear scalarisations. Finally, we did not compare against Envelope, PG-MORL and Q-Pensieve as GPI-LS has been reported to reach state-of-the-art results on the chosen benchmarks.

Reviewer comment (Writing):
What do "occupancy measure" and "occupancies" mean in Section 3.2.1? An introduction is lacking.

Author response:
We moved this section to the appendix to avoid confusion. To briefly clarify, an occupancy measure intuitively denotes how frequently a specific state is visited when employing a particular policy. This metric is commonly used in concave utility RL and for deriving theoretical results in classical RL settings.

Reviewer comment (Reproducibility):
How long does the RL optimization process before pruning take? More detailed training specifics are needed.

Author response:
Each iteration of the Pareto oracle runs for a fixed number of steps, which is reported in the appendix. After the entire execution of IPRO has concluded, we perform one pruning step to remove weakly Pareto dominant solutions. While not strictly necessary, this step avoids presenting unnecessary solutions to decision-makers. To enhance clarity, we have removed the mention of pruning from the main text, addressing this aspect instead in the appendix. Additionally, we've expanded the appendix to provide additional details regarding the algorithm (Appendix B1) and training procedure (Appendix D).