Paper: Prioritized Soft Q-Decomposition for Lexicographic Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
The proposed method appears to be sensitive to the parameter ε, and the manual selection of this parameter is non-trivial.

Author response:
*Reviewer rq81 is concerned about the sensitivity of our method with respect to manually selected thresholds.*

Our method is indeed dependent on the manually selected $\varepsilon_i$ thresholds, however, this is the case for all $\varepsilon$-lexicographic MORL methods, since these thresholds are part of the problem definition. In the limitation section of our paper, we suggest multiple approaches to finding adequate values for $\varepsilon_i$ and note these are fundamentally problem-dependent, the same way that a reward function is.

See also our answer "Threshold ablation" to reviewer *tsYf*. 
We have included an additional ablation study with different $\varepsilon_i$ values in the revised manuscript (see Appendix H.3), to further illustrate how these scalars affect performance and agent behavior.
We further note that we describe how adequate $\varepsilon_i$ thresholds can be selected in Section 6 of the paper.

Reviewer comment (Experiments):
The paper falls short in providing a comparative analysis with existing lexicographic MORL algorithms.

Author response:
*Reviewer rq81 suggests additional comparison with lexicographic RL algorithms.*

As we write in the related work section, we believe to contribute the first algorithm for **continuous** action-space lexicographic MORL problems. As stated in our answer "8. Discrete PSQD version" to reviewer *SKG7*, a discrete version of our algorithm would be conceptually similar to existing works and we do not consider this comparison informative for learning in continuous spaces.

Reviewer comment (Theory):
The paper employs equation 7 to approximate ε-optimality based on equation 1, but the relationship between the two equations is not very clear. Including more theoretical insights could enhance the paper's rigor.

Author response:
Equation (1) defines a general lexicographic constraint and can also be found in related works on lexicographic RL. Equation (1) simply states that the optimization of subtask $i$ is constrained to a set $\Pi$ of policies, where all policies in $\Pi$ are also (near-) optimal to all higher-priority subtasks ${1, \dots, i-1}$. 

Since the explicit computation of $\Pi$ is not practical, especially for probabilistic policies with continuous action spaces, in Eq. (7) we instead make a state-based version of Eq. (1), where the performance measures $J_i$ are the Q-functions. This is also consistent with existing related work on lexicographic RL. 

We hope this clarifies the relationship of these two equations.

Eq. (1) and Eq. (7) are not equivalent. We primarily feature Eq. (1) to provide intuition for lexicographic RL as part of the problem definition.

Strictly speaking, Eq. (1) is a global constraint, where the thresholds $\varepsilon_i$ would be expressed on some global performance criteria like the value of the starting state or the policy's expected return. Eq. (7) is a local and state-based approximation of Eq. (1). Importantly, Eq. (7) is more restrictive, gives us more control over the agent's behavior, and the satisfaction of the constraint in Eq. (7) **implies** satisfaction of the constraint in Eq. (1).  When the lexicographic constraint is enforced in every state, as in Eq. (7), it clearly also holds globally and in expectation.

We will add the words *global* and *local* to avoid confusion about the equivalent of Eq. (1) and Eq. (7). We note again that the conversation of global Eq. (1) into local Eq. (7) is common practice (see 2022 paper by Skalse, Hammond, Griffin and Abate or 2022 paper by Wray, Tiomkin, Kochenderfer and Abbeel) and especially important for safety-critical applications.

Reviewer comment (Theory):
It's worth pondering whether εi should be state-dependent. Is it possible to find a constant εi that precisely represents the task requirements? If not, it might indicate that the action space is overly restricted in some states, hindering exploration in subsequent tasks, or, conversely, that undesired actions cannot be excluded in certain states.

Author response:
> It's worth pondering whether εi should be state-dependent.

This is indeed an interesting discussion and an interesting research topic.

However, we want to highlight that, although $\varepsilon_i$ are fixed scalars, the Q-functions to which these thresholds relate are very much state-dependent. In effect, this results in indifference spaces that are very large (i.e. permissive) in some states, while only being restrictive in states *where it really matters*. For example, when far from the obstacle, the lexicographic constraint on obstacle avoidance allows nearly every action, however when close to the obstacle, it forbids the selection of those actions that would lead to obstacle collision (since those actions are clearly sub-optimal for the obstacle-avoidance task).

This can also be seen in Figure 7 in the appendix, where we visualize the agent and the indifference space at multiple locations in the environment. Most of these are relatively close to the obstacle, nevertheless, the indifference space in Subfigure 7.h is much larger and more permissive than in Subfigure 7.d or 7.f, due to varying proximity between the agent and the obstacle.

Reviewer comment (Presentation):
Presentation needs to be improved. While the considered setting - lexicographic MORL - is clear, the flow of the algorithm is hard to understand. High-level pictograms can help the readers understand the content.

Author response:
We fully agree with reviewer SKG7 that pseudocode and pictograms will benefit understandability. 

Therefore, in the revised version of the manuscript in Appendix "D PSQD algorithm details", we have added algorithm pseudocode for the pre-training step, the adaptation step, a pictographic overview of our framework (Fig. 5), and additional details regarding our method. We will release our full codebase once the paper is accepted, which will benefit the understandability and reproducibility of our work.

Reviewer comment (Reproducibility):
There is no pseudocode, so I am confused about the implementation of the algorithm. As far as I understand, the proposed algorithm is one of the following:

Candidate 1) For subtask 1 to n-1, run parallel SQL in equation (12). Then restrict action space satisfying epsilon-optimalty of subtasks 1 to n-1. In the restricted action set, run SQL for subtask n in equation (12) and recover optimal policy using (13).

Candidate 2) For subtask 1, run SQL in equation (12). Acquire restricted action space A_1. Run (12) on A_1 and acquire A_2. ... After acquiring A_{n-1}, run (12) and (13) for subtask n.

Which one is right? Please provide a pseudocode.

Author response:
We fully agree with reviewer SKG7 that pseudocode and pictograms will benefit understandability. 

Therefore, in the revised version of the manuscript in Appendix "D PSQD algorithm details", we have added algorithm pseudocode for the pre-training step, the adaptation step, a pictographic overview of our framework (Fig. 5), and additional details regarding our method. We will release our full codebase once the paper is accepted, which will benefit the understandability and reproducibility of our work.

Reviewer comment (Theory):
For clarity of the setting, I want to raise several fundamental discussions regarding the lexicographic setting.  
- When lexicographic MORL setting is considered in practice?  
- What is the clear difference between constrained (MO)RL?  
- Who decides the priority order? Is it valid to assume that we always know the priority order?  
- How do orders of subtasks affect the final performance in experiments? If the order is crucial, discussion on setting order is important.  
- What if there are tasks that are not prioritized (e.g., "equally" important, or "we do not know")?

Author response:
+ 3.1 We believe lexicographic MORL has numerous benefits over traditional MORL which relies on scalarization: Lexicographic MORL algorithms allow for the transfer of simple subtask agents to complex lexicographic problems, benefits interpretability by decomposition, and offer constraint-satisfaction guarantees that can make for a safe exploration framework. Our method is for continuous action-space MDPs, unlike existing lexicographic RL methods for discrete action-space MDPs. We make an important, fundamental contribution by extending lexicographic MORL to continuous action-space MDPs, which have many applications in real-world problems, like robot control.

+  3.2 Lexicographic MORL can be seen as a special form of constrained MORL, where the constraints are of a lexicographic nature and for a chain-like directed, acyclic graph. This is not the same as, e.g. constrained policy optimization (Achiam et al, 2017), which places multiple constraints on the policy that are not lexicographically ordered. Furthermore, in lexicographic MORL, the constraints are based on the learned subtask solutions (e.g. Q-functions) and not based on manually defined functions, as is usually the case in constrained MORL.

+  3.3 The RL practitioner/designer decides on priority order. Priority order is part of the problem definition, in the same way that defining the MDP (state-space, action-space, reward function) is. Defining a lexicographic RL problem requires a priority order, in the same way that defining a regular RL problem requires a reward function. If the RL practitioner can not decide the order of two subtasks, it is possible to treat those subtasks with equal priority by simply summing those corresponding reward functions. We discuss this more in our response below to question 3.5.

+ 3.4 We have experiments with three subtasks in Appendix G.2. These experiments show that changing the priority order changes the resulting behavior, which is the expected result. Changing the priority order changes the problem definition, therefore comparing the performance of multiple such agents does not make sense. Related to this is also our new ablation study on the priority thresholds in Section H.3 in the appendix. 

+ 3.5 If tasks are of equal importance, then it is valid to sum their reward functions. We can then learn a single Q-function $Q_{1+2}$ for $r_1 + r_2$. We can use $Q_{1+2}$ like any subtask Q-function and define a corresponding $\varepsilon_{1+2}$ threshold for this Q-function. Keep in mind, however, that simply adding the subtask reward functions can produce unexpected behavior, especially when these subtasks Q-functions are semantically incompatible. We argue that practitioners should consistently order tasks by priority. If the tasks are compatible, then the lexicographic constraints do not hurt performance, and if they are incompatible, we have performance guarantees for the higher-priority tasks, which is not the case for MORL methods that don't use priority.

Reviewer comment (Experiments):
Experiments deal with only two subtasks. Can the authors show another environment containing more than two subtasks?

Author response:
We have experiments that use more than two subtasks in Appendix G.2.

Reviewer comment (Reproducibility):
For reproducibility, it would be better for the authors to provide anonymous source code.

Author response:
We fully agree with reviewer SKG7 that pseudocode and pictograms will benefit understandability. 

Therefore, in the revised version of the manuscript in Appendix "D PSQD algorithm details", we have added algorithm pseudocode for the pre-training step, the adaptation step, a pictographic overview of our framework (Fig. 5), and additional details regarding our method. We will release our full codebase once the paper is accepted, which will benefit the understandability and reproducibility of our work.

Reviewer comment (Writing):
There is confusion in eq. (7). Does Q_i in eq. (7) mean the optimally trained one (i.e., Q_i^*)?

Author response:
$Q_i$ in Eq. (7) refers to the subtask Q-functions of an agent that is lexicographically optimal for all $n-1$ higher-priority tasks. These are not optimal Q-functions $Q_1^*, \dots, Q_n^*$. We have changed the sentence preceding Eq. (7) to emphasize this.

Reviewer comment (Writing):
Confusing notation in eq (1). J_0, epsilon_0, Pi_0 is not explicitly defined.

Author response:
This is the standard lexicographic constraint consistent with related work. As we write in the paper $J_i$ are some performance measures (like value functions, Q-functions), the symbol $\pi$ is defined as policy, and the symbol $\varepsilon_i$ is defined, in the next line, a performance threshold scalar for subtask $i$.