Paper: Reward Design for Justifiable Sequential Decision-Making (ICLR-2024)

Reviewer comment (Theory):
Bottom of page 1, “the human as a suboptimal decision maker”: do you mean that the RL agent is optimal? But if optimality refers to the actions deployed in practice, the RL agent will not be optimal either, no?

Author response:
What we intended to say there is that a human may not act optimally or rationally in every situation, as it is inherently limited in capacity. To extend on this, our line of reasoning touched upon the fact that in certain situations, a human may not be able to comprehend the full state available to the acting agent, which may hinder its judgment. We also acknowledge your comment about optimality of the deployed agent. Indeed, the agent is learned, and will therefore also be suboptimal, although the referenced sentence does not imply any assumptions about this. Lastly, we have slightly expanded Sec. 3.1 to better clarify the assumptions made by our framework about optimality and choice of the baseline policy.

Reviewer comment (Novelty):
The authors in [A,B] also consider sequential decision-making (MDPs) influenced by a baseline policy and an agent to model partial adherence to an algorithmic recommendation. Is the setup in this submission close to the setup in [A,B]?

Author response:
Thank you very much for sharing these works. We have also updated the related work section to include them. Regarding [A], we find a conceptual similarity in that both approaches aim to improve what is assumed to be a given baseline policy. However, [A] operates in a context of expert-in-loop setting, where a baseline policy is considered to represent a human decision-maker and a policy to be learned (in their terms, recommendation policy) aims to adapt its actions to it. Another prominent difference is found in a way the environment evolves: in [A] both the baseline policy and the learned policy can influence dynamics of the environment. Differently, in our setup, baseline policy never influences state transitions: the justifiable agent is a sole actor in the environment. Likewise, authors in [B] examine a setting similar to the one in [A], which also exhibits mentioned similarities and differences. As a general note, both [A] and [B] consider a different problem setup, where the primary goal is to improve and adapt the AI policy to the baseline policy representing a human, which is the main decision-maker in the environment. In contrast, we consider a problem of improving the baseline policy in a setup where a learned (i.e., justifiable) policy is the main actor in the environment, and a baseline policy is only used as a reference point to improve upon.

Reviewer comment (Theory):
Definition of $R_{J}$ : what is the expectation over? The transition probabilities (since policies are deterministic)? Is the decision to be optimized $a_{t}$ or $a^{B}_{t}$ ?

Author response:
In the definition of the justifiability reward $R^J$, the expectation is over the trajectories, which are stochastic due to non-deterministic transitions. The decision to be optimized is the one coming from the justifiable policy, namely $a_t$, whereas the decision $a_t^B$ is coming from a (fixed) baseline policy. We have also slightly updated the paper to make this part more easily understandable to the reader.

Reviewer comment (Writing):
p4: the definition of $\mathbb{U}$ introduces the notation $\mathcal{J}$ but the next sentence uses $J$. Please clarify.

Author response:
Thank you for noticing inconsistencies in our presentation here. The definition of U introduces J as a model of a human judge. Later mentions also refer to the same entity. We have corrected the paper accordingly.

Reviewer comment (Experiments):
Section 4.4: can you please justify the exact architecture and neural nets used in the paper (two-fully connected layer, size 256, choice of parametric relu, etc.)? This section appears very ad-hoc, which it may be, and that is fine, but it requires some discussion/empirical justification.

Author response:
Thank you for raising this concern. The architecture and hyperparameters used for our argumentative agents and the judge model were obtained through careful hyperparameter tuning procedures, as we had no prior work to rely on. We did, however, include a full list of hyperparameter tuning ranges and further details about the model’s architecture (e.g., insights about including a binary mask as part of the judge’s input, usage of action masking, etc.) as part of our appendix. For policies treating sepsis, we designed our models on the basis of [1], and we have now also included a list of their hyperparameter tuning ranges in App. C.3.

Reviewer comment (Evaluation):
The most glaring flaw is that the paper gives its setup as being motivated by requiring agents' actions to be justifiable to humans, but the actual methodology is a few steps removed from humans: both in the training (the synthetic judge is *not* trained on a dataset of humans ranking actions by their justifiability, but instead on a dataset of ground-truth clinician actions) and in evaluation (the evaluations measure the synthetic judge's judgments, instead of human judgments). The paper would be stronger with at least human evaluations, if not human preference data for training.

Author response:
Thank you for the comment. We acknowledge that our results are reported using a learned, proxy judge, without conducting further human evaluations. In this work, we have focused on a particular domain of treating sepsis and aimed at performing an in-depth examination of the intricacies of our method, providing a proof-of-concept exploring a particular approach to learning to justify. Future work could provide further human evaluations, strengthening the claims made in this paper, and tackle the challenges in collecting of human preference data, which we briefly touched upon in the discussion section.

Reviewer comment (Writing):
The abstract says that "we showcase that agents trained via multi-agent debate learn to propose evidence that is resilient to refutations and *closely aligns with human preferences*." It is unclear which human preferences this refers to. Does it just mean that the actions taken are similar to the actual clinicians' actions? Does this presuppose that clinicians' actions == justifiable actions? If so, is this stated anywhere?

Author response:
Thank you for the question. This refers to the synthetic preference we designed in Sec. 5.1, which in turn presupposes that a clinician’s action is in fact a justifiable action. We have updated the paper to make this clearer to the reader.

Reviewer comment (Experiments):
The definition of the Preference Dataset (Section 4.1) seems strange: the preference indicates which of the two decisions is more justified by a given set of evidence in a particular state. The description does not make explicit what that set of evidence is, and it seems that the evidence should be included in the tuple. In the experiments, the preference dataset used does not fulfill this definition, since $p$ simply indicates the actions that the real clinicians took, not the action that they thought was more justifiable given the (sampled at random) evidence.

Author response:
Thank you for noticing inconsistencies with our presentation here. We have mistakenly referred to the evidence used during training of a judge. The preference dataset D, as it was intended to be defined in the section 4.1, simply indicates which of the two actions was more preferred/justified in a given state. We have corrected the typo, and we apologize for any confusion this might have caused.

Reviewer comment (Presentation):
Perhaps footnote 3 is important enough to not be a footnote. The reviewer had difficulty finding where $a^B_t$ came from in the experiments.

Author response:
Thank you for the comment. We have revised the paper accordingly.

Reviewer comment (Theory):
The method appears to require explicit knowledge of the optimal deterministic policy (called the baseline policy $\pi_B$) as part of the debate framework (unless I have misunderstood something). This seems like a big limitation in terms of practically applying this method to larger-scale problems.

Author response:
The method does require access to what we refer to as a baseline policy. However, this policy can be any well-performing policy, that is not necessarily optimal. The reason why we require a baseline policy to be well-performing is to avoid trivial solutions. In particular, if we consider a case involving a poor quality baseline policy, it might be easy, but useless, to be more justifiable than it. We have updated the paper to mention this.

Reviewer comment (Theory):
Sec. 4.2 mentions that the judge proxy model follows a 'Bradley-Terry' model (and provides a citation). This seems like an important point in the design of your method, but I'm not familiar with this model or the background or history here. Can you elaborate on why this model is chosen?

Author response:
The choice of a Bradley-Terry model was inspired by the seminal work that examined specification of the reward function for reinforcement learning agents via preference ratings over trajectories [1]. It is a standard model for estimating score functions from pairwise preferences. In our framework, it can be understood in the context of equating judge’s rewards with a justifiability ranking scale. In particular, the difference in judge’s reward of two actions estimates the probability that a human considers one more justified than the other, given a particular set of evidence.

Reviewer comment (Theory):
The training of the justifiable agent (and the argumentative agent) appears to use offline i.i.d. sampling of (s, a, a) tuples (Sec 4.3). This seems like it could pose over/under-representation problems for some MDPs with deep trajectories that might be less or more common (i.e. problems where the i.i.d. assumption is more heavily violated by the dynamics). Can you comment about this limitation of the method?

Author response:
Thank you for your comment. Regarding the argumentative agents (Sec. 4.3), it is important to acknowledge that in a general case of online RL there is a mismatch between a distribution of contexts used during their training (the i.i.d assumption), and a distribution of contexts that arises during the training of the justifiable agent. This distributional mismatch might affect the quality of the justifiable policy, since $\hat{r}^d$ might not approximate well $r^d$ in states that are much more frequently visited by the justifiable policy than by the behavior policy that generated the offline data. In our case, however, because we are operating in an offline reinforcement learning setup, the dataset used to train both argumentative and justifiable agents is the same, so we do not encounter such a distributional mismatch (i.e., states are sampled i.i.d during training of both agents). Regarding the justifiable agent, we believe that the limitations one could face in this setup are the same as those encountered in standard offline reinforcement learning problems [3]. Therefore, one could tackle out-of-distribution challenges using existing offline RL approaches, such as batch-constrained Q-learning [4] or Conservative Q-Learning [5].

Reviewer comment (Experiments):
Due to the additional structure of the debate game, the method introduces a number of additional hyper-parameters (e.g. evidence size $L$, mixing coefficient $\lambda$, scaling coefficient $\alpha$, as well as architecture of the various policy models, to say nothing of the learning algorithm hyper-parameters. Because of this, evaluation on one or more additional MDP problems would strengthen the evidence for the efficacy of this method.

Author response:
Thank you for your comment. In this work, we focused on an in-depth analysis within the domain of treating sepsis, with a goal of achieving a comprehensive understanding of the intricacies of our method in a setting where learning to justify is of great importance. We agree that additional evaluation in different domains will provide further insights into the efficacy of the method, and it represents a part of the future work.

Reviewer comment (Theory):
The use of an evidence-proposal debate setup for explainability is interesting and seems really promising to me. It seems like some deeper connections could be drawn to prior literature on the philosophical merits of debate as a form of explainability though. For instance, why did you choose to have a multi-step debate framework, rather than a single step of evidence proposal? What is the interpretation or significance of the trajectory through the debate game tree (is there any?), or is the reached leaf node the only relevant outcome of the debate?

Author response:
The multi-step debate was chosen to best reflect a real-world argumentative procedure and is also based on the prior work [8]. The multi-step debate, or more specifically, a full path in the debate game tree, holds additional information that can be used to judge the outcome. From the perspective of a human, seeing the order of proposed evidence might give additional information about their relationship and mutual level of attack/defense that can be used to issue judgments [14]. Note that we have not fully exploited the structure of the multi-step debate in this work. In particular, the judge model is invariant to different paths that all lead to the same leaf node. Future work could consider approaches that can leverage the additional information provided by the multi-step debate framework.

Reviewer comment (Evaluation):
How is a human RL designer or domain expert to interpret the proposal of multiple pieces of 'evidence' at each step of the debate game? If I understand correctly, evidence data are individual entries from the state-space vector (I think?). Is the interpretation of multiple pieces of evidence supposed to be a Boolean AND combination of each entry? (e.g. state feature A is present, AND B is present, therefore my action is better than the baseline policy)? A rich area for future exploration would be more structured evidence formats - e.g. "A is present, and C is present, but B is not present, therefore action X is chosen". Qualitative evaluations with domain experts would be beneficial in exploring this too.

Author response:
Thank you for this insightful comment. In this work, the actual interpretation of logical relationships between proposed evidence is underspecified and left at the “discretion” of a judge. You are correct to say that the evidence are individual entries from the state-space vector, those most supportive of the decision to be justified. This represents all information the judge has to make a judgment. At the moment, argumentative agents are not capable of outputting rich relationships between evidence, in the form you suggested (e.g., “A” is present, but “B” is not, or “A” is present and makes the previously proposed evidence “C” obsolete, therefore “X” is chosen). There is a rich line of work that examines argumentation, both in social sciences [6] and in the domain of artificial intelligence [7], including argument definition, their mutual relations and ways of provably defending a particular claim. Like you already suggested, we see ample potential in combining our framework with symbolic AI approaches that would structurally enrich the debate with additional information that would both ease the judging of evidence and enable argumentative agents to better justify their claims.

Reviewer comment (Theory):
There seem to be similarities between the debate-judge model and actor-critic RL approaches. Do you see any connections here? Could the proposed method be connected theoretically with A-C RL methods?

Author response:
Thank you for this very interesting comment. A potential connection we might see is in the existence of a judge model, which does act as a sort of critic: its judgments about justifiability of a made action are akin to “judgments” obtained from the critic in AC methods. However, while the two approaches might share similarities on a high level, there are notable technical differences. In particular, the critic in AC methods is based on return estimates, whereas in our framework, a judge model is used to directly assess the justifiability of an action in a given state.

Reviewer comment (Novelty):
As you note, the debate framework seems to have connections with pairwise preference ranking (e.g. Christiano et al. 2017). Given the popularity of RLHF at the moment, it might be illustrative to dig into the connections between this method and RLHF some more.

Author response:
Thank you for your comment. We point out two notable connections to RLHF [1]-[2] that are worthy of investigation in the future work. First, in our work we perform a single training cycle: the judge, argumentative and justifiable agents are all trained once. It would be interesting to examine a procedure akin to RLHF, where we repeat the training procedure for multiple cycles, each time aiming in improving the justifiability of the policy obtained in the previous cycle. Second, we note that RLHF is generally limited to problems/domains where a human can effectively judge the outcome given a complete state/trajectory. Methods such as debate [8], recursive reward modeling [9] or iterated amplification [10] all aim in alleviating this restriction. Future work could also examine connection of the proposed method with RLHF in an attempt of scaling to domains where a human cannot fully comprehend the complete state/trajectory.

Reviewer comment (Presentation):
Fig 1. It appears that the baseline policy $\pi_B$ never acts in the environment - is this correct?

Author response:
This is correct, $\pi^B$ serves to provide the baseline action, but does not otherwise influence the dynamics of the environment. The justifiable agent is the sole acting entity in the MDP.

Reviewer comment (Writing):
The paper refers to the debate-based reward approach as 'reward shaping' several times - please be careful with this term. Reward shaping refers technically to a specific subset of reward transformation that value functions are invariant to (see [A]). Although people often do abuse this term to refer to arbitrary designer interventions to alter environmental reward, it is best to be more careful with the terminology here to avoid confusion.

Author response:
Thank you for your comment. We acknowledge that we are slightly abusing the term “reward shaping”, as our “shaped” rewards do in fact induce learning of a new, justifiable policy, instead of preserving the original policy. We revised the paper to incorporate your feedback.

Reviewer comment (Theory):
I note the judge reward $J \in \mathbb{R}$ is unbounded - this concerns me as an infinite range within which to rank items poses difficulties for a human judge (see psychology and human factors literature for example). Does your method still work with bounds on the judge reward? E.g. $J \in [-1, 1]$?

Author response:
In our implementation, the judge is following the Bradley-Terry (BT) model, which in turn implies it is unbounded and real-valued. However, a human judge only gives a binary preference of the form “A is preferred to B” or “B is preferred to A” and does not otherwise specify any numerical value further describing this preference. The learned judge model we introduced in Sec. 4.2 is only a latent factor describing the human judgments, as estimated from the dataset.

Reviewer comment (Evaluation):
Can $\pi$ and $\pi_B$ output the same action and/or evidence? What happens in this situation? Is the judge proxy model able to output a 'no preference for either' choice?

Author response:
Thank you for the comment. We apologize for the confusion that may have occurred due to our mistake in writing of the utility function (Sec. 3.2), as described in our general response. Throughout the paper, we assumed that when two actions are equally justifiable (e.g., as in a case when they are equal) the debate game draws, ending in both players obtaining a score of 0.

Reviewer comment (Theory):
Sec 4.4 mentions self-play is used during training. However I thought the agents being trained are deterministic. How can self-play be used in this case?

Author response:
We use PPO to train the both, maxmin and self-play argumentative agents. During training, the argumentative policy is actually stochastic (i.e., we sample evidence from a categorical distribution defined by agent’s logits). During evaluation, we obtain a deterministic policy by taking the most likely action in a particular state (i.e., we perform the argmax operator over agent’s logits). We have updated the App. C.2 to describe this.

Reviewer comment (Experiments):
A compelling goal for future work would be the design of sample-efficient methods for specific MDPs that can operate with actual human-in-the loop judgements, rather than a proxy model.

Author response:
Thank you for the comment. We acknowledge that in this work we have not examined the sample-efficiency of the proposed framework. Future work could investigate the performance of argumentative agents trained with varying sizes of the preference dataset or, as you suggested, investigate approaches that can effectively operate with a human-in-the-loop.

Reviewer comment (Theory):
The Sepsis reward design (App. B.3) is important - please provide references for the choice of $C_0, C_1, C_2$.

Author response:
Thank you for noticing this. We have updated the paper to cite the relevant resource which was used to select the parameters.

Reviewer comment (Presentation):
Given the specificity of your empirical experimental case-study, it would be helpful to have some more elaboration of the nature of the problem of sepsis control in the Appendix. I also assume the authors have some prior work or expertise or domain knowledge in this area. It would be beneficial to state if this is the case in the appendix to demonstrate that the research is being undertaken by an appropriately credentialed and/or experienced team (i.e. not computer scientists pretending to be doctors).

Author response:
Thank you for your comment. The problem of treating sepsis, in particular by applying reinforcement learning, has received significant attention in previous years [11]-[13], and it is on this line of work that we build upon. Because our goal was not to beat the state-of-art method for treating sepsis, we were very careful in making any premature conclusions about the performance of trained policies, apart from what has already been done in the previous work. We have additionally added a more detailed description of a general problem of sepsis treatment in App. B.1, in a hope of providing a better context and a necessary background for the reader.

Reviewer comment (Experiments):
Sec 3.2 Why does $a_t$ act first in the debate game, not $a^B_t$? Did you test the reverse design?

Author response:
Actually, the starting player in the debate game is randomized, both during training and during evaluation. We briefly touch upon this in footnote 2. From the perspective of our framework, a starting player does not make a difference and only affects the order of evidence proposed.

Reviewer comment (Experiments):
You set the hyper-parameter $\alpha = 5$ in your experiments - why this value? Were other values tried?

Author response:
The value for the parameter $\alpha$ was obtained through the hyperparameter tuning procedure. We have updated the paper to also include the range of values we tried, in this case [1, 5, 10, 15].