Paper: Discovering Logic-Informed Intrinsic Rewards to Explain Human Policies (ICLR-2024)

Reviewer comment (Evaluation):
since the contribution is on logic-informed IRL, would be good to show and analyze the logic rules discovered along a trajectory as opposed to a single snapshot.

Author response:
**Q6: Since the contribution is on logic-informed IRL, would be good to show and analyze the logic rules discovered along a trajectory as opposed to a single snapshot.**

A6: Thank you very much for your valuable recommendation. You raise an important point regarding the potential insights that could be gained from analyzing the evolution of logic rules along a trajectory in our logic-informed IRL study. Currently, our methodology centers around a logic tree that primarily captures static conjunctions ('and' relations) among various logical components. Our predicates are not time-based, which means that our current framework might not be ideally suited to capture the dynamic evolution of these rules over time. However, your suggestion has certainly inspired us to consider the possibilities of integrating spatio-temporal logic rules in our future research.

Reviewer comment (Theory):
what is the numerical form of the tree? Eq. 6 only specifies the likelihood of the tree in terms of pre-order traversal sequence, but not the structure thereof (i.e. parent/children nodes).

Author response:
**Q7: What is the numerical form of the tree? Eq. 6 only specifies the likelihood of the tree in terms of pre-order traversal sequence, but not the structure thereof (i.e. parent/children nodes).**

A7: The numerical form of the logic tree is represented in two ways:
- Conjunction Clauses (Eq. 5): The logic tree comprises a set of logic rules, each written as a conjunction clause. These rules include positive and negative predicates, which are the conditions required to reach the goal.
- Energy Function Representation (Eq. 7): The logic tree is also conceptualized as an energy function. In this representation, trained parameters   can be interpreted as the weights of each rule in the logic tree. This representation captures the tree's numerical form, integrating both the structural logic and the significance of each rule within the framework.

Reviewer comment (Theory):
are the cardinalities of the index sets $I_k^1$ and $I_k^0$ same across $k$?

Author response:
**Q8: are the cardinalities of the index sets $I_k^1$ and $I_k^0$ same across k?**

A8: The cardinalities of  $I_k^1$ and $I_k^0$ indeed vary across different k. Specifically, $I_k^1$ corresponds to the indices of positive predicates within a rule, whereas $I_k^0$ relates to indices of negative predicates. This variability is intrinsic to the design of our rules and reflects the distinct conditions each rule captures within our logic-informed framework.

Reviewer comment (Reproducibility):
how is the node chosen for expansion and how is it expanded? By what criterion you'd know the tree cannot be further expanded?

Author response:
**Q9: how is the node chosen for expansion and how is it expanded? By what criterion you'd know the tree cannot be further expanded?**

A9: In our model, node selection for expansion is informed by the current state of the partial logic tree and input data. The expansion involves a classification process for predicates, where a decoder assesses the likelihood of each option. We also employ a pointer network to directly identify a predicate from the input, designating it as a logic tree's terminal node. Importantly, a gating mechanism, derived from the decoder's last feature, guides the decision to either expand or terminate, ensuring that expansion aligns with the tree’s current state and the available data.

Reviewer comment (Theory):
does the order of the predicates generated matter? at the same tree horizon?

Author response:
**Q10: does the order of the predicates generated matter? at the same tree horizon?**

A10: Thank you for your question. In our logic tree, the order of predicates at the same level does not affect the outcome since each is connected by a logical "AND" conjunction. The tree's structure dictates that the satisfaction of the root node condition relies on the collective satisfaction of all its associated leaf nodes, regardless of their order. This design ensures that the interpretability of the tree's decision-making process remains clear and logically sound.

Reviewer comment (Theory):
how is the goal $X^0$ decided, is it always success v. fail? Is it constant throughout a trajectory or changing over timesteps? If it's the former how can you make sure it is the right goal for all timesteps (one step of mishap would not necessarily reasult in an overall failure and vice versa), if the latter then how is the step-wise goal specified without too much human knowledge?

Author response:
**Q11: How is the goal $X^0$ decided, is it always success v. fail? Is it constant throughout a trajectory or changing over timesteps? If it's the former how can you make sure it is the right goal for all timesteps (one step of mishap would not necessarily reasult in an overall failure and vice versa), if the latter then how is the step-wise goal specified without too much human knowledge?**

A11: The goal $X^0$ in our model is conceptualized based on the desired outcome of the task, which may indeed be as binary as success versus failure in certain scenarios. It is generally fixed throughout a trajectory to provide a consistent objective. However, we understand that a single mishap does not necessarily dictate the final outcome, and our model accounts for this by considering the cumulative progress towards the goal, rather than stepwise success or failure.

Reviewer comment (Theory):
in estimating the overall energy function, since the tree generator is amortized, why you can use the top-K logic trees with generated probabilities as the probabilities may change later? And can you not use unweighted trees (e.g. taking average of all trees) to approximate the expectation?

Author response:
**Q12: in estimating the overall energy function, since the tree generator is amortized, why can you use the top-K logic trees with generated probabilities as the probabilities may change later? And can you not use unweighted trees (e.g. taking average of all trees) to approximate the expectation?**

A12: In our model, the top-K logic trees are selected based on their current probabilities. This selection reflects a snapshot of the most probable trees at a given iteration, which contributes to a more tractable optimization process. We acknowledge that probabilities may evolve; however, this approach allows us to approximate the expectation without the computational expense of considering all possible trees. As for using unweighted trees, it could lead to a less accurate estimate since it would not account for the varying significance of different trees. We prioritize weighted contributions to capture the nuances, which is more effective.

Reviewer comment (Evaluation):
The paper lacks a clear evaluation of the reward discovery aspect, which is claimed as a contribution in Section 4.2. While Section 4.2 provides informative content and follows prior work of deep PQR (Geng et al., 2020b), the central focus is on the relationship between the discovered rewards and the learned logic rules without concrete examples or evaluations to support this argument.

Author response:
Thanks for your suggestions. To evaluate the estimated reward functions, we will compare the decisions made by experts with the decisions predicted by the estimated reward functions. Although our benchmarks do not contain the decisions made by experts, we follow [1] to evaluate our method against GAIL and AIRL on the Swimmer dataset. We predict the decision-making using the estimated reward function. Numerical results are presented in the following table. Our method can also obtain comparable scores in the Swimmer dataset.

Table1: Results on swimmer tasks. Negative log-likelihood (lower is better) are reported across 5 runs.

|  Methods   | Negative log-likelihood  |
| :---  | ---:  |
| AIRL | 0.39 |
| GAIL | 0.34 |
| Ours | 0.26 |

[1] Fu J, Luo K, Levine S. Learning robust rewards with adversarial inverse reinforcement learning[J], 2017.

Reviewer comment (Evaluation):
There has been no evaluation conducted on the quality of the learned logic rules.

Author response:
Thanks for your advice. In the real-world dataset, there is no ground-truth of learned logic rules, so we follow [3] to verify our model’s rule discovery ability on synthetic datasets with a known set of ground-truth rules and weights. Note that it was originally utilized for the temporal point process, so we modify it by adding spatial variables (such as “left, right, front, and behind”) to fit in our settings. The weight learning results on 4 synthetic datasets are shown in the following table.

Table2: Rule discovery and weight learning results (GT weights/learned weights) on 4 synthetic datasets.

|  Weights   | Dataset-1  | Dataset-2  | Dataset-3  | Dataset-4  |
| :---  | ---:  | ---:  | ---:  | ---:  | 
| w0| 1.00/0.98 | 0.50/0.45 | 1.50/1.47 | 2.00/1.82 |
| w1| 1.00/0.91 | 0.50/0.40 | 1.50/1.44 | 1.00/0.97 |
| w2| 1.00/0.81 | 0.50/0.34 | 1.50/1.39 | 1.00/0.92 |

[3] Li S, Feng M, Wang L, et al. Explaining point processes by learning interpretable temporal logic rules[C]/ICLR. 2021.

Reviewer comment (Experiments):
For practical application of this approach, users are required to define predicate sets beforehand to facilitate the learning of logic rule-informed energy functions, but it is unclear whether an ablation study has been conducted to assess the algorithm's performance concerning the quality and suitability of the provided predicates.

Author response:
Thanks for your suggestions. Our generator takes the observed state-action demonstrations as input, and initially encodes them into the symbolic predicate space. So the input predicate has a great influence on the final results. Moreover, we removed different portions (denoted as p) of the input predicate, and the results are shown in the following table. we can see that when we remove some input predicates, the performance of R@10 and R@20 drops significantly.

Table 3: Ablation study of removing different portions (denoted as p) of the input predicate. Note that p=10% means that we remove 10% predicates from the framework. We show diagnosis prediction results on MIMIC-III using w-F1 (%) and R@k (%).
|  Metric | p=0%  | p=10%  | p=20%  | p=30%  | p=40% |
| :---  | ---:  | ---:  | ---:  | ---:  | --:  | 
| R@10 | 29.01(0.11) | 25.32(0.16) | 23.21(0.18) | 19.93(0.08) | 18.24(0.09) |
| R@20 | 38.10(0.08) |33.45(0.14)  | 32.81(0.07) | 31.64(0.11) | 30.83(0.06) |

Reviewer comment (Evaluation):
It is unclear whether the precision and recall of the learned logic rules have been assessed in comparison to expert decisions on the provided benchmarks.

Author response:
Thanks for your advice. In the real-world dataset, there is no ground-truth of learned logic rules, so we follow [3] to verify our model’s rule discovery ability on synthetic datasets with a known set of ground-truth rules and weights. Note that it was originally utilized for the temporal point process, so we modify it by adding spatial variables (such as “left, right, front, and behind”) to fit in our settings. The weight learning results on 4 synthetic datasets are shown in the following table.

Table2: Rule discovery and weight learning results (GT weights/learned weights) on 4 synthetic datasets.

|  Weights   | Dataset-1  | Dataset-2  | Dataset-3  | Dataset-4  |
| :---  | ---:  | ---:  | ---:  | ---:  | 
| w0| 1.00/0.98 | 0.50/0.45 | 1.50/1.47 | 2.00/1.82 |
| w1| 1.00/0.91 | 0.50/0.40 | 1.50/1.44 | 1.00/0.97 |
| w2| 1.00/0.81 | 0.50/0.34 | 1.50/1.39 | 1.00/0.92 |

[3] Li S, Feng M, Wang L, et al. Explaining point processes by learning interpretable temporal logic rules[C]/ICLR. 2021.

Reviewer comment (Experiments):
Sections 5.1 and 5.2 use different baselines, and the reason for this choice is not explained.

Author response:
The difference of baselines results from the different datasets. We compare our method with several baselines (e.g., NLM and MemNN) in the Blockworld dataset because their authors have evaluated the performance in the Blockworld dataset in published papers. Moreover, HiTANet and Chet are also representative baselines in the MIMIC dataset, so we compared our method with it to demonstrate our performance in healthcare event prediction and meanwhile provide strategic explanations.

Reviewer comment (Experiments):
It is unclear whether the evaluation was conducted in a fair manner for all the baselines. For example, NLRL does not utilize expert trajectories and learns directly from an MDP, while the proposed approach benefits from access to expert trajectories.

Author response:
Our evaluation was designed to demonstrate the enhanced learning potential when expert trajectories are incorporated, as opposed to a direct comparison of learning methods. While NLRL learns solely from an MDP, our approach leverages expert trajectories for a more guided learning process. This difference in methodology is crucial for the context of our study, as it highlights the advantages of integrating expert knowledge. The evaluation is thus fair within the scope of our research objectives, which is to showcase the benefits of using expert data in learning algorithms.

Reviewer comment (Evaluation):
Results in Table 2 on Diagnosis Prediction may be too close to edge out competing methods (Chet [1]) albeit being more interpretable by means of the Logic Tree.

Author response:
Thank you for your insightful comment regarding the results presented in Table 2. While the performance enhancements over Chet [1] may seem modest, the significant increase in interpretability achieved through our Logic Tree method represents a crucial advancement. In domains where the stakes of decisions are high, the ability to comprehend and trust the decision-making process is invaluable. Our approach underscores the importance of explainable AI, balancing a slight uptick in performance with a marked improvement in transparency. This balance is further supported by [2], which articulates the often necessary trade-off between precision and interpretability in model design.

[1] Context-aware Health Event Prediction via Transition Functions on Dynamic Disease Graphs, Lu et al., 2022

[2] Rudin C. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. Nat Mach Intell. 2019.

Reviewer comment (Reproducibility):
How does the algorithm handle redundant sets of nodes in a given Logic tree? Is there a pruning procedure?

Author response:
Our algorithm proactively minimizes redundancy in logic trees through the use of expertly crafted predicates, ensuring tree compactness from the outset. Additionally, our tree generator operates within an amortized framework, which facilitates the concurrent optimization of the tree structure and the parameters of an energy-based Generative Adversarial Network (GAN) during policy function estimation. This integrated approach effectively prunes unnecessary nodes, maintaining the efficiency and relevance of the logic tree representation.

Reviewer comment (Reproducibility):
How are the predicate variables determined? Are the functions of the observation space manually provided by the user for a given dataset? E.g. Above(x,y) in the Blockworld experiments

Author response:
Thank you for your insightful comment regarding predicate determination. In our approach, we rely on domain expertise, which is a widely adopted practice within this field, to guide the selection of predicates. While there is existing research on automated predicate invention, such as Meta-Interpretive Learning (MIL) [3], these methods often face limitations in complex domains like healthcare, where the intricacies of the data require expert knowledge. Predicates crafted with the input of medical professionals ensure reliability and relevance, leading to more meaningful and actionable logic rules. 

[3] Muggleton S H. Meta-interpretive learning of higher-order dyadic datalog: Predicate invention revisited[J]. Machine Learning, 2015.

Reviewer comment (Experiments):
Could there be any additional experiments showing the effect of input predicate set choice on the final result?

Author response:
Thanks for your suggestions. Our generator takes the observed state-action demonstrations as input, and initially encodes them into the symbolic predicate space. So the input predicate has a great influence on the final results. Moreover, we removed different portions (denoted as p) of the input predicate, and the results are shown in the following table. we can see that when we remove some input predicates, the performance of R@10 and R@20 drops significantly.

Table 1: Ablation study of removing different portions (denoted as p) of the input predicate. Note that p=10% means that we remove 10% predicates from the framework. We show diagnosis prediction results on MIMIC-III using w-F1 (%) and R@k (%).
|  Metric   | p=0%  | p=10% | p=20% | p=30% | p=40%|
| :---  | ---:  | :--: |  :--: |  :--: |  :--: | 
| R@10  | 29.01(0.11) |25.32(0.16)|23.21(0.18)|19.93(0.08)|18.24(0.09)|
| R@20  | 38.10(0.08)|33.45(0.14)|32.81(0.07)|31.64(0.11)|30.83(0.06)|

Reviewer comment (Evaluation):
The major weakness I see in the paper is in the framing of the problem as an interpretability problem, where authors acknowledge the lack of interpretability in black box policies and how in high-stakes domains. The logic rules are presented as the high level explanations that can provide interpretability. While this is a reasonable assumption, this is not detailed further in the paper, or discussed in the evaluation section. E.g. A large logic tree might not be inherently interpretable, where the selection of the rules to be explained can be important.

Author response:
A1: Thank you for your valuable feedback. In our paper, we constrain the size of the logic tree to enhance interpretability. Each branch within the tree uses a conjunction ("and") to maintain simplicity and directness in the reasoning process. Disjunctions ("or") are used between separate trees, indicating alternative reasoning paths to the root node. This structured approach ensures that only when all predicates in a path are true, a conclusion at the root is reached. Each distinct tree represents a unique reasoning pathway, simplifying the interpretability of complex decisions.

Reviewer comment (Experiments):
While there is a good selection of baseline methods compared, the synthetic/toy datasets need further additions of planning domains/datasets.

Author response:
A2: Thanks for your suggestions. To demonstrate the performance of our framework, we evaluate our methods in benchmark tasks (the shifting maze [1]) for planning datasets.  This task involves a 2D point mass navigating to a goal position in a small maze when the position of the walls is changed between train and test time. At test time, the agent cannot simply mimic the actions learned during training, and instead must successfully infer that the goal in the maze is to reach the target. In this task, a reward is learned via IRL in the training environment, and the reward is used to reoptimize a new policy on a test environment. We compare our method with several competing IRL algorithms, including NLM, MemNN, MaxEnt-IRL, Deep PQR and AIRL. The results are shown in the following table. Our method still obtains the best performance.  

Table1: Results on planning dataset (the shifting maze).  
|  Methods   | NLM  | MemNN | MaxEnt-IRL | Deep PQR | AIRL | Ours |  
| :---  | ---:  | :--: | :--: | :--: | :--: | :--: |  
|  Reward  | 393.5 |372.8|353.8| 440.6| 384.4| 468.3|  



[1] Fu J, Luo K, Levine S. Learning robust rewards with adversarial inverse reinforcement learning[J]. arXiv preprint arXiv:1710.11248, 2017.

Reviewer comment (Reproducibility):
What are the computation times for the MIMIC and synthetic datasets? It will be helpful for the reader to understand the computation requirements.

Author response:
A3: Following the reviewer’s suggestion, we show the runtime for all methods averaged on BlocksWorld tasks. Note that all methods are tested on an RTX 2080Ti GPU. The following table shows the time of 100 evaluation episodes. Our method is simpler to train at the same time achieving even better performance. 


Table2: Comparison of runtime.  
|  Methods   | NLM  | MemNN | MaxEnt-IRL | Deep PQR | AIRL | Ours |  
| :---  | ---:  | :--: | :--: | :--: | :--: | :--: |  
|  Runtime| 66.9s |101.3s|32.6s| 161.0s | 70.8s | 57.2s|