From: robot-learning.org/CoRL/2022/Conference/Paper306/Reviewer_GMPv
title: Thanks for the response
comment: Thank you for carefully responding to my comments and addressing them in the revised version of the paper. I find the revised manuscript much stronger and remain favorable for acceptance.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: Further concerns / questions / comments?
comment: Dear reviewers,

we thank you for all your very valuable reviews and comments so far. Since we are approaching the end of the rebuttal phase we wanted to ask if there are any more comments, questions or suggestions (new or follow-up) that we can address to improve the paper?

Thanks!

From: robot-learning.org/CoRL/2022/Conference/Paper306/Reviewer_LhcT
title: Thanks for the response
comment: Thanks for the responses and for amending the paper. The added limitations and discussion does help to contextualize the contribution. While not groundbreaking, I believe the work formalizes an issue with evaluation that should be addressed. The paper still falls short of providing/testing a concrete solution so the contribution remains limited. I lean towards acceptance, partially because this work is non-standard. Rather than propose yet-another method, this paper takes a step back and asks: are we really comparing and optimizing the right metrics?

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: Phrasing, Typos
comment: "Line 33: the wording says high (but not optimal) but before explaining the measure I think it makes more sense to say that the parameters values are very similar (but not identical)"

That is a good catch, thanks for pointing it out. We also want to thank the reviewer for pointing us to some typos throughout the manuscript.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: The paper needs some explanation of how prior works have avoided the pitfalls presented by the paper.
comment: "The paper needs some explanation of how prior works have avoided the pitfalls presented by the paper."

"Given that there are these possible worst-case scenarios, the paper could talk about how existing works (the first 15 references) avoid these worst-case scenarios or if there are published instances in the sim2real literature where this might be part of the issues."

Thanks for the feedback. Unfortunately, it is very difficult to gain insight about how existing work avoids these worst-cases, since such characteristics of the used measures are not discussed in the cited work. We purposely included the Driver experiment which has been widely used in previous work. Our results show that the issues described in our paper are strongly present in this experiment. However, we would like to not make any claims about previous experiments being invalidated without carefully redoing the exact same experiments. Also notice that the unreliability of the measures reported in our paper can be both, an over- and an underestimate of the actual performance.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: More technical disussion
comment: "The paper lacks a discussion of what aspects of the measures make them fail. It just states that they do. The discussion poses very generic questions."

"More discussion of either the specific attributes of the measures that lead to failure in the given cases or how one might approach a reward learning problem to avoid the 'worst-case' scenarios is needed."

Thanks for pointing this out. We deepened the discussion in two parts: At the end of the theoretical analysis, we added a subsection discussing the non-linear relation of estimated weight and collected reward. Further, we extended the future work section at the end of the paper. One of our proposed future work directions is an adversarial selection of test instances when using reward-based measures. This would allow for getting a tighter bound on the test error, and thus avoid the worst-case results.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: There’s no robots? This seems like it may not be the best fit for CoRL. The criticisms of reward learning could be applied to many different learning problems. The paper does not make a convincing case as to why it is critical to robot learning in particular.
comment: Since this comment is closely related to a comment by ktP2, we would like to refer to our respective response to the Area Chair.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: Reply to the reviewer
comment: We want to thank the reviewer for their helpful comments. We revised the paper, and respond to the individual concerns below.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: On a related note, the paper falls short of analyzing why and when the reward/parameter measures fail to be meaningful.
comment: Thanks for the recommendation. We addressed this in two ways: We added a brief discussion that our key finding is essentially that reward is a non-linear function of the estimated weights, and that this function is differs between instances. Moreover, we also reflect on this in the limitation section.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: The multi-scenario setting in Sec 3.2 points to mis-modeling of the problem; the path length and jerk are not invariant to the goal location? Is it that the rewards are transferable from train to test, but the relative difference between accumulated rewards is not preserved?
comment: To make sure we understand correctly: Is the question pointing at the fact that the reward $R$ collected by $w^{user}$ varies between the instances, as indicated in Table 1?
    Indeed, path length and jerk are not invariant to the goal location. Normalizing features for each instance could ensure that the reward collected by $w^{user}$ is the same (i.e., $1$) across all instances. However, the relative reward, or absolute regret will still differ between instances, causing the reported failcase that one estimate is better in Instance 1, but the other is better in Instance 2.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: The paper would be more impactful if the authors can discuss the implications of their work, e.g., if previous comparisons in the cited works are invalidated?
comment: This is a very good question. Given the fact that the discussed measures are used in many HRI works, we would conjecture that various simulation experiments are potentially affected by the reported problems. 
To relate to previous experiments by other researchers in the field our numerical experiments include the Driver experiment, which has been used in numerous reward learning papers. 
However, without redoing previous experiments, it is hard to asses how severe issues are, and we would not want to make any claims about previous results being invalidated. 
For instance, in Section 5.1 (line 211) we discussed the case that alignment is an underestimate -- thus, reported results might appear to perform more poorly than they actually do.
Since we do not know in which case previous experiments fall, we cannot make any claims that previous evaluations are invalidated. Instead, we hope that our analysis can help to design better measures for future work in the field. Better measures could then be used to re-evaluate the performance of different existing learning algorithms.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: Reply to the reviewer
comment: We would like to thank the reviewer for their careful comments. We updated the paper, and provide responses to the individual comments below.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: Include a deeper discussion of the literature and a more insightful closure at the end of the paper. See my notes above.
comment: Both parts have been extended.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: Is there a reason why the server experiment is not featured in Fig. 2? Is it not interesting?
comment: A plot for the server experiment was simply left out to allow for large and readable figures. However, the plot is included in the supplementary material.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: How were the five problem instances in Sec. 5.1 decided? How much they vary and how representative are they?
comment: We hand-designed a set of problem instances to ensure that they i) vary sufficiently from one another, and ii) are not trivial (e.g., just going straight in the Dubins experiment). In detail, the Dubins instances vary in the goal location and the position of the human, the Driver instances vary in the initial position of the autonomous and human driven vehicle, and the server instances vary in which of the recorded real-world trajectories are blocked during execution.

The user preferences $w^{user}$ were generated randomly.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: The authors discuss this briefly but it seems to me that one fundamental limitation of this whole space of approaches is the assumption that $w^{user}$ is somehow available when in the whole field of HRI this is most likely not the case.
comment: Thanks for the feedback. Indeed we only focus on evaluation in simulation, where ground truth weights $w^{user}$ are known. We stated this limitation in lines 20-21. However, to make it more explicit, we revised the first paragraph of Section 2.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: This seems to be in part a "statement" paper with relatively limited novelty. The main point is definitely intriguing and interesting but the key insight is relatively straightforward and the final discussion is quite limited.
comment: Thanks for the feedback. Following similar suggestions from other reviewers, we have added a new subsection in the theoretical analysis, giving additional insights into the weight-reward relationship. Furthermore we have extended the discussion section at the end outlining different approaches to tackle the challenges.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: The paper does not discuss related work but rather lists a large set of references. Most of the references are coming from 1-2 groups.
comment: Thanks for the comment. We now include a specific related work section that looks at papers that have also encountered some of the key issues we report on. Furthermore, since space limitations prevent us from providing a proper, broader survey, we decided to avoid that term to not create false expectations.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: Response to the reviewer
comment: We want to thank the reviewer for their careful and insightful comments and appreciation of our work. We addressed all comments in the revised paper, please also find our individual responses in separate comments here.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: End of section 6 has some interesting ideas, but they seem very difficult and it would be nice to provide some ideas for how they could be accomplished. How would the robot modify its reward estimate. Also, future work direction (1) seems like it is already done in the literature. Many works on active learning select training instances.
comment: Thanks for the comment. We revised the future work discussion significantly. For direction (1), we acknowledge that it is common in ML, and give some more details for an adversarial approach. When actively choosing test instances, the test error might be bounded, which could help to overcome Theorem 1.
We added a second direction about modifying features to address the issues, and extended a bit on the third direction about modifying a reward estimate.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: Presentation recommendations
comment: We agree that this would be very beneficial, but is infeasible with the space constraints. However, the Dubins experiments are very similar to the ones shown in Figure 1, but with a static human present in the environment. The Driver experiment comes from published work and is frequently used in the reward learning community. Finally, the Server task is adapted from a dataset from published work, a detailed description can be found in the respective paper.

We consider adding these descriptions to our supplementary material.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: In table 1, R^rel is defined but I do not find any definition of R^abs  \\ Line 117: [1] and [2] above provide ways of knowing whether the robot will solve the task, even if w' != w^user  \\ Line 160: Shouldn't this be -N + 1? [looking at the values in line 155]
comment: Thanks for pointing out the oversight about $R^{abs}$! The notation for the absolute regret $R^{abs}$ has been added.


 "Line 117: [1] and [2] above provide ways of knowing whether the robot will solve the task, even if w' != w^user"

We thank the reviewer for their careful consideration. Yet, we think that the results of [1] and [2] (reviewer's reference numbers) are specific to MDP / RL formulations. Thus, there might be no information in general, for instance in the our illustrative example and Theorem 1. However, to reflect the results from [1] and [2], we rephrased the sentence.

" Line 160: Shouldn't this be $-N + 1$?"

Thanks for reading the paper so carefully, this is a very good catch! The mistake is actually happening in line 154, where $T^A$ and $T^B$ where flipped. We changed the proof accordingly, resulting in $-1+N$. The proofs are now also in the appendix out of space concerns.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: The paper mentions other measures in 2.3, but does not evaluate them. If this is designed to be an analysis and survey paper, then I think these should be included for study.
comment: In the format of a conference paper, it is unfortunately not possible to extend our analysis to these measures. The measures mentioned in 2.3 are quite different from the two main groups in the paper, since they do not have access to the ground truth. Thus, the discussion would be orthogonal to our current analysis simply not fit the 8 page format

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: comparing reward parameters is not a good way to evaluate // no mention is made of the common practice to test learned rewards in transfer domains
comment: We agree that it is common practise in machine learning to use separate test and training data, and the two references show that this is also the case for IRL techniques. Levine et al. use different random test domains in their evaluation to measure how well a reward function is learned. While this is a more robust evaluation than only considering the training setup the fundamental issue that we characterized in Theorem 2 (Unbounded Test Error) remains.
The work of Fu et al. has now been included in the related work, and is also referenced in the future work. 

"Recent work has looked at characterizing learned reward functions"
We would like to thank the reviewer for pointing us to these citations. The listed references are indeed related to our work, we include them now in the related work discussion. However, we would like to point out that these works are only focusing on RL settings. Yet, the problem of learning reward (or cost) functions that represent user preferences is present beyond RL.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: It is unusual to have a reward function that maps directly from trajectories to rewards. Most work considers $R(s)$ or $R(s,a)$.
comment: We believe that there are two slightly different concepts of reward functions. While in RL reward functions are usually formulated over states or states and actions, reward functions in HRI often describe how a user evaluates the entire robot behaviour, often in the form of the executed trajectory. We would like to refer to the works in [1], [2], [3], [5], or [11] to name a few where reward functions take trajectories as an input.
To clarify, we slightly rearranged the first paragraph in Section 1.1 to emphasise the HRI context.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: The analysis seems overly simple and does not seem to contribute to the field---many of the claims made are commonly understood in the field. 
comment: While we agree that many researches in the field might have come across the issues described in our paper, we think that an explicit and detailed description would be an important contribution to the field.
Parameter-based measures are widely used - yet, researches do not study their relation to the collected reward, the primary objective in these problems.
Similarly, the transfer problem is well understood in the machine learning community. Yet, this is often not (sufficiently) reflected in HRI experiments.
We hope that our theoretical and numerical analysis can bring some more spotlight these issues, and thus help that the research community addresses them by designing better measures.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: missing related work // not broad enough to qualify as a survey
comment: We have significantly extended the discussion of related work, following your suggested references.

Related to our response the Area Chair, we agree with the reviewer's critique. While we have extended the related work, we do not want to create false expectations and have thus removed the wording 'survey'.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: theory seems overly simple (focused on simple 2 feature examples) and it is unclear how useful the worst-cast analysis is
comment: The examples in Section 3 are chosen to be as simple as possible to illustrate the issue. We could generate more complex examples with higher number of features, but that would distract from the key point.

The worst-case analysis shows that without any further knowledge of the planning problem, the mismatch between parameter-based measure and collected reward can be arbitrarily bad and similarly rewards in training can be completely uninformative about rewards in testing.
Such a worst-case is not necessarily encountered in practise. Yet, it is important to be aware of the worst-case outcome to not be overly confident about any results. This is similar to other worst-case characteristics such as heuristic solutions to NP-hard problems -- the solution can be good quite often, but there is no guarantee that it is not arbitrarily poor.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: not focused on robotics
comment: Since this comment is closely related to a comment by PQx1, we would like to refer to
our respective response to the Area Chair.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: Reply to the reviewer
comment: We would like to thank the reviewer for their insightful and detailed comments, this helped us  significantly to improve the paper.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: There is no limitations statement, which is a required component.
comment: We added a limitation section, discussing three major limitations of our work.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: A couple of reviewers questioned the connection to robot learning in particular.
comment: We respectfully disagree with reviewers ktP2 and PQx1 that the paper has no -- or only a weak -- connection or relevance to robot learning. The problem of learning a user's reward function -- and thus evaluating learning algorithms -- is studied extensively in the HRI community, as indicated by the vast majority of our references coming from robotics venues (RSS, CoRL, HRI, ICRA, IJRR, IROS). Our work hopefully helps researchers to improve evaluation methods for simulation experiments.

The reviewers point out that the issue we describe in our paper might apply to other machine learning problems. We believe that this might be the case. However, we wrote our work with a clear focus on common evaluation methods used in HRI -- the field we are familiar with -- and think that CoRL is a great publication venue to draw attention from the robot learning and HRI community to our findings.

Moreover, reviewer ktP2 pointed out that the issues described in our paper are commonly understood in the field. While some researchers might have encountered these issues, there seems to be little open discussion on how to deal with them, especially with the mismatch between parameter-based measures and actually collected reward.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: Although the paper bills itself as a survey [...]
comment: We thank the reviewers for their valuable comments regarding this. We agree that our paper does not qualify for a survey. Unfortunately, space limitations prevent us from a broader review without significantly shortening the theoretical and numerical analysis. Therefore, we address the concerns in two ways: i) We added a related work section that discusses other work addressing the two main issues described in our paper (training-test-transfer and parameter-reward mismatch). ii) we avoid the word survey to not create expectations on which our paper cannot deliver.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Authors
title: The main critique is that reviewers feel that the paper doesn't go deeply enough [...]
comment: We thank the reviewers for their interest in how learning measures can be improved.
We address these concerns at two parts of the revised paper: In the theoretical analysis we have a new subsection that characterizes the relation of weights and rewards, and we extended the discussion at the end of the paper.

From: robot-learning.org/CoRL/2022/Conference/Paper306/Area_Chair_UELd
metareview: Post discussion update
------------------------

The authors completed a rebuttal that addressed all of the reviewers' critiques, and also uploaded a revised and improved manuscript. The most significant improvement seems to be the addition of a small related work section in the introduction and a limitation plus future work section at the end of the paper. However, since the final paper has an 8 page limit, the current 9-page manuscript will need to be shortened. 

The clarifications seemed to sway two of the reviewers toward increasing their ratings. Generally, it's clear that although this is a non-standard paper, it raises interesting and potentially provocative points that would be great for the CORL community to discuss. Therefore, I am recommending acceptance.



Original review
----------------


This paper addresses the HRI challenge of measuring how well a robot has learned a user's reward function. It describes two main types of measures---parameter-based and reward-based---and provides both theoretical and empirical examples of the shortcomings of each. 


Strengths
* This seems like the type of paper that would be useful to assign in an HRI course or give to a new grad student, because it clearly explains the different measures and provides a comparison of them.

* Reviewers noted that the issues brought up by this paper are important and very relevant to current approaches. The paper is timely.

* Reviewers agreed that paper balances well between theory and empirical examples. 

* The paper was well written and easy to understand.


Weaknesses

* The main critique is that reviewers feel that the paper doesn't go deeply enough into the discussion of what to do with these findings. How should robotics researchers use this information to design better measures? In what cases are these measures likely to fail, or to be accurate? Section 6 poses such questions, but it would add substantially to the novelty of the paper if there was some suggestion of where to go from here.  

* Although the paper bills itself as a survey, reviewers noted that it doesn't provide much of a discussion of related work. Reviewers also provided additional citations that address some of the issues brought up regarding current measures of performance. 

* A couple of reviewers questioned the connection to robot learning in particular.

* There is no limitations statement, which is a required component.
recommendation: Accept (Poster)
confidence: 4: The area chair is confident but not absolutely certain

From: robot-learning.org/CoRL/2022/Conference/Paper306/Reviewer_ktP2
summary: This paper analyzes and evaluates several different measures for evaluating reward learning. The authors provide a theoretical worst-case analysis showing limitations of common measures and provide empirical tests across several domains. 

---
I thank the reviewers for their responses. I still struggle with the novelty of this paper as it relates to well known theoretical and empirical work. However, I can see that there may be benefit to explicitly discussing these issues explicitly so I am raising my score to a weak reject.
strengths_and_weaknesses: Strengths
- interesting problem area
- good empirical study of several reward learning measures
- includes both theoretical and empirical analysis

Weaknesses
- not focused on robotics
- theory seems overly simple (focused on simple 2 feature examples) and it is unclear how useful the worst-cast analysis is
- missing related work
- not broad enough to qualify as a survey
originality: Fair: Incremental work, no significant advance.
technical_quality: Fair: Significant additional evidence required to support the claims.
clarity_of_presentation: Fair: Major revision needed to improve organization or clarity.
impact: 2: Unsure, difficult to assess.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations section not present
recommendation: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
summary_of_recommendation: The authors study an interesting machine learning problem. However, I do not think there is enough depth or breadth for CoRL and while there is a strong connection to robot learning, the paper seems better suited for a machine learning conference. The analysis seems overly simple and does not seem to contribute to the field---many of the claims made are commonly understood in the field. While I appreciate the effort to survey measures for reward learning, the paper provides insufficient breadth for a survey and does not have as much novelty as claimed. There are several important missing works and the ideas of reward measures being uninformative and the importance of testing in transfer domains do not appear to be significant contributions given prior work.
issues: It is unusual to have a reward function that maps directly from trajectories to rewards. Most work considers R(s) or R(s,a).

It is well known that just comparing reward parameters is not a good way to evaluate a learned reward. Indeed, much work has studied and analyzed the problem of which rewards lead to the same optimal policy [1] and which rewards make a given policy optimal [2]. 

[1] Brown et al. "Value alignment verification." International Conference on Machine Learning. 2021.

[2] Ng and Russell. "Algorithms for inverse reinforcement learning." ICML. 2000.

Also, no mention is made of the common practice to test learned rewards in transfer domains:
- Levine et al. "Nonlinear inverse reinforcement learning with gaussian processes." Advances in neural information processing systems 24 (2011).
- Fu, Justin, Katie Luo, and Sergey Levine. "Learning Robust Rewards with Adverserial Inverse Reinforcement Learning." International Conference on Learning Representations. 2018.

Recent work has looked at characterizing learned reward functions:
- Michaud, Eric J., Adam Gleave, and Stuart Russell. "Understanding learned reward functions." arXiv preprint arXiv:2012.05862 (2020).
and also, importantly, at evaluating learned rewards using metrics other than alignment or mean squared error:
- Gleave, Adam, et al. "Quantifying Differences in Reward Functions." International Conference on Learning Representations. 2020.

The paper mentions other measures in 2.3, but does not evaluate them. If this is designed to be an analysis and survey paper, then I think these should be included for study.

In table 1, R^rel is defined but I do not find any definition of R^abs

Line 117: [1] and [2] above provide ways of knowing whether the robot will solve the task, even if w' != w^user

Line 160: Shouldn't this be -N + 1? [looking at the values in line 155]

Presentation recommendations: It would be beneficial to show the different domains being studied. Based on the descriptions in the paper it is hard to tell how complex or interesting these tasks are.

No limitations section! This was explicitly asked for in the author instructions. The authors need to address limitations of their paper and analysis.

End of section 6 has some interesting ideas, but they seem very difficult and it would be nice to provide some ideas for how they could be accomplished. How would the robot modify its reward estimate. Also, future work direction (1) seems like it is already done in the literature. Many works on active learning select training instances.
reviewer_expertise: 4: The reviewer is confident but not absolutely certain that the evaluation is correct

From: robot-learning.org/CoRL/2022/Conference/Paper306/Reviewer_GMPv
summary: This paper brings up some of limitations underlying existing methodologies for evaluating reward learning algorithms in HRI. After classifying existing evaluation methodologies into two different classes, the authors demonstrate practical issues through empirical and theoretical analysis over a set of planning benchmarks.
strengths_and_weaknesses: *Strengths*

-I enjoyed reading this paper. It is very well written and easy to follow. I liked the balance between exemplary discussions, theoretical work, and empirical work. Well done!

-The paper brings up some of the limitations of existing reward learning approaches. The point is interesting and timely. While there is a very rich literature in this space with major claims and cool demonstrations, it looks like the evaluation methodologies followed are often fundamentally flawed. Specifically, it looks like the measures that many researchers tend to use to evaluate how well a robot has adapted to a user's preferences are not reliable indicators.

Weaknesses
-The paper does not discuss related work but rather lists a large set of references. Most of the references are coming from 1-2 groups. If this aspires to serve as a survey as the authors state, I would recommend a richer and deeper discussion of the literature within the narrative of the introduction.

-This seems to be in part a "statement" paper with relatively limited novelty. The main point is definitely intriguing and interesting but the key insight is relatively straightforward and the final discussion is quite limited. It would have been more interesting/impactful (to me) to expand upon the closing discussion on what comes next and what paths might be viable. I understand that the space limitations might not allow that but in its current form, I feel like the paper is missing an insightful discussion.

-The authors discuss this briefly but it seems to me that one fundamental limitation of this whole space of approaches is the assumption that w^{user} is somehow available when in the whole field of HRI this is most likely not the case. The included analysis (as well as the reviewed approaches) makes use of that assumption. Perhaps it would be valuable to expand a bit on that.

-The limitations statement, a required component for this year's conference is missing.

*Minor*

Line 32: “These observation…" --> These observations 

Explicitly define R_abs

Make the scale of axes in Fig. 1 equal 

Line 207: “five categories” --> four categories

Clearly name the planning problems, perhaps use bold to define Dubins, Driver, Server in text so that the correspondence from text to table 3 is clear.
originality: Good: Some advance.
technical_quality: Very Good:  Convincing.
clarity_of_presentation: Very Good:  Well organized and clear.
impact: 3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations section not present
recommendation: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
summary_of_recommendation: Overall, this paper makes an interesting and intriguing point about the rigor of evaluation methodologies in the field of reward learning for HRI. It is well written and makes nice use of examples, theoretical and empirical work to support the main observation. While interesting, the main point is still relatively straightforward and I would have appreciated a more elaborate discussion on what comes next. I am generally favorable if such a discussion is included.
issues: -How were the five problem instances in Sec. 5.1 decided? How much they vary and how representative are they?

-Is there a reason why the server experiment is not featured in Fig. 2? Is it not interesting?

-Include a deeper discussion of the literature and a more insightful closure at the end of the paper. See my notes above.

-Add a limitations statement.

reviewer_expertise: 4: The reviewer is confident but not absolutely certain that the evaluation is correct

From: robot-learning.org/CoRL/2022/Conference/Paper306/Reviewer_LhcT
summary: This work serves to point out that the current methods of evaluating reward function algorithms do not appear to be sound. In particular, the paper shows both parameter and reward based comparisons to ground truth reward functions can fail to give a reasonable sense of how good a given solution is, and by proxy, how effective the associated algorithm is. Both theoretical and empirical arguments are provided to support their case. 

strengths_and_weaknesses: Strengths:
- The paper demonstrates a fault with the evaluation methodology employed by the community. In my opinion, it does a good job in showing the existence of the issue, which is perhaps tacitly known but, to my knowledge, had yet to be formally investigated. 
- Both theoretical and empirical evidence / arguments are presented, which makes it clear the problem exists across multiple settings. 
- A well-written paper that is easy-to-follow (up to the points below)

Weaknesses:
- Although the paper clearly shows a problem exists, it is not clear how severe or widespread the issue is, especially with more complex robot tasks. The theoretical results are based on very simple constructed examples, and the numerical results are for relatively simple domains. The paper would be more impactful if the authors can discuss the implications of their work, e.g., if previous comparisons in the cited works are invalidated? 
- The multi-scenario setting in Sec 3.2 points to mis-modeling of the problem; the path length and jerk are not invariant to the goal location? Is it that the rewards are transferable from train to test, but the relative difference between accumulated rewards is not preserved? 
- On a related note, the paper falls short of analyzing why and when the reward/parameter measures fail to be meaningful. I believe the paper will be more impactful if the authors can provide a discussion around the factors affecting the veracity of these measures. Alternatively, this limitation should be discussed in the paper. 
- The paper lacks a limitations section.
- The paper has several typos, e.g., 
   - line 115: “they do consider” -> “they do *not* consider”
   - line 162: “now upper bound” -> “no upper bound”
   - line 180: “now” -> “no”
originality: Very Good:  Major advance in one or more aspects listed above.
technical_quality: Good: Generally convincing, but some additional experiments or analyses needed.
clarity_of_presentation: Good: Minor revision needed to improve organization or clarity.
impact: 4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations section not present
recommendation: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
summary_of_recommendation: Overall, I believe the paper provides interesting results that will make the community re-evaluate our we should compare algorithms for reward learning. There are some minor issues with the work, but I think the results are interesting enough to warrant publication. A major weakness of the paper is that it does not clearly identify the limitations of the provided theoretical and empirical analysis. 

issues: Please see the weaknesses listed above and I hope the authors can modify the paper to provide a more clear statement of the limitations surrounding their analysis. 
reviewer_expertise: 4: The reviewer is confident but not absolutely certain that the evaluation is correct

From: robot-learning.org/CoRL/2022/Conference/Paper306/Reviewer_PQx1
summary: The paper is an unusual contribution for a robotics conference. It does not directly contribute to advancing understanding of robot control or learning. Rather it presents a critical analysis of the measures used to assess the quality of policies learned via HRI driven reward learning.  The key disadvantages of the two main classes of performance metrics (Parametric similarity and Relative Reward) are demonstrated with a simple example, theoretical mathematical abstraction, and numerical simulation on several examples. 
Central to the paper are two proofs: (1) that even when the alignment of the learned weight vector is arbitrarily close to 1, there is not upper bound on the difference between the learned reward and the optimal reward(according to user preferences). (2)no lower bound exists on the relative reward of the test data. These two theorems are further demonstrated with evaluation of randomly sampled weight vectors in 2 examples.

strengths_and_weaknesses: #### Strengths
- The motivating example in section 3 provides a clear explanation of the way in which the two classes of measures might fail.
- The proofs and numerical results support the central contribution that typical measures do not provide guaruntees of the actual performance of the learner.
- Overall ideas are well-communicated.

#### Weaknesses
- There’s no robots? This seems like it may not be the best fit for CoRL. The criticisms of reward learning could be applied to many different learning problems. The paper does not make a convincing case as to why it is critical to robot learning in particular.
- The paper lacks a discussion of what aspects of the measures make them fail. It just states that they do. The discussion poses very generic questions.
- Given that there are these possible worst-case scenarios, the paper could talk about how existing works (the first 15 references) avoid these worst-case scenarios or if there are published instances in the sim2real literature where this might be part of the issues.


originality: Good: Some advance.
technical_quality: Very Good:  Convincing.
clarity_of_presentation: Excellent: Enjoyable to read.
impact: 3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations section not present
recommendation: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
summary_of_recommendation: Overall the paper is well-written and presents an interesting perspective. This perspective could be valuable to the greater community in analyzing why learners fail despite high performance expectations based the chose performance measure. The paper does lack some discussion of either why these performance measures fail, more specific requirements for 'good' measures, or how existing measures might be modified or combined to avoid some of these pitfalls. 
The main issue is that the work includes neither a robot nor any learning, and the manuscript does not make a strong case for why this is critical to the robot learning community specifically as opposed to machine learning community in general. 
issues: #### Major Issues
-  The paper needs some explanation of how prior works have avoided the pitfalls presented by the paper.
-  More discussion of either the specific attributes of the measures that lead to failure in the given cases or how one might approach a reward learning problem to avoid the 'worst-case' scenarios is needed.

#### Minor Issues
- Line 33: the wording says high(but not optimal) but before explaining the measure I think it makes more sense to say that the parameters values are very similar (but not identical)
- Line 162: now -> no
- Line 180: now -> no


reviewer_expertise: 4: The reviewer is confident but not absolutely certain that the evaluation is correct