Paper: STARC: A General Framework For Quantifying Differences Between Reward Functions (ICLR-2024)

Reviewer comment (Presentation):
The results in Figure 1 are a bit difficult to parse. Each column corresponds to a different metric family, and each point is a different choice of either the metric \( m \), the norm \( n \) or the canonicalisation \( c \). But what is the None column? Are the choices of \( m \), \( n \) and \( c \) consistent throughout different families? Although I appreciate the space constraints imposed by the venue, I really feel that some details on how these are chosen should fit into this section, as these are crucial to properly assess the proposed results.

Author response:
The None column corresponds to the case where no canonicalisation function is used. The choices of $m$ and $n$ are consistent throughout these families, except that the MinimalPotential canonicalisation function was not used with the $L_\infty$-norms, because the optimisation algorithm for MinimalPotential fails to converge for these norms. The full details are provided in Appendix F (and especially F.3), but we will move more of this information into the main text, to make the figure easier to parse.

Reviewer comment (Experiments):
I think it would have been also interesting to assess the individual impact of the choice of \( m \), \( n \) or \( c \) when the values of the other is fixed: for example, how important is the choice of the canonicalisation \( c \) w.r.t. the chosen metric and norm? Although there is some discussion w.r.t. this at the end of page 8, it is a bit difficult for me to relate such discussion to the results in Figure 1.

Author response:
We agree that this comparison is relevant, though it would be difficult to fit it in Section 4 given the space constraints. This information *can* be read from the raw data provided in Appendix G, but we will make this information more accessible.

Reviewer comment (Theory):
From the experimental results shown in Figure 1, we can see that the choice of normalization function $n$ and distance metric $m$ can have a significant impact on the metric’s accuracy. But the theoretical results presented in this paper cannot indicate which STARC metrics work best in practice.

Author response:
It is true that our theoretical results cannot indicate which STARC metric would work best in practice, but our experiments suggest that the $L_1$-norm may be the best choice for both $m$ and $n$.

Reviewer comment (Theory):
The theoretical results assume that $\mathcal{S}$ and $\mathcal{A}$ are finite, which are not applicable to continuous environments.

Author response:
We should note that we expect our theoretical results to generalise to (sufficiently well-behaved) continuous environments, as also indicated by our experiments. However, we think it is reasonable to consider this extension of our analysis to be out of scope for this paper.

Reviewer comment (Writing):
In Definition 1 and the entire paper, the phrase “differ by” frequently occurs. But what do you mean by it? Say, “$C(R)$ and $R$ differ by potential shaping and $S’$-distribution” in Definition 1, do you mean that we can get $C(R)$ by first potential shaping $R$ and then $S’$-distribution? Does the order matter?

Author response:
Yes, that is correct; when we say that $R_1$ and $R_2$ differ by potential shaping and $S'$-redistribution, we mean that there is a potential shaping transformation $t_1$ and an $S'$-redistribution transformation $t_2$ such that $R_2 = t_1 \circ t_2(R_1)$. Equivalently, it means that there is a potential function $\Phi$ such that $E[R_2(s,a,S')] = E[R_1(s,a,S') + \gamma \Phi(S') - \Phi(s)]$. This also means that the order does not matter. We will amend the text, to make this more clear.

Reviewer comment (Theory):
Considering Definition 3, you define $s(R)=c(R)/n(c(R))$ and claim in the following sentence that $s$ would reduce rewards that have the same ordering of policies to the same one. Could you please give a simple example (MDP) and illustrate the calculation process of your defined STARC metric and verify your claim.

Author response:
We can certainly do this. Consider a Bandit MDP with several actions and one state $s$, let $c$ be the VAL-canonicalisation for the uniformly random policy, and let $n$ be the $L_2$-norm. In this case, two reward functions induce the same ordering of policies if and only if they differ by an affine transformation. Moreover, for any reward function $R$, we have that $V^\pi(s) = E[R(s,A,s)]/(1-\gamma)$, where $\pi$ is the uniformly random policy, and $A$ is sampled uniformly at random. This means that $c(R)(s,a,s) = R(s,a,s) - E[R(s,A,s)]/(1-\gamma) + \gamma E[R(s,A,s)]/(1-\gamma) = R(s,a,s) - E[R(s,A,s)]$, where $A$ is sampled uniformly at random. In this case, it is easy to see that $c(R_1) = c(R_2)$ if and only if there is a constant $b$ such that $R_2 = R_1 + b$. Division by $n(c(R))$ normalises the resulting reward. This means that $s(R_1) = s(R_2)$ if and only if there is a positive constant $a$ such that $c(R_2) = a \cdot c(R_1)$. Since $c$ is linear, this means that $c(R_2) = c(a \cdot R_1)$. Putting this together, we get that $s(R_1) = s(R_2)$ if and only if $R_2 = a \cdot R_1 + b$ for some $a$ and $b$, which in turn is the case if and only if $R_1$ and $R_2$ induce the same ordering of policies.

Reviewer comment (Theory):
In Proposition 2, what is the reward function when calculating $V^\pi$?

Author response:
$V^\pi$ should be calculated using the reward function that is being canonicalised. I.e., when calculating $c(R_1)$, $V^\pi$ should be calculated using $R_1$, and when calculating $c(R_2)$, $V^\pi$ should be calculated using $R_2$, etc. We will clarify this in the text.

Reviewer comment (Theory):
In the proof of Propositions 9 and 10, you just prove that “$c(R_1)=c(R_2)$ if $R_1$ and $R_2$ differ by potential shaping and $S’$-redistribution.” However, in the definition of canonicalization function, a canonicalization must satisfy that “$c(R_1)=c(R_2)$ if and only if $R_1$ and $R_2$ only differ by potential shaping and $S’$-redistribution.” Why not consider “only if”?

Author response:
It is correct that it should be "if and only if", and we will amend the proofs to make this explicit. Note that if $R$ and $c(R)$ differ by potential shaping and $S'$-redistribution for all $R$, then $R_1$ and $c(R_1)$ differ by potential shaping and $S'$-redistribution, and likewise for $R_2$ and $c(R_2)$. Then if $c(R_1) = c(R_2)$, we can combine these transformations, and obtain that $R_1$ and $R_2$ also differ by potential shaping and $S'$-redistribution.

Reviewer comment (Evaluation):
One substantial weakness I see with the reward comparison literature in general is its applicability: the paper is missing a discussion of how reward comparison might be used in practice, when there is no access to the ground truth reward. Is it meant to be used to compare two learned rewards? How does the value influence a ML system designer's decision on what to do next?

Author response:
The intended use for these metrics is to *evaluate* reward learning algorithms, in a setting where the true reward function *is* known. For example, we may randomly generate reward functions, use a reward learning algorithm, and then measure how different the learnt reward function is from the true reward function. Alternatively, they can also be used in theoretical work, where we may wish to bound the distance between the true reward and the learnt reward (given different amounts of training data, for example). Such analysis would then indicate how reliable a given reward learning method is, which in turn then tells us how accurate we should expect a learnt reward function to be, in a setting where the true reward function is not known. Note that Gleave et al is cited 37 times, and Wulfe et al is cited 6 times; these references will provide examples of practical situations where reward function pseudo-metrics can be used. For example, see eg Adeniji et al, 2023, Figure 1 in Sanghvi et al, 2021, or Figure 4(c) in Liang et al, 2022.

Reviewer comment (Experiments):
In section 2.3, the authors argue for why they think the dependence on T is meaningful. I think it is missing a discussion of the sim2real setting. Does it require a simulator with the true transitions, or simply samples from the true transitions? If the former, I would argue it is a meaningful limitation for many real world tasks due to the sim2real gap.

Author response:
We do not require a simulator with the true transitions --- samples from the true transitions are sufficient. For example, the VAL canonicalisation can be estimated using samples. This is discussed in the first paragraph of Section 2.3.

Reviewer comment (Evaluation):
In Reacher, the authors acknowledge that estimating the metric via sampling involves summing over absolute values which makes all noise positive, thus inflating the metric for PotentialShaped which should have 0 distance. Indeed in table 1, the distance values are not linearly related to what I'd want the metric to say. The authors argue it is not problematic for ranking, which I agree with, but it is crucial for absolute evaluation (as opposed to ranking), which I think is also important. The paper is missing a discussion of absolute evaluation, as well as a discussion of how big an issue they expect this to be for environments in general.

Author response:
We agree that this may be a limitation in some contexts. However, note that the error can be reduced by increasing the number of samples used in the estimation. Moreover, it can be eliminated completely if the STARC metric is computed exactly (although this may not be feasible in large environments).

Reviewer comment (Theory):
The authors claim to provide a complete answer to the question of how to measure distance between reward functions, but I'd argue completeness requires providing a tight bound. We're not sure how large the spread L to U is.

Author response:
We agree that the spread between L and U is important. However, note that any metric which is sound and complete must be bilipschitz equivalent to STARC metrics -- this means that whichever metric has the smallest spread between L and U must be quite similar to STARC metric (and it might be an instance of a STARC metric). Note also that the exact value of L and U depends on the transition function; to see this, consider the second figure in Appendix B (the transition function affects the diameter of $\Omega$).

Reviewer comment (Writing):
Typo on pg 3? Definition of pseudometric: should be n(x-v).

Author response:
Yes, this is indeed a typo, thank you for pointing this out.

Reviewer comment (Writing):
It would be great to provide (even 1-sentence) intuition for EPIC (explain the various normalization terms that EPIC paper explains), because it is crucial for understanding your definition of canonicalization function.

Author response:
We describe EPIC in Section 1.2. We can make this description more extensive, to make it easier to understand.

Reviewer comment (Experiments):
How are D_S and D_A chosen in EPIC?

Author response:
In our experiments, $D_S$ and $D_A$ were both uniform for EPIC. We will make sure that this is included in Appendix F; thank you for spotting this omission.

Reviewer comment (Theory):
Can you give a brief explanation/intuition for S'-distribution? Since it is the difference between your definition of canonicalization function and that used in EPIC.

Author response:
$S'$-redistribution is simply any transformation that never changes the expected value of the reward function. For example, suppose we have a nondeterministic MDP where an action $a$ in state $s$ leads to $s_1$ or $s_2$ with equal probability, and that $R_1(s,a,s_1) = 1$, $R_1(s,a,s_1) = 0$. Then $S'$-redistribution could be used to produce a reward function $R_2$ where $R_2(s,a,s_1) = R_2(s,a,s_2) = 0.5$.

Reviewer comment (Writing):
In def 1: where do R, R_1, R_2 come from? Is it all such R's in the reward class?

Author response:
The conditions in Definition 1 are meant to hold for all reward functions -- it can thus be read as "for all $R$, ..., and for all $R_1, R_2$, ... We will amend the text to make this more clear.

Reviewer comment (Writing):
Can you provide intuition for Im(c)? It appears somewhat out of nowhere.

Author response:
$\mathrm{Im}(c)$ is the output of the canonicalisation function, when it is applied to the space of all reward functions. From the definition of canonicalisation functions, we get that this is a linear subspace of $\mathcal{R}$ in which no reward functions differ by potential shaping or $S'$-redistribution. A geometric description of this is given in Appendix B, but we can also describe this in more detail in the main text.

Reviewer comment (Theory):
In def 1 and 4: should it say differ *only* by (potential shaping and S'-redistribution)?

Author response:
Yes. We will amend the text to make this clear.

Reviewer comment (Theory):
Why is it not given that minimal CFs exist for any given norm or are unique?

Author response:
For example, relative to the $L_1$-norm or the $L_\infty$-norm, there will be multiple minimal canonicalisation functions. This is because the unit balls of these norms are not strictly convex.

Reviewer comment (Reproducibility):
How should norm and distance metric be chosen in practice?

Author response:
We expect that it in practice often will be best to pick $n$ and $m$ so that they can be computed easily. Our experiments suggest that the $L_1$-norm is the best (unweighted) norm for both $n$ and $m$.

Reviewer comment (Experiments):
How do you choose the policy for VAL canonicalization? Why uniform for Reacher?

Author response:
VAL is a valid canonicalisation function for any choice of policy, so we simply used a policy for which $V^\pi$ would be easy to learn. The benefit of using a uniform policy, rather than some deterministic policy, is that exploration then is built in automatically.

Reviewer comment (Evaluation):
The premise of this work is that the proposed pseudo-metric could be used to quantify how a learned reward function differs from the true one. Unfortunately, in practice, when one needs to learn a reward function, the true one is not known, which makes the usefulness of such pseudo-metric not clear to me.

Author response:
We should clarify that the intended use for these metrics is to *evaluate* reward learning algorithms, in a setting where the true reward function *is* known. For example, we may randomly generate reward functions, use a reward learning algorithm, and then measure how different the learnt reward function is from the true reward function. Alternatively, they can also be used in theoretical work, where we may wish to bound the distance between the true reward and the learnt reward (given different amounts of training data, for example). Such analysis would then indicate how reliable a given reward learning method is, which then tells us how accurate we should expect a learnt reward function to be, in a setting where the true reward function is not known. Note that Gleave et al is cited 37 times, and Wulfe et al is cited 6 times; these references will provide examples of practical situations where reward function pseudo-metrics can be used. For example, see eg Adeniji et al, 2023, Figure 1 in Sanghvi et al, 2021, or Figure 4(c) in Liang et al, 2022. We will amend the introduction, to make this intended use case more clear.

Reviewer comment (Writing):
The interchangeable use in the text of pseudo-metric and metric makes things a bit confusing sometimes. I think it would be better to use consistently pseudo-metric to refer to pseudo-metric.

Author response:
We will amend the text to consistently use the term pseudo-metric, to make it less ambiguous. We also fix the typos that you have spotted -- thank you for pointing them out to us!

Reviewer comment (Writing):
The definition of metric m from norm n contains a typo (page 3).

Author response:
We will amend the text to consistently use the term pseudo-metric, to make it less ambiguous. We also fix the typos that you have spotted -- thank you for pointing them out to us!

Reviewer comment (Writing):
In Definition 6, the phrase "there exists two" should be "there exist two".

Author response:
We will amend the text to consistently use the term pseudo-metric, to make it less ambiguous. We also fix the typos that you have spotted -- thank you for pointing them out to us!

Reviewer comment (Presentation):
The paper "Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking" appears twice in the references.

Author response:
We will amend the text to consistently use the term pseudo-metric, to make it less ambiguous. We also fix the typos that you have spotted -- thank you for pointing them out to us!

Reviewer comment (Evaluation):
The authors first write: "STARC metrics in practice CAN have a much tighter correlation with worst-case regret than both EPIC and DARD" then continue with: "This means that STARC metrics both attain better empirical performance". Is this always true?

Author response:
1. Our experiments suggest that this is not always true, but that it is true for most reasonable choices of normalisation norm and metric --- see Figure 1.

Reviewer comment (Writing):
What does this sentence mean? "the property that increasing R1 cannot decrease R2"

Author response:
2. This means that if two reward functions $R_1, R_2$ and two policies $\pi_1, \pi_2$ satisfy that $J_1(\pi_1) > J_1(\pi_2)$, then it must be the case that $J_1(\pi_1) \geq J_1(\pi_2)$. In other words, if the expected cumulative sum of $R_1$-reward increases when we go from $\pi_2$ to $\pi_1$, then it cannot be the case that the expected cumulative sum of $R_2$-reward decreases. We will amend the text, to make this more clear.

Reviewer comment (Evaluation):
Could you provide some examples where the true reward is not known, but such pseudo-metrics could be helpful?

Author response:
3. We can evaluate a reward learning algorithm in a setting where the true reward is known. The result of this evaluation would then inform how much we should trust that algorithm, in a setting where the true reward is not known.