Paper: Reward Adaptation Via Q-Manipulation (ICLR-2024)

Reviewer comment (Experiments):
The advantage baseline in A2C seems low-cost; how about considering it for pruning action?

Author response:
The advantage baseline in A2C seems low-cost; how about considering it for pruning action?

-We plan to consider such approaches in the future.

Reviewer comment (Reproducibility):
When does action pruning occur? In the exploration, in the target Q update, or both. It seems that the prune behavior is conservative.

Author response:
When does action pruning occur? In the exploration, in the target Q update, or both. It seems that the prune behavior is conservative.

-Action pruning occurs before starting to learn for the target behavior. When applying Q-learning for the target reward function, during exploration only set of non-pruned actions is used which helps in faster convergence.

Reviewer comment (Experiments):
Could you visualize the pruned action space in the training process and compare it to the original action space?

Author response:
Could you visualize the pruned action space in the training process and compare it to the original action space?

-Learning on original action space is equivalent to traditional Q-learning which is used as one of our baselines and learning after pruning action is our approach which performs better in empirical evaluations.

Reviewer comment (Theory):
I am afraid that the theoretical results for reward shaping is incorrect. Specifically, I have doubts regarding the correctness of Lemma 5. When reward shaping is applied, $Q_{R_{F}}^{\*}$ and $Q_{R_{F}}^{\mu}$ should be offset towards the same direction, not the opposite directions. In fact, as the authors note in Eq. (15), Lemma 5 implies that $Q^{\*}$ can be smaller than $Q^{\mu}$ after reward shaping, meaning that the optimal policy changes, which conflicts with the theory built in [3].

Author response:
We appreciate your feedback on Lemma 5. We need stronger constraints on the structure of the MDP where Q* is not greater than Q^{\mu} after shaping and we would provide a more concrete reasoning for our approach in the next iteration of this paper. We would also consider related works pointed out and show a comparison if necessary or provide an explanation of how they differ.

Reviewer comment (Theory):
Could the authors address my question regarding the correctness of Lemma 5?

Author response:
We appreciate your feedback on Lemma 5. We need stronger constraints on the structure of the MDP where Q* is not greater than Q^{\mu} after shaping and we would provide a more concrete reasoning for our approach in the next iteration of this paper. We would also consider related works pointed out and show a comparison if necessary or provide an explanation of how they differ.

Reviewer comment (Novelty):
As a core contribution, the authors claim that “We introduce the problem of reward adaptation”. However, I am not sure that it is valid. It seems to me the reward adaptation formulation in Section 2.1 of this paper is an extension of the “Transfer via successor features” problem of [1], where the authors assumed the target reward function to be the linear combination of source reward functions (see Section 4 of [1]). If this is not the case, please clarify. If this is the case, it is important to refer to the successor features literature, and compare your method with the methods proposed in [1], both conceptually and empirically.

Author response:
The reward adaptation problem requires knowing source behaviors and how it is combined whereas the “Transfer via successor features” depends on the feature vector and its weight. That work requires a supervised learning of feature and weights whereas RA only aims to reuse behaviors that exists. In addition to this successor features work is dependent on generalized policy improvement theorem where one action can be chosen from available policies at a timestep t, but fails to learn a new policy which none of the source behaviors can achieve combined for the target. Theoretically, a polynomial combination of rewards would be sufficient to approximate any target reward function given enough degrees of polynomial and that’s the reason for defining our problem where the target is a polynomial combination of rewards.