Paper: Reward Translation via Reward Machine in Semi-Alignable MDPs (ICLR-2024)

Reviewer comment (Reproducibility):
The construction of propositional symbols is hinted at several times but never explained. It is not clear how an LLM can be used to extract these symbols.

Author response:
`Q2: Can you clarify what the purpose of the propositional symbols is and how they determine mappings? Maybe with an example?`

Propositional symbols serve as checkpoints within the environment and are often defined in the task description. As an example, in a game where the objective is to find a key and open a door, the key and the door would represent the propositional symbols.

Reviewer comment (Writing):
Definition 1 is not very clear. The notation $O$ is not defined and as a result it is unclear what exactly the constraints on policy optimality and y-dynamics are. See Q2.

Author response:
`Q1: The test uses an O notation that is not explained.`

We sincerely apologize for any confusion this might have caused. In this context, "O" represents the optimality function concerning abstract actions and abstract states. This means that agents within both environments can learn optimal policies that can be mapped.

Reviewer comment (Writing):
Various instances in the paper use notation that is not explained or only explained much later. For example, page 4 section 4 says “seamless mapping between $\mathcal{U}$, $\mathcal{F}$, and $\mathcal{P}$ is achievable”. It is assumed that $\mathcal{U}$ is meant to be $U$ and $\mathcal{F}$ is meant to be $F$.

Author response:
`Q4: Definition of $\mathcal{F}$ and $\mathcal{U}$.`

We apologize if this caused any confusion. Both $\mathcal{F}$ and $\mathcal{U}$ are components of the reward machine. While $\mathcal{F}$ refers to the set of terminal states where the reward machine is done (and subsequently, the reinforcement learning task is complete for that episode), $\mathcal{U}$ denotes a set of general states, such as "The agent has found the key" in the "Find the key and open the door" game example.

Reviewer comment (Experiments):
One of the main claims of the experiments is that “The training results demonstrate that transferring rewards via isomorphic and homomorphic reward machines enhances learning in reinforcement tasks across different domains, improving training efficacy and performance.” However, this claim is not well supported since isomorphic reward functions have not been tested and homomorphic reward functions only worked in a very limited set of the experiments.

Author response:
`Q3: The training results demonstrate that transferring rewards via isomorphic and homomorphic reward machines enhances learning in reinforcement tasks across different domains, improving training efficacy and performance.`

Our experiments with NChain(n=5) to NChain(n=9) showcase the performance of isomorphic reward machines, since both environments share the same reward machine. On the other hand, our Cartpole to Halfcheetah experiment demonstrates the performance of homomorphic reward machines. The structure of these reward machines can be found in the Appendix for comparison with their respective definitions. However, we must acknowledge that our experiments' scale may not make them entirely persuasive, and we will strive to enhance their robustness.

Reviewer comment (Presentation):
The intuitive motivation in Figure 1 (Cartpole -> HalfCheetah) is very confusing, as the goal of balancing the cartpole is significantly different from the goal of halfcheetah that makes it run faster as far as possible. Giving a more intuitive example or supporting the argument here could be useful for improving the clarify of the paper.

Author response:
As explained in the commonly mentioned questions, we've adjusted the cartpole's target to reach the edge of the screen. In this new setting, we want to illustrate that a direct mapping doesn't exist between state and action spaces or transition functions in the two environments, which means they cannot be characterized as an alignable MDP and as defined by Kim, et al. The NChain might serve as a better example where the NChain(n=5) and NChain(n=9) share the same goal, but their different horizon lengths prevent them from ...

Reviewer comment (Experiments):
The usage of GPT-4 for constructing a reward machine is interesting, but if it's possible, it's not clear to me why it's still necessary to transfer the reward, because it might be possible to generate the rewards for the downstream task directly. This should be thoroughly investigated by including an additional baseline.

Author response:
We recognize that utilizing LLMs to learn rewards for reinforcement learning is a fascinating domain, although it extends beyond our current objectives. Our primary intention in this paper is to construct a bridge allowing semi-alignment between MDPs in different environments, enabling the reuse or transfer of rewards acquired from trained environments without the direct learning of rewards. Reward Machines serve as the bridge we employ. Ultimately, our experiments involve both hand-defined reward machines and LLM-generated reward machines, and we've found that LLMs can effectively learn a similar reward machine based on the task manual.