Paper:
Reinforcement learning agents can include different components, such as policies, value functions, state representations, and environment models. Any or all of these can be the loci of knowledge, i.e., structures where knowledge, whether given or learned, can be deposited and reused. Regardless of its composition, the objective of an agent is behave so as to maximise the sum of a suitable scalar function of state: the reward. As far as the learning algorithm is concerned, these rewards are typically given and immutable. In this paper we instead consider the proposition that the reward function itself may be a good locus of knowledge. This is consistent with a common use, in the literature, of hand-designed intrinsic rewards to improve the learning dynamics of an agent. We adopt the multi-lifetime setting of the Optimal Rewards Framework, and propose to meta-learn an intrinsic reward function from experience that allows agents to maximise their extrinsic rewards accumulated until the end of their lifetimes. Rewards as a locus of knowledge provide guidance on "what" the agent should strive to do rather than on knowledge of "how" the agent should behave that is more directly captured in policies or value functions for example. Thus, our focus here is on demonstrating the following: (1) that it is feasible to meta-learn good reward functions, (2) to show that the learned reward functions can capture interesting kinds of "what" knowledge, and (3) that because of the indirectness of this form of knowledge the learned reward functions can generalise to other kinds of agents and to changes in the dynamics of the environment.
Reinforcement learning agents can store knowledge in their policies, value functions, state representations, and models of the environment dynamics. These components can be the loci of knowledge in the sense that they are structures in which knowledge, either learned from experience by the agent's algorithm or given by the agent-designer, can be deposited and reused. The objective of the agent is defined by a reward function, and the goal is to learn to act so as to optimise cumulative rewards. In this paper we consider the proposition that the reward function itself is a good locus of knowledge. This is unusual in that most prior work treats the reward as given and immutable, at least as far as the learning algorithm is concerned. At the same time, especially in challenging reinforcement-learning problems, agent designers do find it convenient to modify the reward function given to the agent to facilitate learning. It is therefore useful to distinguish between two kinds of reward functions : extrinsic rewards define the task and capture the designer's preferences over agent behaviour, whereas intrinsic rewards serve as helpful signals to improve the learning dynamics of the agent. Intrinsic rewards are typically hand-designed and then often added to the immutable extrinsic rewards to form the reward optimised by the agent.
Most existing work on intrinsic rewards falls into two broad categories: task-dependent and taskindependent. Both are typically designed by hand. Hand-designing task-dependent rewards can be fraught with difficulty as even minor misalignment between the actual reward and the intended bias can lead to unintended and sometimes catastrophic consequences (Clark & Amodei, 2016). Task-independent intrinsic rewards are also typically hand-designed, often based on an intuitive understanding of animal/human behaviour or on heuristics on desired exploratory behaviour. It can, however, be hard to match such task-independent intrinsic rewards to the specific learning dynamics induced by the interaction between agent and environment. The motivation for this paper is our interest in the comparatively under-explored possibility of learned (not hand-designed) taskdependent intrinsic rewards (see Zheng et al., 2018, for previous work).
We emphasise that it is not our objective to show that rewards are a better locus of learned knowledge than others; the best locus likely depends on the kind of knowledge that is most useful in a given
task. In particular, knowledge captured in rewards provides guidance on "what" the agent should strive to do while knowledge captured in policies provides guidance on "how" an agent should behave. Knowledge about "what" captured in rewards is indirect and thus slower to make an impact on behaviour because it takes effect through learning, while knowledge about "how" can directly have an immediate impact on behaviour. At the same time, because of its indirectness the former can generalise better to changes in dynamics and learning architectures. Therefore, instead of comparing different loci of knowledge, the purpose of this paper is to show that it is feasible to capture useful learned knowledge in rewards and to study the kinds of knowledge that can be captured.
How should we measure the usefulness of a learned reward function? Ideally, we would like to measure the effect the learned reward function has on the learning dynamics. Of course, learning happens over multiple episodes, indeed it happens over an entire lifetime. Therefore, we choose lifetime return, the cumulative extrinsic reward obtained by the agent over its entire lifetime, as the main objective. To this end, we adopt the multi-lifetime setting of the Optimal Rewards Framework (Singh et al., 2009) in which an agent is initialised randomly at the start of each lifetime and then faces a stationary or non-stationary task drawn from some distribution. In this setting, the only knowledge that is transferred across lifetimes is the reward instead of the policy. Specifically, the goal is to learn a single intrinsic reward function that, when used to adapt the agent's policy using a standard episodic RL algorithm, ends up optimising the cumulative extrinsic reward over its lifetime.
In previous work, good reward functions were found via exhaustive search, limiting the range of applicability of the framework. Here, we develop a more scalable gradient-based method (Xu et al., 2018c) for learning the intrinsic rewards by exploiting the fact the interaction between the policy update and the reward function is differentiable (Zheng et al., 2018). Since it is infeasible to backpropgate through the full computation graph that spans across the entire lifetime, we truncate the unrolled computation graph of learning updates up to some horizon. However, we handle the long-term credit assignment by using a lifetime value function that estimates the remaining lifetime return, which needs to take into account changing policies. Our main scientific contributions are a sequence of empirical studies on carefully designed environments that show how our learned intrinsic rewards can capture interesting regularities in the interaction between a learning agent and an environment sampled from a distribution, and how the learned intrinsic reward can generalise to changed dynamics and agent architectures. Collectively, our contributions present an effective approach to the discovery of intrinsic rewards that can help an agent optimise the extrinsic rewards collected in a lifetime.

Review:
The paper proposes a meta-learning approach to learn reward functions for reinforcement learning agents. It defines an algorithm to optimize an intrinsic reward function for a distribution of tasks in order to maximise the agent’s lifetime rewards. The properties of this reward function and meta-learning algorithm  are investigated through a number of proof-of-concept experiments. 
The meta-learning algorithm and the corresponding empirical investigation are the main contributions of the paper. The algorithm seems to be similar to previous meta-learning approaches, but differs by introducing a lifetime value function. While I thought the paper raises some interesting possibilities, I am currently leaning towards rejection. The proposed algorithm does not seem like a major innovation over cited previous work. The empirical evaluation provides a number of proof-of-concept ideas, but no in depth investigation of the properties of the approach. The theoretical properties of the approach are barely discussed.
Detailed remarks:
* The main addition to the meta-learning algorithm is the lifetime value function. The authors mention multiple times that this is crucial to learning, but the properties of this value function are not really investigated or discussed in depth:
- The authors mention that the value function must take into account changing future policies, but do not discuss this further. The value function update seems to be a standard on-policy TD update with the lifetime return and the complete history as input. The policy for this value function, however, is still a standard policy with only state as input (but it will be non-stationary over the agent lifetime). It would be good to discuss this learning problem in more detail.
- The algorithm uses an n-step return. Is this important? What effect does n have on learning?
* Another issue which I would have liked being discussed in more detail is the non-stationarity of the learning problem in general. Most of the approaches discussed in related work (e.g. shaping)  are aimed at learning/designing more informative reward functions. These reward functions still fit in the MDP framework, however, and map from states and actions to rewards. In the case of shaping approaches guarantees can be given that this does not alter the learning problem. The intrinsic reward functions used in this paper map the full life-time history of the agent to rewards. While this is a richer framework that can express more complicated tasks (like exploration over multiple episodes), it also invalidates many of the basic assumptions of reinforcement learning. The rewards are now no longer Markovian when only observing the current state. Moreover, the reward function will change over time. To what extent does this require non-stationary / history-based policy and value function learning to solve these issues? While some of these issues also apply to count based exploration strategies, (Strehl and Littman,2008 ) provided results that the  exploration bonuses result a Bellman Equation that accounts for uncertainties. No real guarantees seem to exist here.
* The empirical contribution focuses on trying to answer a number of questions regarding the properties of the learnt intrinsic rewards. I found these questions to be very broad, while the answers are mostly anecdotal evidence through proof-of-concept examples.  These examples do show potential benefits of meta-learning intrinsic rewards, but I was somewhat disappointed that there was no more systematic investigation. For example, questions like ‘how does the distribution of tasks affect intrinsic rewards’ or ‘does intrinsic reward generalise’ are not really answered by providing metrics of performance or generalisation in controlled experiments, but by providing some example cases. Several of these questions (including optimising exploration and dealing with non-stationarity) also seem to have been investigated to some extent in the original Optimal reward papers (Singh, 2009/2010). It would be good to clearly indicate what we have learned beyond these previous results. 
* There seems to be a bit of a mismatch between the learning objective for intrinsic rewards in the optimal reward framework and the results shown in the experiments. The learning objective aims to optimise lifetime rewards for a distribution of tasks. Most of the experiments seem to analyse episodic reward performance and compare against single-task (or task agnostic) methods.
Minor comments:
- The architecture / parameterization of the lifetime value function does not seem to be defined anywhere. Given that it takes histories as input I assume this is another RNN?
- There seems to be some small overloading in the notation with \eta occasionally being used to denote the parameters of the reward function r_eta or the reward function itself.