  For many reinforcement learning (RL) applications, specifying a reward is
difficult. In this paper, we consider an RL setting where the agent can obtain
information about the reward only by querying an expert that can, for example,
evaluate individual states or provide binary preferences over trajectories.
From such expensive feedback, we aim to learn a model of the reward function
that allows standard RL algorithms to achieve high expected return with as few
expert queries as possible. For this purpose, we propose Information Directed
Reward Learning (IDRL), which uses a Bayesian model of the reward function and
selects queries that maximize the information gain about the difference in
return between potentially optimal policies. In contrast to prior active reward
learning methods designed for specific types of queries, IDRL naturally
accommodates different query types. Moreover, by shifting the focus from
reducing the reward approximation error to improving the policy induced by the
reward model, it achieves similar or better performance with significantly
fewer queries. We support our findings with extensive evaluations in multiple
environments and with different types of queries.
