Paper:
PREFERENCES IMPLICIT IN THE STATE OF THE WORLD
Reinforcement learning (RL) agents optimize only the features specified in a reward function and are indifferent to anything left out inadvertently. This means that we must not only specify what to do, but also the much larger space of what not to do. It is easy to forget these preferences, since these preferences are already satisfied in our environment. This motivates our key insight: when a robot is deployed in an environment that humans act in, the state of the environment is already optimized for what humans want. We can therefore use this implicit preference information from the state to fill in the blanks. We develop an algorithm based on Maximum Causal Entropy IRL and use it to evaluate the idea in a suite of proof-of-concept environments designed to show its properties. We find that information from the initial state can be used to infer both side effects that should be avoided as well as preferences for how the environment should be organized. Our code can be found at https://github.com/HumanCompatibleAI/rlsp.

Review:
This paper considers the problem of inferring unspecified costs in an RL problem (e.g., inferring that vases in a room should not be broken). The primary insight is that the initial state of the environment conveys rich information about such unspecified costs since environments are often optimized for humans. The paper frames the problem of inferring unspecified costs from the initial condition as an inverse reinforcement learning (IRL) problem and applies the Maximum Causal Entropy IRL framework to solve this problem. Two methods are proposed for combining the inferred unspecified costs with specified costs. The efficacy of the proposed approach is demonstrated on a number of simulated examples.
Overall, I was impressed by this paper and I believe that it makes a strong contribution. The paper presents an interesting perspective on a relatively old problem (the frame problem in AI). The primary intuition of the paper (that the initial state conveys information about unspecified costs) and the framing of this problem in terms of IRL is novel. The simulated examples (while relatively simple in terms of the number of states and actions) are informative and demonstrate the strengths of the approach (and also some of the weaknesses; the paper is explicit about the current challenges). The paper is very clearly written and is easy to read.
My concerns are relatively minor:
- Perhaps the weakest bit of the paper is Section 5 (combining the specified reward with the inferred reward). As presented, the Additive method is somewhat hard to justify. However, the simulated results suggest that the Additive method performs slightly better than the Bayesian method. I would suggest either presenting a bit more intuition and justification for the Additive method or getting rid of this method altogether (since the results are not too different from the Bayesian method, which seems a bit more justifiable).
- One practical (and potentially important) question that the paper does not directly address is the problem of choosing the time horizon T (i.e., the time horizon for the past). In the standard IRL setting, it is reasonable to assume that the time horizon is given (since the demonstrations have an associated horizon). However, it is not entirely clear how to choose T in the setting considered in this paper. It is possible that if one chooses T to be too small, the inferred rewards will not be accurate (and one may have to look further back in the past to correctly infer rewards). A discussion of this issue and possible ways to choose T would be helpful.
- In Section 6.1 (baselines), the paper mentions that "while relative reachability makes use of known dynamics, it does not benefit from our handcoded featurization". Is it possible to modify the relative reachability method to also take advantage of the handcoded features, perhaps by considering dynamics over the feature space? If not, a sentence explaining that this is not straightforward would be helpful.
- In the related work section (and also in the introduction), I would recommend being more explicit about precisely what the differences are between the presented work and the approaches presented in (Krakovna et al. 2018) and (Turner, 2018). The paper is currently slightly vague about the differences.
- Currently, the title of the paper is a bit uninformative. On first reading the title, I expected a paper on control theory; the title makes no mention of unspecified costs, or reinforcement learning, or humans, etc. I believe that this is a good paper and that the paper would have more readers if the title was more inline with the content of the paper. Of course, this is at the discretion of the authors. My suggestion would be something along the lines of "Inferring Unspecified Rewards in RL from the Initial State".
Typos:
- Pg. 1, second paragraph, 3rd line: there is a placeholder for citations.
- Periods are missing at the end of equations.