Paper:
In the multi-objective reinforcement learning (MORL) paradigm, the relative importance of each environment objective is often unknown prior to training, so agents must learn to specialize their behavior to optimize different combinations of environment objectives that are specified post-training. These are typically linear combinations, so the agent is effectively parameterized by a weight vector that describes how to balance competing environment objectives. However, many real world behaviors require non-linear combinations of objectives. Additionally, the conversion between desired behavior and weightings is often unclear. In this work, we explore the use of a language based on propositional logic with quantitative semantics-in place of weight vectors-for specifying non-linear behaviors in an interpretable way. We use a recurrent encoder to encode logical combinations of objectives, and train a MORL agent to generalize over these encodings. We test our agent in several grid worlds with various objectives and show that our agent can generalize to many never-before-seen specifications with performance comparable to single policy baseline agents. We also demonstrate our agent's ability to generate meaningful policies when presented with novel specifications and quickly specialize to novel specifications.

Review:
This paper proposes using logical specifications to facilitate Q-learning in multi-objective reinforcement learning (MORL). Empirically the proposed method can generalize to unseen reward specifications with performance competitive to agents being fully trained in the new environment. The proposed setting employs a more expressive objectives space induced by propositional logic. The proposed method uses a recurrent encoder to embed specifications into vectors and uses them to parametrize the Q-function. 
Overall, I weakly recommend accepting the submission for the following reasons: (+) it proposes using propositional logic to specify reward functions, which broadens the objective space in an interpretable way, (+) the learned objective embedding demonstrates the ability to generalize to unseen environments(rewards). However, there is still room for improvement. I will raise my score if the following problems are addressed: (-) Needs more diverse experiments (instead of grid worlds) to support the paper (-) It might be hard to express objectives using logic formulas in real-world applications.
More specifically, the problems of the paper are,
(-) The scalability issue. The most complicated problem in the experiments has a 20x20 state space, which is pretty small for a typical RL problem. I wonder whether the model is still generalizable for larger problems. Even in the case of 20x20 grids, we can notice the gap between the baseline and the proposed method. 
(-) The assumption. This paper assumes the logical specifications are given by human. However, we usually don't know how to describe the true objective with logic formulas. Sometimes, finding the specification(reward) itself is as difficult as finding a good policy. Is it possible that we can relax this assumption?
Minor comments:
Figure 6: The word "baseline" here is misleading. A better choice would be "upper-bound"?