Inferring Rewards from Language in Context (2022)

Abstract: In classic instruction following, language like “I’d like the JetBlue flight” maps to actions (e.g., selecting that flight). However, language also conveys information about a user’s underlying reward function (e.g., a general preference for JetBlue), which can allow a model to carry out desirable actions in new contexts. We present a model that infers rewards from language pragmatically: reasoning about how speakers choose utterances not only to elicit desired actions, but also to reveal information about their preferences. On a new interactive flight–booking task with natural language, our model more accurately infers rewards and predicts optimal actions in unseen environments, in comparison to past work that first maps language to actions (instruction following) and then maps actions to rewards (inverse reinforcement learning).

## introduction language is a natural interface for systems like robots or personal assistants that interact with human users. one way to interpret language in these interactive settings is to train an instruction following agent: a model that learns to map commands like "go three steps forward to the door" to a sequence of actions in context (e.g., @xcite . instructions describe how an agent should act in an immediate context, but to build models that can generalize-carrying out a user's goals in new contexts and learning user preferences over repeated interactions-agents should also infer why actions are taken. grounding language to reward functions extends the standard instruction following setup in this way, representing the goals and preferences that underlie actions, and allowing agents to autonomously carry out correct actions in new contexts (e.g., @xcite . available flights: jetblue delta price stop length user's true reward function: user: i'd like the jetblue flight update reward posterior after observing language + context in each interaction price @xmath010 longest stop 2h delta price @xmath160 longest stop 2h delta price @xmath2100 longest stop 2h jetblue jetblue delta price stop length ? ? ? ? ? ? ? ? ... rather than literally describing their preferences in full. how do we infer general goals and preferences from utterances in these settings? consider a flight booking agent like the one in figure 1 . by inferring the user's reward function (indicating their preference for carrier, price, and other flight features) beyond just selecting the right flight, such a system would be able to autonomously book flights on behalf of the user in other instances. to do so, the system might use the actions the user commands as evidence about what they prefer, recovering rewards from actions using (language-free) techniques like inverse reinforcement learning (irl; @xcite . for example, the system can select a flight the user might like in a new instance by matching features from their past flight bookings. the key idea of our work is that the way that a user refers to their desired actions with language also reveals important information about their reward: the fact that they said "the jetblue flight" and not "the expensive flight" conveys what matters to them. intuitively, in settings with repeated interactions, utterances are optimized to communicate information that is generalizable-implicitly helping listeners make useful inferences for acting on a longer horizon. we implement this idea with a pragmatic model of how speakers (humans) generate such language: speakers choose utterances that both elicit reward-maximizing actions in a particular context and faithfully describe the reward. given an utterance, our model infers that the most likely rewards are the ones that would have made a speaker likely to choose that utterance. to evaluate our model, we construct and release a dataset for mapping language to rewards, flight-pref, containing natural language utterances from humans with underlying preferences. humans interact in a multi-turn flight booking game similar to figure 1 , where we provide a "user" player with a reward function representing flight preferences. the goal of the game is for the user to communicate these preferences in natural language to an "assistant" player, who is tasked with booking preferred flights for the user. we present this dataset as a challenging benchmark for reward learning from language and interaction. in our experiments, we show that our model can infer reward functions from natural language, improve reward estimates consistently over repeated interactions, and use inferred rewards to accurately select optimal actions in held-out environments. our full model obtains relative accuracy improvements of 12% when compared to models that only treat language as descriptions of actions. foot_0 2 related work instruction following. a long line of work on grounded instruction following has developed various methods for producing actions from language, including approaches that use intermediary structured semantic representations @xcite @xcite @xcite @xcite @xcite @xcite or map directly to primitive actions @xcite @xcite @xcite @xcite @xcite . all of these approaches interpret any given utterance (instruction) solely in the context that elicited the utterance, producing one particular sequence of actions. the method we present extends these approaches, using utterances to infer the rewards that underlie the actions that should be taken across a range of environments: both the context that elicited the utterance, and other unseen environments. reward learning. the majority of work on reward learning has been in the robotics and reinforcement learning communities and has not incorporated language, rather using techniques such as inverse reinforcement learning (irl; ng and russell 2000; @xcite @xcite to infer the rewards that underlie human demonstrations of actions. even works that incorporate language into reward learning also take this primarily actioncentric approach: either by using datasets pairing utterances with trajectories and using (languagefree) irl to then recover reward functions from trajectories @xcite , or learning an instruction-following model guided by a language-conditioned discriminator @xcite . the language in these settings are unambiguous commands, giving a complete description of a goal (e.g., "go to the red door"). in contrast, we are concerned with language used to guide agents in repeated interactions (where language may be a partial or ambiguous mix of instructions and reward descriptions). ## the flightpref task we design flightpref, a task for reward inference from natural language in the flight booking domain. flightpref is designed to simulate a simplified interaction with a flight booking agent, where users communicate with the agent via language to book flights from a set of options. effective agents must not only learn to book the preferred flight given an instruction in the immediate context (instruction following), but also learn the user's preferences over repeated interactions to book preferred flights in unseen contexts. we collect a dataset of natural language in a multi-turn game between a user (the "speaker") and an assistant (the "listener" agent). each flight is represented by a feature vector φ(ξ) ∈ r 8 (e.g., features of carrier, price, etc.). we assume the user has a linear reward function with parameters θ ∈ r 8 , specifying a reward for a particular flight r θ (ξ) = θ φ(ξ). in the first round of the game, the user and assistant observe a set of three flight options and the user provides an utterance to describe the flight they want (the optimal flight under the reward function), e.g., "the flight with the most stops." in each of the subsequent rounds, the user and assistant are presented with a new set of three flights. the assistant can either choose by guessing the user's preferred flight (under the same reward function), or prompt the user for another utterance describing the desired flight in the new set. if the assistant chooses but does so incorrectly, the user is prompted for another utterance describing the correct flight. both players are penalized if the assistant chooses incorrectly, and earn points if the assistant chooses correctly (with more points for each round the assistant can do so without asking for help). the user is thus incentivized to provide utterances that inform the agent which flight to choose, while enabling long-term success over later rounds. figure 4 : sample text from the task, exhibiting a diversity of instructive and reward-descriptive language. ## model implementation our pragmatic model (section 3.1) relies on base listener and speaker models l base and s base . in this section, we describe implementations of these models for the flightpref dataset. to train the base models, we use the speaker-side data of (utterance, option set, reward function) tuples from each round. our base listener and speaker models assume that the utterances are generated conditionally independently given the reward; we capture the dynamics of multiple turns in the posterior reward inference. both base models learn neural encodings of utterances u, actions ξ, and rewards θ, and produce distributions by applying softmax functions to inner products between these encodings. we use ξ * to denote the optimal action in each context, i.e., ξ * = arg max ξ r θ (ξ). base listener model. the base listener model l base is defined using inner product similarities between learned representations of actions ξ produced by an mlp encoder, and learned representations of utterances produced by a bert-base @xcite encoder: we set the rationality parameter β = ∞ in p opt as speakers tend to refer primarily to the optimal option in our domain. base speaker model. the base reward speaker model s base is defined using an inner product between representations of rewards θ from an mlp encoder, and utterance representations from a bert encoder: training. we fine-tune all model parameters, including the parameters of the initially-pretrained bert utterance encoders in the listener and speaker on (u, ξ, m) pairs from the training data using the adamw optimizer @xcite . the listener and speaker models are trained separately, without sharing any parameters between the encoders used in the two models. we independently train 5 random seeds of each base model and ensemble them together in evaluation by averaging their output probabilities, which we found improved performance of all models (both our full model and baselines). see section c in the appendix for details and model hyperparameters. ## experiments we evaluate models in the same repeated turn setup that humans carried out in the task. for each game, models play the role of the listener in that game, updating the reward posterior (section 3.1) after observing the utterance and option set in each round. our goal is to estimate rewards that allow the agent to carry out the person's preferences: choosing the optimal option (flight) in unseen contexts (sets of flight options). to that end, we directly compare models on held-out accuracy: on 1,000 randomly-generated sets of three options, how often the model's estimate of the reward, θ, selects the option that is optimal under the true reward. 3 we use the model's reward posterior mean as the estimate, θ = e p θ θ. we additionally provide com- figure 5 : multi-turn performance on held-out accuracy (left) and l2 distance to the true reward (right). we show the performance of each model for varying numbers of observed utterances for a given reward. we combine five-and six-utterance rounds as there @xmath3. our full action+belief model substantially outperforms an action-only model at all numbers of utterances (@xmath4.05), and performs comparably to or better than a belief-only model, with statistically significant benefits for 5+ utterances (@xmath5.05). for our full action + reward model, we set the nearsightedness parameter α = 0.5 for all posterior updates. we compare to an action-only model that uses only p action (i.e., setting α = 1.0). this model is representative of approaches from past work on language-conditioned reward learning (e.g., @xcite ) that infer rewards purely from the actions that utterances refer to. we also compare to a reward-only model that uses only p reward (inferring rewards purely from the utterance, without conditioning on actions, i.e., setting α = 0.0). for comparison to versions of our approach that remove pragmatic modeling, see section d.1 in the appendix. ## discussion & conclusion we presented a method for using natural language to infer reward functions: representing the goals, preferences, and intents underlying action. conceptually, our work builds on previous work on language grounding by exploring how language serves a dual purpose. utterances can refer directly to actions to be taken, as studied in instruction following. beyond that, they communicate information about "why" those actions should be taken, and what actions may be desirable in new contexts. to build language-guided agents that can interact with people over longer horizons, it may be useful to model this relationship between language, actions, and rewards. furthermore, language is ambiguous about both actions and goals. standard settings for studying pragmatics (e.g., reference games) address how to resolve ambiguity about what object or action the speaker is choosing to refer to. we have explored how these settings can be extended by considering the preferences underlying those choices. we introduced flightpref, a new dataset of naturalistic interactions between people in a multi-turn flight booking game. flightpref uses held-out accuracy as a metric for evaluating interpretation success beyond selecting the right action in a single environment. future work can build on the task by 1) learning or evaluating with more complex reward functions (e.g., using deep reward representations); 2) exploring how people communicate about their real preferences and modeling a natural prior (e.g., that people tend to prefer cheaper flights), instead of providing annotators with ground-truth preferences; 3) allowing other ways to handle uncertainty, e.g., leveraging the reward posterior to interactively learn to ask; or 4) extending these approaches to other domains where modeling goals and preferences may be important (e.g., language-conditioned robotics).