How to talk so your AI will learn: Instructions, descriptions, and autonomy
Anonymous Author(s)
AfﬁliationAddress email
Abstract
From the earliest years of our lives, humans use language to express our beliefsand desires.
Being able to talk to artiﬁcial agents about our preferences wouldthus fulﬁll a central goal of value alignment.
Yet today, we lack computationalmodels explaining such language use.
To address this challenge, we formalizelearning from language in a contextual bandit setting and ask how a human mightcommunicate preferences over behaviors.
We study two distinct types of language:instructions , which provide information about the desired policy, and descriptions ,which provide information about the reward function.
We show that the agent’sdegree of autonomy determines which form of language is optimal: instructionsare better in low-autonomy settings, but descriptions are better when the agent willneed to act independently.
We then deﬁne a pragmatic listener agent that robustlyinfers the speaker’s reward function by reasoning about how the speaker expressesthemselves.
We validate our models with a behavioral experiment, demonstratingthat (1) our speaker model predicts human behavior, and (2) our pragmatic listenersuccessfully recovers humans’ reward functions.
Finally, we show that this formof social learning can integrate with and reduce regret in traditional reinforcementlearning.
We hope these insights facilitate a shift from developing agents that obeylanguage to agents that learn from it.
Introduction
As artiﬁcial agents proliferate in society, aligning them with human values is increasingly important [1–3].
But how can we build machines that understand what we want?
Prior work has highlighted thedifﬁculty of specifying our desires via numerical reward functions [3–5].
Here, we explore languageas a means to communicate them.
While most previous work on language input to AI systems focuseson instructions [6–19], we study instructions alongside more abstract, descriptive language [20–25].
We examine how humans communicate about rewards and formalize learning from this input.
To consider how humans communicate about reward functions, imagine taking up mushroom foraging.
How would you learn the rewards associated with different fungi (i.e. which are delicious and which are deadly)?
In such a setting learning from direct experience [26] is risky; most humans would seek to learn socially instead
So how might we learn reward functions from others work social infer their reward expert pairs observe which
However, few humans would rely on such indirect data if they had access to a cooperative teacher [32–35].
For example, an expert guiding a foraging trip might demonstrate [36, 37] or verbally instruct [7]the learner to pick certain mushrooms, licensing stronger inferences.
While such pedagogical actionshave been useful for guiding RL agents [10, 17–19], natural language affords richer, under-exploredforms of teaching.
For example, an expert teaching a seminar might instead describe how to recognizeedible or toxic mushrooms based on their features, thus providing highly generalized information.
To formalize this process, we present a model of learning from language in a contextual bandit setting(Fig 1).
We propose a speaker model that chooses utterances to maximize the listener’s expectedrewards over some task horizon . The horizon quantiﬁes notions of autonomy described in previouswork [13, 14, 25].
If the horizon is short, the agent is closely supervised; at longer horizons, they areexpected to act more and more independently.
We then analyze instructions (which provide a partialpolicy) and descriptions (which provide partial information about the reward function).
We show thatinstructions are optimal at short horizons, while descriptions are optimal at longer ones.
Second, we consider how a listener might learn from such a speaker.
We deﬁne a pragmatic listenerthat infers the speaker’s latent reward function based on their utterances.
While prior work suggestspragmatic learning can be vulnerable to model mis-speciﬁcation [38], we show that jointly inferringthe speaker’s horizon and reward function can mitigate this risk.
Finally, we conduct an online behavioral experiment showing our models support strong rewardinference.
We integrate this social learning with traditional reinforcement learning, ﬁnding that itaccelerates learning and reduces regret.
Overall, our results suggest that descriptive language andpragmatic inference are powerful mechanisms for value alignment and learning.
Related work
Classic RL assumes that the reward function is given to the agent [26].
However, in practice, itis difﬁcult to specify a reward function to obtain desired behavior [3], motivating learning thereward function from social input.
2 Most social learning methods assume the expert is simply acting optimally [27, 28, 42], but recent pragmatic methods [36, 43, 44] instead assume the expert is actively
Learning reward functions from observed actions.
When the desired behavior is known (but thereward function is not), inverse reinforcement learning [IRL, 13, 14, 27, 28, 42, 45, 46] can be usedto infer an expert’s reward function from their actions.
However, such approaches face fundamentalissues with identiﬁability: observed behavior can often be explained by multiple reward functions.
One solution allows the agent to actively query the human [30, 31, 47].
An alternative is to make astronger assumption: that the human is actively teaching [i.e. behaving pedagogically , 36, 43, 44]. We next review work on learning rewards from language, then return to these methods.
Learning reward functions from instructions.
Instructions use language to communicate speciﬁcactions or goals [6–8].
Prior work has used instructions to shape reward functions [15–17] or learna language-conditioned reward model [9–12]. An alternative approach uses instructions for valuealignment. This paradigm assumes instructions reﬂect underlying preferences and uses them to inferthis latent reward function [13, 14].
Our work extends this to incorporate other forms of language: weask when a human would use an instruction (vs. other forms of language), and—given an instruction(vs. other forms of language)—what an agent should infer about the speaker’s reward function.
Learning reward functions from descriptions.
Rather than expressing speciﬁc goals, rewarddescriptive language encodes abstract information about preferences or the world. The educationliterature suggests such rich feedback is crucial for human teaching [48, 49]. However, a relativelysmaller body of work uses it for RL: by learning reward functions directly from existing bodies oftext [20–23] or interactive, free-form language input [24, 25].
Our work provides a formal modelof such language in order to compare it with more typically studied instructions. Other relatedlines of work use language which describes agent behaviors . This language, whether externallyprovided [50–56] or internally generated [57–60], is typically used to structure the task representationor guide exploration.
In contrast, we study descriptive language which provides information abouttask-relevant properties of the environment.
Learning reward functions from pedagogy.
The preceding algorithms all assume that trainingexamples (whether demonstrations or language) are generated by a human that is indifferent to thelearning process .
Recent work has begun to challenge this assumption by considering pedagogicalsettings [34–37].
In particular, the Rational Speech Act framework [61, 62] builds on classic Griceantheories [63] to formulate optimal communication in terms of recursive Bayesian inference anddecision-making.
These ideas have been applied to a variety of language tasks including referencegames [64, 65], captioning [66], and instruction following [67].
Developing analogues of linguisticpragmatics in reinforcement learning — i.e., algorithms that assume data are intentionally designedto be informative — is an active area of research [36, 38, 43, 44].
In particular, inverse reward design(IRD, [4]) uses pragmatic inference on numerical reward functions: rather than take the providedreward function literally, IRD quantiﬁes uncertainty over the function to mitigate alignment risk. Similarly, we apply pragmatic inference to reward-related language.  Background
In this work, we consider the problem of learning from language in a contextual bandit setting. Thissection introduces models of language use from cognitive science, then describes contextual bandits.
The Rational Speech Act (RSA) framework . Our theoretical approach is based on the Rational Speech Act framework (RSA, [61, 62, 68–70]), a computational model of language understanding.
RSA uses theory-of-mind reasoning to explain how human listeners regularly derive additionalmeaning from words.
It begins by deﬁning a literal listener L 0 which considers only the conventional(e.g. dictionary) deﬁnitions of words.
These conventions are represented by a “lexicon” function
L which maps from utterances to world states w . 3 RSA then assumes that a speaker S 1 makes anoisy-optimal selection from a ﬁnite set of utterances, choosing u according to some utility function U ( · ) . Finally, a pragmatic listener L 1 infers the intended meaning by inverting the speaker model: [Equation]
In Eq. 2,  S 1 > 0 is an inverse temperature parameter.
RSA models typically assume the speaker’sobjective is informativeness .
Formally, the speaker tries to reduce the listener’s information-theoreticuncertainty [75] over the true state of the world w ⇤ :
U ( u, w ⇤ )
=
ln P L 0 ( w ⇤ | u ) .
Explicitly modelingthe speaker as making a rational choice from a set of possible utterances based on the world state wallows the pragmatic listener to make additional inferences beyond the literal content of the utterance. To take an intuitive example, we consider the classic phenomenon of scalar implicature [76–78]. If Alice announces “I ate some of the cookies,” a typical listener Bob understands this as “some butnot all .”
RSA explains this as follows. In Eq. 1, the word “some” maps to any world state where Alice ate more than zero cookies (e.g. 1, 2, 3... or all), while “all” maps only to the world state whereshe ate all of them.
In Eq. 2, S 1 Alice is presumed to choose a maximally-informative utterance(i.e. one that speciﬁes the world state as precisely as possible).
Then, in Eq. 3, L 1 Bob reasons asfollows: if Alice had eaten all of the cookies, saying “I ate all of the cookies” would have beenmore precise and thus preferred. Because she chose not to say this, he then can then infer that herstatement means “some but not all .” In this work, we deﬁne a pragmatic L 1 listener which uses RSAto infer the speaker’s reward function. Crucially, such inferences are based on theory-of-mind andthus vulnerable to misspeciﬁcation [38]: if L 1 Bob’s model of S 1 Alice is wrong, his inferences willbe wrong.
We show how a carefully speciﬁed speaker model, incorporating both instructions anddescriptions, allows our pragmatic listener to more robustly infer a speaker’s reward function. Contextual bandits. Contextual bandits can be seen as one-step Markov Decision Processes [26, 79],thus isolating the problem of learning a reward function from the problem of planning sequentialactions to maximize it.
As such, they are a popular testbed for RL algorithms [80–84].
Formally, wedeﬁne a set of A possible actions.
Actions are associated with a binary feature vector  : A ! { 0 , 1 } K(e.g.
a mushroom may be green (or not), or striped (or not)).
Following other work in IRL [4, 27, 42],we assume rewards are a linear function of these features (e.g. green mushrooms tend to be tasty):
[Equation]
so w is a vector that deﬁnes the value of each feature (see Fig.
1A).
Each task consists of a sequenceof H i.i.d.
states.
We refer to H as the horizon. At each time step t < H , the agent is presentedwith a state s t consisting of a subset of possible actions: s t ✓ A (e.g., a patch containing a set ofmushrooms).
They choose an action a 2 s t according to their policy, ⇡ L : S !
( A ) .
Formalizing learning from language in contextual bandits
While bandits are typically considered as an individual learning problem, we instead ask how an agentshould learn socially from a cooperative, knowledgeable partner.
We formalize this by introducinga second agent: a speaker who knows the true rewards w and the initial state s 0 , and produces anutterance u .
This utterance may affect the listener’s policy; with a slight abuse of notation, wedenote this updated policy as ⇡ L ( a | u, s ) . The listener then uses this policy to choose actions.
The horizon H determines how many actions the listener will perform under this updated policy.
Intuitively, H = 1 represents minimal autonomy (or maximal supervision, i.e. guided foraging),whereas H !
1 is maximal autonomy (or minimal supervision, i.e. teaching the listener to forageindependently).
We ﬁrst assume H is known to both listener and speaker, then relax this assumption.
This framework exposes two interrelated problems.
First, what should a helpful speaker say?
Andsecond, how should the listener update their policy in light of this information?
4.1 Speaker model
We apply the RSA framework (Section 3) to a RL setting by introducing a new speaker objective:instead of Gricean informativeness [62, 63] we assume that speakers seek to maximize the listener’s rewards [85]. Conceptually, this changes RSA’s scope from epistemic to decision-theoretic. Insteadof reasoning about how utterances affect the listener’s beliefs , such a speaker reasons about how theyaffect the listener’s policy .
When the state is known, we deﬁne the present utility of an utterance asthe expected reward from using the updated policy to choose an action:
[Equation]
However, we are interested in settings where agents may need to act autonomously [13, 24, 25].
Thus,we must consider how well the policy generalizes to unknown, future states.
The future utility of anutterance with respect to a distribution over states P ( s ) can be written as:
[Equation]
Because states are i.i.d.
in the bandit setting, a speaker optimizing for a horizon H can be deﬁned asa linear combination of Eqs.
5 and 6:
[Equation]
where H = 1 reduces to Eq.
5 and as H !
1 reduces to Eq. 6.
At short horizons, speakers arebiased towards producing utterances that optimize the listener’s policy in the present state.
As thehorizon lengthens, the speaker expects the listener to behave more autonomously.
Generalizationbecomes more important, and this bias is reduced.
4.2 Formalizing instructions
We now consider how utterances affect the listener’s policy.
Instructions map to speciﬁc actions ortrajectories [7, 13]; in our work, “instruction” utterances correspond to the nine actions (Fig 1A).
Given an instruction, a literal listener executes the corresponding action.
If the action is not available,the listener acts randomly:
[Equation]
where
J u
K ( a ) represents the meaning of u , evaluating to one when utterance u grounds to a and zerootherwise.
An instruction is a partial policy : it designates an action to take in a subset of states.
4.3 Formalizing descriptions
Rather than mapping to a speciﬁc action, descriptions provide information about the reward function [24, 25, 46].
Following [85], we model descriptions as providing the reward of a single feature,similar to feature queries [31].
Descriptions are thus a tuple: a one-hot binary feature vector anda scalar value, h 1 K , R i .
These are messages like h Blue, -2 i .
In this work, we consider the set of6 features ⇥ 5 values in [  2 ,  1 , 0 , 1 , 2] , yielding 30 descriptive utterances.
Formally, L 0 “rules out”inconsistent hypotheses about reward weights w :
[Equation]
where
J u
K ( w ) represents the meaning of u , evaluating to one when u is true of w and zero otherwise.
In this work, we assume P ( w ) is uniform and there is no correlation between weights.
The listenerthen marginalizes over possible reward functions to choose an action:
[Equation] [Equation]
where  L 0 is again an inverse temperature parameter.
4.4 Comparing instructions and descriptions
Prior work suggests that humans use a mix of instructions and descriptions [24, 25].
What modulatesthis—when should a rational speaker prefer instructions over descriptions?
To explore the effects ofhorizon on utterance utility, we simulate a nearly-optimal speaker (  S 1 = 10 ).
Fig.
1A shows ourbandit setting.
We assume the listener begins with a uniform prior over reward weights throughout,and use states consisting of three unique actions (giving 84 possible states).
Fig 1B shows how the speaker’s choice of utterance varies as the horizon lengthens.
At short horizons,the speaker optimizes for present rewards (Eq. 5) and chooses instructions and descriptions that targetthe “Spotted Red” action.
At longer horizons, future rewards (Eq. 6) play a larger role, and the speakerblends the two objectives (Eq. 7) by describing the highly negative blue feature.
Finally, at sufﬁcientlylong horizons, future rewards dominate and it settles on describing the green feature—which isirrelevant to the start state, but the most important feature for generalization.
To quantify how rationalspeakers should use instructions and descriptions, we repeat the task for all 84 start states usinghorizons ranging 1-10 and different utterance sets.
Fig 2A plots rewards for speakers with access toonly instructions or descriptions, illustrating why this shift from instructions to descriptions occurs.
Instructions outperform descriptions at short horizons (achieving the theoretical maximum averagereward of 1.75); as the horizon lengthens, however, descriptions generalize better.
5 Overall, as thehorizon lengthens, speakers with access to both instructions and descriptions choose descriptionsexclusively, producing highly generalizable information (see Appendix B).
4.5 Learning from utterances: a pragmatic listener
We now ask how the listener should learn from the speaker’s utterance. Following RSA (Section 3),a pragmatic listener L 1 inverts the speaker model to infer their latent reward function.
This approachis closely related to inverse reward design [4]: because the speaker maximizes rewards (Eq. 7), ourpragmatic listener effectively treats utterances as proxy rewards [86] speciﬁed in language.
Known horizon.
If the speaker’s horizon H is known, we can write a L 1 listener as:
[Equation]
Given an instruction, L 1 infers the reward weights that would make such an instruction optimal [13,14]; given a description, L 1 can recover information about features that were not mentioned [24, 25].
The L 1 listener then chooses actions by substituting this posterior belief into Eq. 10.
In practice,however, the horizon is not known, and so this approach is not feasible for real-world applications.
This model instead serves as a theoretically-optimal baseline, and we next consider three practicalways of handling this uncertainty.
Assuming the horizon.
Perhaps the most straightforward approach is to simply assume a speakerhorizon.
However, prior work has highlighted the risks of assuming a human is actively trying toteach [38], and suggests that the safest approach is to assume they are not.
To test the effects ofsuch mis-speciﬁcation, in Section 5 we deﬁne two pragmatic listener models which assume a short( H = 1 ) or long ( H = 4 ) horizon.
A pragmatic listener assuming H = 1 will constrain inference: itwill assume the utterance reﬂects only the speaker’s preference between actions in the present state,and thus generalize conservatively.
In contrast, a pragmatic listener assuming H  1 expects theutterance to generalize broadly, risking overﬁtting.
Inferring the horizon.
To mitigate the risk of horizon mis-speciﬁcation, we can instead assume thespeaker’s horizon is unknown.
Given an utterance, the latent horizon pragmatic listener jointly infersboth their horizon and rewards, then marginalizes out the horizon:
[Equation]
Intuitively, because short-horizon speakers prefer instructions (and long-horizon speakers preferdescriptions, Fig.
1B), the latent-horizon pragmatic listener can use the utterance type to infer thespeaker’s horizon and determine the appropriate scope of generalization.
To demonstrate this, wesimulate a pragmatic listener with a uniform prior over H 2 [1 , 2 , 3 , 4 , 5 , 10] .
Fig.
2B shows exampleutterances and resulting inference about the speaker’s rewards and latent horizon.
Crucially, whileboth example utterances indicate a preference for the spotted red mushroom, the description suggeststhe speaker is H > 1 and uniquely identiﬁes the spotted feature as high-value.
Behavioral experiment
To validate our theoretical models, we collected a behavioral dataset.
Participants played the role of amushroom foraging guide and produced utterances for tourists to help them choose good mushrooms.
We manipulated the speaker’s horizon by varying tourists’ itineraries: each tourist was shown visitinga different visible mushroom patch, plus a variable number of unknown future patches (0, 1, or 3,matching horizons 1, 2, and 4 respectively).
In the following sections, we analyze the participants’choice of utterances, compare the resulting inference from different listener models, and show howthis socially-learned information can accelerate traditional reinforcement learning.
5.1 Experiment setup
We recruited 119 participants on the Proliﬁc experimental platform ( prolific.co ).
Participantswere trained and tested on the game dynamics then advised a total of 28 tourists.
They were toldto consider the tourists’ itinerary and choose “the most helpful utterance” from drop-down menusallowing them to specify an instruction or description.
Because the space of possible descriptions(30) is larger than possible instructions (9), choosing a description required more effort than choosingan instruction.
To equalize this, we reduced the set of descriptions by removing neutral features (“red”and “solid”) and the 0 value, yielding 4 ⇥ 4 = 16 possible descriptive utterances.
For the remainderof the paper, all pragmatic listeners assume the speaker chooses from this reduced utterance set.
Participants received no feedback on how their utterances affected tourists’ behaviors, ensuring theychose utterances according to their own sense of how to help.
After screening out participants whofailed comprehension or attention checks, we were left with 99 participants who produced a total of2772 utterances.
For more details on the data collection, see Appendix C.1.
We next use this set of utterances to explore value alignment (whether our models can infer thespeakers’ reward function) and then integrated social and reinforcement learning (whether thissocially-acquired information can reduce regret in traditional RL).
5.2 Inferring rewards: value alignment from language
Overall, our results validated our theoretical models.
First, participants were sensitive to the horizonmanipulation: there was a statistically-signiﬁcant shift in their utterance choices between horizons(for H = 1 to 2 ,  2 (26 , 1856)
= 336 .
1 , p < .
001 ; for H = 2 to 4 ,  2 (26 , 1833)
= 40 .
3 , p = . 04 ).
Almost all participants (96 out of 99) used a mix of instructions and descriptions, favoring instructionsat H = 1 and descriptions at H > 1 (Fig. 3A).
This led to lower literal future rewards when H = 1 ,and higher future rewards at longer horizons (Fig 3B, “Empirical Results”).
To calibrate our pragmaticlisteners, we tested  S 1 2 [1 , 10] and found that  S 1 = 3 optimized Known H and Latent H listeners(see Appendix C.3 for details).
Simulating utterances produced by our speaker model and resultingpragmatic inference shows a close match to theoretical predictions (Fig.
3B, “Model Predictions”,see Appendix C.4 for details).
Our pragmatic listeners offered statistically-signiﬁcant improvements over the literal listener (Table 1).
These gains were particularly large when the speaker has a short horizon, matching our simulations.
Somewhat surprisingly, our Latent H model ( M = 0 .
94 , SD = 0 . 39 ) outperformed all other models,including the Known H model ( M = .
93 , SD = .
40 ), by a signiﬁcant albeit small margin (meandifference 0.01, paired-samples t-test t (2771) = 4 .
18 , p < .
001 ; see Appendix D for other pairwisetests).
This is particularly notable because it underscores the inevitability of misspeciﬁcation [38]:even when we experimentally controlled the horizon, participants did not perfectly follow our theory,leading our Latent H model to outperform.
We also conﬁrmed the risks of assuming a horizon: thetwo ﬁxed-horizon listeners ( H = 1 and H = 4 ) underperformed the Latent H listener.
Despite these successes, we found a notable discrepancy between theoretical predictions and empiricalresults, suggesting a possible future reﬁnement.
As described in [85], utility-maximizing speakersagnostic to the truth of an utterance may send messages wtih exaggerated values (e.g. preferringthe false utterance h
Spotted, +2 i to the true utterance h
Spotted, +1 i ).
In our experiment, however,participants regularly chose the lower-reward, true utterance (Appendix C).
Breaking out results byutterance type indicates that pragmatic gains come primarily on instructions (Appendix E).
Thissuggests that our reward-design objective is in fact too weak: participants’ tendency to use truedescriptions licenses stronger inference, which we return to in the discussion.
Listener
Future rewards
Regret after from utterance 25 actions
Individual
[Table]
[Table] 5.3 Reducing regret: integrating social and reinforcement learning We now explore how this form of social learning could augment traditional RL approaches. Inparticular, we take the inferences from our listeners in the previous section and use them as a priorfor reinforcement learning (or in the case of literal instructions, we integrate them into the learner’spolicy; see Appendix F for details on the experimental setup).
We then use Thompson sampling [87–89] to learn the reward function and compare regret over the course of learning for ﬁve independentlearning trials on each context-utterance pair.
We report results for our listener agents, as well as anagent with no social information (the “Individual” agent; Fig. 4, Table 1).
We ﬁnd that social information can substantially reduce regret.
Intriguingly, when comparingregret, our Known H pragmatic listener now achieves the best results; this is likely due to thegreater uncertainty inherent in the Latent H inference.
To conﬁrm these differences are statisticallysigniﬁcant, we use a linear regression to predict regret with a ﬁxed effect of listener and randomeffects for each of the 2772 utterance-context pairs, again comparing the Latent H listener to all otherlisteners.
This conﬁrms that the Known H listener achieves lower regret than the Latent H listener(  =  .
12 , t (80400)
=
2 .
14 , p = .
03 ), and all other models suffer higher regret (see Appendix D).
Discussion
We formalized the challenge of using language to teach an agent to act on our behalf [13, 14, 43, 44,81] in a bandit setting, introducing the notion of a horizon to reﬂect the agent’s degree of autonomy.
We ﬁnd instructions are optimal for low-autonomy settings, while descriptions are optimal for highautonomy.
This distinction allows a pragmatic listener to jointly infer the speaker’s horizon andreward function, reducing misspeciﬁcation risk [38].
Our behavioral experiment supports our theory,demonstrating the beneﬁt of learning from language for value alignment [1–3] and traditional RL [26].
We note several limitations and future directions.
First, human participants preferred truthful descriptions; this licenses stronger pragmatics [90] such as combining our model with epistemicobjectives [61, 62]. Second, we assumed language groundings were known, but future work couldincorporate uncertainty [73, 74] or learn them [24, 25, 72].
Third, we studied a simple contextual bandit setting; sequential decision settings will require models of planning, and may consider additionallanguage such as instructions at different levels of abstraction or transition descriptions [91]. Lastly,we modeled a simple interaction consisting of a single utterance.
More naturalistic interactions wouldallow multiple utterances, bidirectional dialogue, or interleaved action and communication [43, 44]. Our work can inform agent design by clarifying how and when different forms of linguistic inputcan be useful.
Instruction following may be optimal when autonomy is unnecessary or preferencesare non-Markovian [92].
However, if autonomy is desired [1–3, 13, 14, 38, 43, 44], agents shouldbe equipped to understand descriptions of the world or our preferences [21–25].
Finally, pragmaticagents can infer whether preferences are local or general. This suggests that learning from a widerange of language is a promising approach for both value alignment and RL more broadly.