How to talk to your model: Instructions, descriptions, and learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
From the earliest years of our lives, humans use language to express our beliefsand desires.
Being able to talk to artiﬁcial agents about our preferences would thusfulﬁll a central goal of value alignment.
Yet today, we lack computational modelsexplaining such ﬂexible and abstract language use.
To address this challenge, weconsider social learning in a linear bandit setting and ask how a human mightcommunicate preferences over behaviors (i.e. the reward function).
We study twodistinct types of language: instructions , which ground to concrete actions, anddescriptions , which provide information about the reward function.
To explain howhumans use these forms of language, we suggest they reason about both knownpresent and unknown future states: instructions optimize for the present, whiledescriptions generalize to the future. We formalize this choice by extending rewarddesign to consider a distribution over states.
We then deﬁne a pragmatic listeneragent that infers the speaker’s reward function by reasoning about how the speakerexpresses themselves.
We validate our models with a behavioral experiment,demonstrating that (1) our speaker model predicts spontaneous human behavior,and (2) our pragmatic listener is able to recover their reward functions.
Finally, weshow that in traditional reinforcement learning settings, pragmatic social learningcan integrate with and accelerate individual learning.
Our ﬁndings suggest thatsocial learning from a wider range of language—in particular, expanding the ﬁeld’spresent focus on instructions to include learning from descriptions—is a promisingapproach for both value alignment and reinforcement learning more broadly.
Introduction
As artiﬁcial agents proliferate in society, aligning them with human values is increasingly important [1–3].
But how can we build machines that understand what we want?
Prior work has highlighted thedifﬁculty of specifying our desires in numerical reward functions [3–5].
Here, we explore language asa means to communicate them.
While most previous work on language input to AI systems focuseson instructions [6–19]—we study instructions alongside more abstract, descriptive language [ how reward functions imagine
How would you learn the rewards associated with different fungi (i e. which are delicious and which are deadly)?
In such a setting, learning from direct experience [25] is risky; most humans would seek to learn socially instead.
How might we learn reward functions from others? Prior work
(RL) has examined a number of social learning strategies, including passive inverse reinforcement learning (observe an expert pick mushrooms, then infer their rewardfunction [26, 27]) or active preference learning (offer an expert pairs of mushrooms, observe whichone they eat, and infer their reward function [28–30]).
However, few humans would rely on such indirect data if they had access to a cooperative teacher [31–34].
For example, an expert guiding a foraging trip might demonstrate [35, 36] or verbally instruct [7]the learner to pick certain mushrooms, licensing stronger inferences.
While such pedagogical actionshave been useful for guiding RL agents [10, 17–19], natural language affords richer, under-exploredforms of teaching.
For example, an expert teaching a seminar might describe how to recognize edibleor toxic mushrooms based on their features. 1 Descriptive language is particularly powerful if learnerscan expect experts to prioritize relevant and context-sensitive information [38, 39].
First, we present a formal model of cooperative social learning in a linear bandit setting (Fig 1).
We propose a speaker model that chooses utterances to maximize the listener’s expected rewardsover some task horizon , generalizing reward design [40] to a distribution over states.
We formalizeand analyze instructions (which designate a speciﬁc action to take) and descriptions (which teachthe reward function).
We ﬁnd that short-horizon speakers (focused on a single, known state) preferinstructions, while long-horizon speakers (reasoning about future, unknown states) prefer descriptions.
Second, we consider how a listener might learn from such a speaker.
We deﬁne a pragmatic listenerthat performs inverse reward design [IRD, 4] to learn about rewards from instructions and descriptions.
While prior work suggests pragmatic learning can be vulnerable to model mis-speciﬁcation [41], weshow that jointly inferring the speaker’s horizon and reward function can mitigate this risk.
Finally, we conduct a behavioral experiment showing our models support strong reward inferencefrom human-chosen utterances.
We integrate this social inference with traditional RL, with socialinformation accelerating learning and reducing regret.
Overall, our results suggest that descriptivelanguage and pragmatic inference are powerful mechanisms for value alignment and learning.
Related work
Classic RL assumes that the reward function is given to the agent [25].
However, in practice, it is difﬁcult to specify a reward function to obtain desired behavior [3], motivating learning the reward function from social input.
2 Methods for learning reward functions often simply assume the expert issimply acting optimally, but recent pragmatic methods instead assume the expert is actively teaching.
Learning reward functions from observed actions.
When the desired behavior is known (but thereward function is not), inverse reinforcement learning [IRL, 12, 14, 26, 27, 45–47] can be used toinfer an expert’s reward function from their actions.
However, such approaches face fundamentalissues with identiﬁability: passively observed behavior can often be explained by multiple rewardfunctions.
This ambiguity can be addressed, for instance, by allowing the robot to actively query thehuman [29, 30, 48].
After reviewing work on learning rewards from language, we will turn to anotherimportant class of IRL approaches based on pragmatics [35, 41, 49, 50].
Learning reward functions from instructions.
Instructions use language to communicate speciﬁcactions or goals [6–8].
Prior work in reinforcement learning has recognized that instructions canreﬂect more general latent desires, and used them to shape [15–17] or infer reward functions [9–14].
Our work provides a theoretical basis for these models, situating them within a broader frameworkof humans’ language use: we ask when a human would want to use instructions (vs. other forms oflanguage), and—given an instruction (vs. other forms of language)—how a robot should generalize.
Learning reward functions from descriptions.
A relatively smaller body of work seeks to learnreward functions directly from existing bodies of text [20–22, 51] or interactive, free-form languageinput [23, 24]. While reward-descriptive language directly encodes generalized information, relatedlines of work use other “descriptive” language, typically instructions or trajectory annotations, toassist in (hierarchical) RL.
Rather than learn the reward function, these approaches use externallyprovided [52–58] or internally-generated language [59–62] to help structure the task representationor guide exploration. While the education literature suggests rich descriptive feedback is crucial forhuman teaching [63, 64], it poses a severe grounding challenge [65, 66] as the language no longermaps directly to the agent’s action or perception space.
Here, we assume groundings are known; thisallows us to characterize the utility of reward-descriptive language to gain theoretical understanding.
Learning reward functions from pedagogy.
The previous algorithms we reviewed all assume thattraining examples (whether demonstrations or language) are generated by a human that is indifferentto how they learn .
Recent work has begun to challenge this assumption by considering pedagogicalsettings [33–36].
In particular, the Rational Speech Act framework [67, 68] builds on classic Griceantheories [69] to formulate optimal communication in terms of recursive Bayesian inference anddecision-making.
These ideas have been applied to a variety of language tasks including referencegames [70, 71], captioning [72], and instruction following [73].
Developing analogues of (linguistic) pragmatics in reinforcement learning — i.e., algorithms thatassume data are intentionally designed to be informative — is currently an active area of research [35,41, 49, 50].
In particular, inverse reward design (IRD [4]) applies pragmatic inference directly to(numerical) reward functions: rather than take the provided reward function literally, the IRD agentquantiﬁes its uncertainty over the function to mitigate alignment risk.
In this work, we unify thesedifferent strains of pragmatic reasoning, extending IRD to a broader class of natural language input.  Social learning as reward design
Our theoretical approach is based on the Rational Speech Act framework [67, 68], which formalizespragmatics as recursive reasoning.
It begins by deﬁning a literal listener L 0 who interprets utterancesusing the literal meanings of the language.
A pragmatic speaker S 1 then (soft-)optimally selectsutterances u to maximize a utility, typically the epistemic goal of increasing the literal listener’sbeliefs in the true world state w ⇤ : U ( u ; w ⇤ )
=
ln P L 0 ( w ⇤ | u ) .
A pragmatic listener L 1 then infers the intended meaning of a given utterance by assuming it was generated by the speaker:
[Equation]
where L is a “lexicon” function mapping utterances to states and  S 1 > 0 is an inverse temperatureparameter.
This recursion can be continued [74], but the behavior of interest emerges at the L 1 level.
3.2 Social learning in linear bandits
We consider social learning in a linear bandit setting, a popular testbed for RL [75–79].
Formally, wedeﬁne a set of A possible actions.
Actions are associated with a binary feature vector  : A ! { 0 , 1 } K(e.g.
a mushroom may be green (or not), or striped (or not)).
The rewards of each action are thendeﬁned as a linear combination of these features (e.g. green mushrooms tend to be tasty; this is acommon assumption in IRL [4, 26, 46]):
[Equation]
so w is a vector that deﬁnes the value of each feature (see Fig.
1A).
Each task consists of a sequenceof H i.i.d.
states.
At each time step t < H , the agent is presented with a state s t consisting of asubset of possible actions: s t ✓ A (e.g., a particular mushroom patch).
They choose an action a 2 s taccording to their policy, ⇡ L : S !
( A ) .
While the bandit problem is typically considered an individual learning problem, we instead ask howan agent should learn from a cooperative, knowledgeable partner.
We formalize this social learningproblem by introducing a second agent: a speaker who knows the true rewards w and the initial states 0 , and produces an utterance u .
The listener updates their policy to ⇡ L ( a | u, s ) before beginning tochoose actions.
Intuitively, the horizon H determines how many actions the listener will performunder this updated policy.
H = 1 represents maximum supervision (i.e. guided foraging), whereas
H ! 1 is minimal supervision (teaching the listener to forage independently in future settings).
Weﬁrst assume H is known to both listener and speaker, then relax this assumption.
This frameworkexposes two interrelated problems.
First, what should a helpful speaker say?
And second, how shouldthe listener update their policy in light of this information?
3.3 Speakers as reward designers
Rather than deﬁning the speaker’s utility U ( u ; w ) as Gricean informativeness [69] (i.e. inducing truebeliefs, Section 3.1), we suggest that a cooperative speaker should maximize the listener’s rewards ,thus grounding utility in the listener’s subsequent actions.
When the state is known, the present utilityof an utterance is the expected reward from using the resulting policy to choose an action:
[Equation]
This formulation is equivalent to reward design [4, 40], where the reward designer chooses a proxyreward for a single, known MDP.
However, because only the ﬁrst state is known, we must alsoconsider how well the policy generalizes .
Thus, unlike reward design, speakers may reason aboutfuture states.
The future utility of an utterance with respect to a distribution over states P ( s ) is:
[Equation]
Because states are i.i.d.
in the bandit setting, a speaker optimizing for a horizon H can be deﬁned asa linear combination of Eqs.
5 and 6:
[Equation]
where H = 1 reduces to Eq.
5 and as H !
1 reduces to Eq. 6.
The following theorem shows thatincreasing the speaker horizon increases the expected utility of the listener across the distribution offuture states. This is because the speaker’s incentive to bias the proxy reward to the local context isreduced as the horizon they consider grows. Theorem 1 (Reward design over longer horizons) . Let S be a set of states that each represent adecision context.
Let ⇡ L be an arbitrary listener policy. Consider the behavior of a speaker thatchooses utterances u based on Eq. As H ! 1 , the future utility generated by ⇡ L increases.
The proof for this and other theorems can be found in Appendix A.
3.4 Formalizing instructions
We now consider how utterances affect the listener’s policy.
Instructions map to speciﬁc actions ortrajectories [7, 12]; in our work, “instruction” utterances correspond to the nine actions (Fig 1A).
Given an instruction, a literal listener executes the corresponding action.
If the action is not available,the listener acts randomly:
[Equation]
where
J u
K ( a ) represents the meaning of u , evaluating to one when utterance u grounds to a and zerootherwise.
An instruction is a partial policy : it designates the correct action in a subset of states.
This formalization of instructions allows us to identify an overlap between pragmatic inference fromdemonstrations, instructions, and reward design.
In learning from demonstrations with, e.g., IRL,we assume that demonstrations are drawn from a distribution of optimal behavior.
Thus, inferenceidentiﬁes reward functions such that the demonstrated trajectories have high reward in comparisonto alternative trajectories available to the demonstrator.
We will show that this is the same as: (1)pragmatic inference over instructions, under the assumption that instructions have a one-to-onemapping to available trajectories; and (2) pragmatic inference over true reward given a proxy, underthe assumption that the set of possible proxy rewards has a one-to-one mapping with trajectories. This reduction allows us to connect inference across these modalities and apply the results from, e.g., Theorem 1 to show that with a longer speaker horizon improves expected listener performance. Theorem 2 (Reward design with instructions is equivalent to demonstrations.) .
Let s be a state,represented as a local context with a set of actions a 2 s that can be taken.
Let U instruct be a setof instruction utterances that reference each a 2 s and let ⇠ R be the following set of (proxy) rewardfunctions { R ( a )
=
I [ a = a 0 ] | a 0 2 s } , where I represents the indicator function.
Then, the posteriordistribution obtained after an observation of noisily-optimal behavior is the same as that obtainedfrom a speaker maximizing Eq.
(5) and that obtained from IRD with the set of proxies ⇠ R .
3.5 Formalizing descriptions
Rather than mapping to a speciﬁc action, descriptions provide information about the reward function [23, 24, 47].
Following [80], we model descriptions as providing the reward of a single feature,similar to feature queries [30].
Descriptions are thus a tuple: a one-hot binary feature vector anda scalar value, h 1 K , R i .
These are messages like h Blue, -2 i .
In this work, we consider the set of6 features ⇥ 5 values in [  2 ,  1 , 0 , 1 , 2] , yielding 30 descriptive utterances.
Formally, L 0 “rules out”inconsistent hypotheses about reward weights w :
[Equation]
where
J u
K ( w ) represents the meaning of u , evaluating to one when u is true of w and zero otherwise.
In this work, we assume P ( w ) is uniform and there is no correlation between weights.
The listenerthen marginalizes over possible reward functions to choose an action:
[Equation] [Equation]
where  L 0 is again an inverse temperature parameter.
4 listener knows the meaning of utterances.
Due to the recursive structure of speaker and listener, simultaneously varying  S 1 and  L 0 introduces identiﬁability issues.
We therefore ﬁx  L 0 = 3 throughout this work, and only vary  S 1 .
This implies that all speakers address the same listener, but do so with different degrees of optimality.
3.6 Comparing instructions and descriptions
Prior work suggests that humans use a mix of instructive and descriptive language [23, 24].
Whatmodulates this—when should a rational speaker prefer instructions over descriptions?
To explore theeffects of speaker horizons on choice of utterance, we simulate a nearly-optimal speaker (  S 1 = 10 ).
Fig.
1A shows our bandit setting.
We assume the listener begins with a uniform prior over rewardweights throughout, and use states consisting of three unique actions (giving 84 possible states).
Fig 1B shows how the speaker’s choice of utterance varies as their horizon lengthens.
At shorthorizons, the speaker optimizes for present rewards (Eq. 5) and chooses instructions and descriptionsthat target the “Spotted Red” action.
At longer horizons, future rewards (Eq. 6) play a larger role,and the speaker blends the two objectives (Eq. 7) by describing the highly negative blue feature.
Finally, at sufﬁciently long horizons, future rewards dominate and it settles on describing the greenfeature—which is irrelevant to the start state, but the most important feature for generalization.
Toquantify how rational speakers should use instructions and descriptions, we repeat the task for all84 start states using horizons ranging 1-10 and different utterance sets.
Fig 2A plots rewards forspeakers with access to only instructions or descriptions, illustrating why this shift from instructions todescriptions occurs.
Instructions outperform descriptions at short horizons (achieving the theoreticalmaximum average reward of 1.75); as the horizon lengthens, however, descriptions generalize muchbetter.
Overall, as their horizon lengthens, speakers with access to both instructions and descriptionschoose descriptions exclusively, producing highly generalizable information (see Appendix B).
3.7 Learning from Utterances: a Pragmatic Listener
We now ask how the listener should learn from the speaker’s utterance, using pragmatic inference torecover information about the speaker’s reward function.
Known Horizon.
Following the standard RSA formulation, a pragmatic listener L 1 can invert thespeaker model. If the speaker’s horizon H is known, this is equivalent to inverse reward design [4]:
[Equation]
Given an instruction, L 1 infers the reward weights that would make such an instruction optimal [9, 14];given a description, L 1 can recover information about features that were not mentioned [23].
The L 1 listener then chooses actions by substituting this posterior belief into Eq. 10.
In practice, however,the speaker’s horizon is not known, and so this approach is not feasible for real-world applications.
This model thus serves as a theoretically-optimal baseline, and we next consider three practical waysof handling this uncertainty.
Assuming the horizon.
Perhaps the most straightforward approach is to simply assume a speakerhorizon.
However, prior work has highlighted the risks of assuming a human is actively trying toteach [41], and suggests that the safest approach is to assume they are not.
To test the effects ofsuch mis-speciﬁcation, in Section 4 we use two pragmatic listener models which assume a short( H = 1 ) or long ( H = 4 ) horizon.
A pragmatic listener assuming H = 1 will constrain inference: itwill assume the utterance reﬂects only the speaker’s preference between actions in the present state,and thus generalize conservatively.
In contrast, a pragmatic listener assuming H  1 expects theutterance to generalize broadly, risking overﬁtting .
Inferring the horizon.
To mitigate the risk of horizon misspeciﬁcation, we can instead assume thespeaker’s horizon is unknown.
Given an utterance, the latent-horizon pragmatic listener jointly infersboth their horizon and rewards, then marginalizes out the horizon:
[Equation]
Intuitively, because long-horizon speakers prefer instructions (and short-horizon speakers preferinstructions, Fig.
1B), the latent-horizon pragmatic listener can use the utterance type to infer thespeaker’s horizon and determine the appropriate scope of generalization.
To demonstrate this, wesimulate a pragmatic listener with a uniform prior over H 2 [1 , 2 , 3 , 4 , 5 , 10] .
Fig.
2B shows exampleutterances and resulting inference about the speaker’s rewards and latent horizon.
Crucially, whileboth example utterances indicate a preference for the spotted red mushroom, the description suggeststhe speaker is H > 1 and uniquely identiﬁes the spotted feature as high-value.
Behavioral Experiment
To validate our theoretical models, we collected a behavioral dataset.
Participants played the role of amushroom foraging guide and produced utterances for tourists to help them choose good mushrooms.
We manipulated the speaker’s horizon by varying tourists’ itineraries: each tourist was shown visitinga different visible mushroom patch, plus a variable number of unknown future patches (0, 1, or 3,matching horizons 1, 2, and 4 respectively).
In the following sections, we analyze the participants’choice of utterances, compare the resulting inference from different listener models, and show howthis socially-learned information can accelerate traditional reinforcement learning.
4.1 Experiment Setup
We recruited 119 participants on the Proliﬁc experimental platform ( prolific.co ).
Participantswere trained and tested on the game dynamics then advised a total of 28 tourists.
They were toldto consider the tourists’ itinerary and choose “the most helpful utterance” from drop-down menusallowing them to specify an instruction or description.
Because the space of possible descriptions(30) is larger than possible instructions (9), choosing a description required more effort than choosingan instruction.
To equalize this, we reduced the set of descriptions by removing neutral features (“red”and “solid”) and the 0 value, yielding 4 ⇥ 4 = 16 possible descriptive utterances.
For the remainderof the paper, all pragmatic inference listeners assume the speaker chooses from this reduced utteranceset.
Participants received no feedback on how their utterances affected tourists’ behaviors, ensuringthey chose utterances according to their own sense of how to help.
After screening out participantswho failed comprehension or attention checks, we were left with 99 participants who produced a totalof 2772 utterances.
For more details on the data collection, see Appendix C.
We next use this set of utterances to explore (1) exclusively social learning (i.e. whether our modelscan infer the speakers’ reward function), and then (2) integrated social and reinforcement learning(i.e. whether this socially-learned information can reduce regret in traditional RL).
4.2 Inferring rewards: social learning from human utterances
Overall, our results validated our theoretical models.
First, participants were sensitive to the horizonmanipulation: there was a statistically-signiﬁcant shift in their utterance choices between horizons(for H = 1 to 2 ,  2 (26 , 1856)
= 336 .
1 , p < .
001 ; for H = 2 to 4 ,  2 (26 , 1833)
= 40 .
3 , p = . 04 ).
Almost all participants (96 out of 99) used a mix of instructions and descriptions, favoring instructionsat H = 1 and descriptions at H > 1 (Fig. 3A).
This led to lower (literal) future rewards when H = 1 ,and higher future rewards at longer horizons (Fig 3B, “Empirical Results”).
To calibrate our pragmaticlisteners, we tested  S 1 2 [1 , 10] and found that  S 1 = 3 optimized both Known H and Latent Hlisteners.
Simulating utterances produced by our speaker model and resulting pragmatic inference amatch to theoretical predictions (Fig.
3B, “Model Predictions”, see Appendix C for details).
Our pragmatic listeners offered statistically-signiﬁcant improvements over the literal listener (Table 1).
These gains were particularly large when the speaker has a short horizon, matching our simulations.
Somewhat surprisingly, our Latent H model ( M = 0 .
94 , SD = 0 . 39 ) outperformed all othermodels, including the Known H model ( M = .
93 , SD = .
40 ), by a signiﬁcant albeit small margin(mean difference 0.01, paired-samples t-test t (2771) = 4 .
18 , p < .
001 ; see Appendix D for otherpairwise tests).
This is particularly notable because it underscores the inevitability of model misspeciﬁcation [41]: even when we experimentally controlled the horizon, participants did not perfectlyfollow our theoretical models, leading our Latent H model to outperform.
We also conﬁrmed therisks of assuming a horizon: the two ﬁxed-horizon listeners ( H = 1 and H = 4 ) underperformed the Latent H listener.
Despite these successes, we found a notable discrepancy between theoretical predictions and empiricalresults, suggesting a possible future reﬁnement.
As described in [80], utility-maximizing speakersagnostic to the truth of an utterance may send messages wtih exaggerated values (e.g. preferringthe false utterance h
Spotted, +2 i to the true utterance h
Spotted, +1 i ).
In our experiment, however,participants regularly chose the lower-reward, true utterance (Appendix C).
Intriguingly, breaking outresults by utterance type indicates that pragmatic gains come primarily on instructions, with pragmaticmodels on average faring slightly worse than the literal on descriptive utterances (Appendix E).
Thissuggests that our reward-design objective is in fact too weak: participants’ tendency to use truedescriptions licenses stronger inference, which we return to in the discussion.
4.3 Why learn socially? Reinforcement learning with social priors
We now explore how this form of social learning could augment traditional RL approaches.
Inparticular, we take the inferences from our listeners in the previous section and use them as a prior
Listener
Future rewards
Regret after from utterance 25 actions
Individual
[Table]
for reinforcement learning (or in the case of literal instructions, we integrate them into the learner’spolicy; see Appendix F for details on the experimental setup).
We then use Thompson sampling [81–83] to learn the reward function and compare regret over the course of learning for ﬁve independentlearning trials on each context-utterance pair.
We report results for our listener agents, as well as anagent with no social information (the “Individual” agent; Fig. 4, Table 1).
We ﬁnd that social information—and in particular, pragmatic social learning—can substantiallyreduce regret.
Intriguingly, when comparing regret, our Known H pragmatic listener now achievesthe best results; this is likely due to the greater uncertainty inherent in the Latent H inference.
Toconﬁrm these differences are statistically signiﬁcant, we use a linear regression to predict regretwith a ﬁxed effect of listener and random effects for each of the 2772 utterance-context pairs, againcomparing the Latent H listener to all other listeners.
This conﬁrms that the Known H listenerachieves lower regret than the Latent H listener (  =  .
12 , t (80400)
=
2 .
14 , p = .
03 ), and allother models suffer higher regret (see Appendix D).
Finally, while the literal listener achieves lowerregret initially than the pure reinforcement learner, it subsequently asymptotes as it rigidly obeysinstructions (similar to the issue raised in [14]).
This suggests that while instructions provide usefulguidance early in learning (as they help the agent choose and learn from good actions), a moreeffective instruction-following strategy could be to gradually shift to belief-based actions.
Discussion
We introduced a unifying model of communication as reward design [40] to explain humans’ useof instructions and descriptions, allowing pragmatic inference of their reward functions—a criticalcapability for value alignment [3, 49, 84].
Analyses show that instructions are optimal at shorthorizons, but descriptive language affords much stronger generalization.
This allows our pragmaticlistener to perform inverse reward design [4] to jointly infer the speaker’s horizon and rewardfunction, reducing risk of model mis-speciﬁcation in pragmatic inference [41].
Finally, our behavioralexperiment shows that humans follow our theoretical predictions, demonstrating the beneﬁts ofpragmatic inference for value alignment and accelerating traditional reinforcement learning.
We note several limitations and future directions.
First, our behavioral experiment suggests possibleimprovements to our speaker model: participants displayed a bias towards truthful descriptions. This licenses stronger pragmatic inference [85] such as enriching our model with traditional RSAobjectives [67, 68], or reformulating it to minimize regret during individual learning [86].
Second,for theoretical clarity and experimental feasibility, we used a simple bandit setting and pre-deﬁnedutterances.
Future work could address more challenging environments and open-ended language. More generally, our work provides a theoretical basis for further work at the intersection of social andreinforcement learning.
Linear bandits are already a popular testbed for RL [75, 77].
We hope that ourformalization of social learning in linear bandits supports more work in this setting [23, 24].
Finally,while most previous work focuses on instructions, our results highlight the power of descriptions forgeneralization, illustrating the potential of learning from richer forms of language.