Modeling Human Exploration through Resource-Rational Reinforcement Learning
Anonymous Author(s)
AffiliationAddress
Abstract
Equipping artificial agents with useful exploration mechanisms remains a challengeto this day.
Humans, on the other hand, seem to manage the trade-off betweenexploration and exploitation effortlessly.
In the present article, we put forward thehypothesis that they accomplish this by making optimal use of limited computational resources.
We study this hypothesis by meta-learning reinforcement learningalgorithms that sacrifice performance for a shorter description length (defined as thenumber of bits required to implement the given algorithm).
The emerging class ofmodels captures human exploration behavior better than previously considered approaches, such as Boltzmann exploration, upper confidence bound algorithms, and
Thompson sampling.
We additionally demonstrate that changing the descriptionlength in our class of models produces the intended effects: reducing descriptionlength captures the behavior of brain-lesioned patients while increasing it mirrorscognitive development during adolescence.
Introduction
Knowing how to efficiently balance between exploring unfamiliar parts of an environment andexploiting currently available knowledge is an essential ingredient of any intelligent organism.
Intheory, it is possible to obtain a Bayes-optimal solution to this exploration-exploitation dilemmaby solving an augmented problem known as a Bayes-adaptive Markov decision process (BAMDP,
Duff, 2003).
However, BAMDPs are intractable to solve in general and analytical solutions are onlyavailable for a few special cases [Gittins, 1979].
The intractability of this optimal solution promptedresearchers to develop a body of heuristic strategies [Auer et al., 2002, Kaufmann et al., 2012, Russoet al., 2017, Russo and Van Roy, 2014].
Most of these heuristics can be divided into two broadcategories: directed and random exploration strategies [Wilson et al., 2014, Schulz and Gershman,2019].
Directed exploration strategies provide bonus rewards that encourage the agent to visit parts ofthe environment that ought to be explored, whereas random exploration strategies inject some formof stochasticity into the policy.
Having access to a vast amount of existing exploration strategies leads to the question: which of themshould we use when building artificial agents?
To answer this question, we may take inspiration fromhow people approach the exploration-exploitation dilemma.
Human exploration has been studiedextensively in the multi-armed bandit setting [Mehlhorn et al., 2015, Wilson et al., 2021, Brändle et al.
Prior work indicates that people substantially deviate from the Bayes-optimal strategy even forthe simplest bandit problems [Steyvers et al., 2009, Zhang and Angela, 2013, Binz and Endres, 2019].
They, however, use uncertainty estimates to intelligently guide their choices through a combination of both directed and random exploration [Wilson et al., 2014, Gershman, 2018].
The question of whenand why individuals rely on a particular exploration strategy has been under-explored so far.
We take the first steps towards answering these questions by looking at human exploration froma resource-rational perspective [Gershman et al., 2015, Lieder and Griffiths, 2020, Binz et al.,2022].
More specifically, we investigate the hypothesis that people solve the exploration-exploitationdilemma by making optimal use of limited computational resources.
To test this hypothesis, wedevise a family of resource-rational reinforcement learning algorithms by combining ideas frommeta-learning [Bengio et al., 1991, Schmidhuber et al., 1996] and information theory [Hintonand Van Camp, 1993].
The resulting model – which we call resource-rational RL 2 (RR-RL 2 ) –implements a free-standing reinforcement learning algorithm that achieves optimal behavior subjectto the constraint that it can be implemented with a given number of bits.
We demonstrate that RR-RL 2 captures many aspects of human exploration by reanalyzing data fromthree previously conducted psychological studies.
First, we show that it explains human choices in atwo-armed bandit task better than traditional approaches, such as Thompson sampling [Thompson,1933], upper confidence bound (UCB) algorithms [Kaufmann et al., 2012], and mixtures thereof[Gershman, 2018].
We then verify that the manipulation of computational resources in our class ofmodels matches the manipulation of resources in human subjects in two different contexts.
Takentogether, these results enrich our understanding of human exploration and provide insights into howto improve the exploratory capabilities of artificial agents.
Methods
We start by describing the general problem setting considered in this article and its optimal solution,followed by a brief summary of the meta-reinforcement learning framework.
We then show how toaugment the standard meta-reinforcement learning objective with an information-theoretic constraint,allowing us to construct reinforcement learning algorithms that trade-off performance against thenumber of bits required to implement them.
2.1 Notation and Preliminaries
Each task considered in this article can be interpreted as a multi-armed bandit problem.
In a k -armedbandit problem, an agent repeatedly interacts with k slot machines that are associated with a rewarddistribution p ( r t | a t , ω ) with unknown parameters ω .
In each time-step, the agent selects an action a tand is provided a reward r t based on the associated reward distribution.
The goal of an agent is to select actions such that the sum of rewards over a finite horizon H ismaximized.
We assume that the agent additionally has access to a prior distribution p ( ω ) over banditproblems that it may encounter, which can be updated after observing a history of observationsh t :=
( a 1 , r 1 , . .
. , a t − 1 , r t − 1 ) by applying Bayes’ rule:
[Equation]
The policy that optimally trades-off exploration and exploitation can be obtained by reasoning how theagent’s beliefs about reward functions evolve with future observations [Martin, 1967].
Formally, thisis accomplished by transforming the original bandit problem into a corresponding BAMDP defined bythe tuple ( H , A , H, T, R ) .
In this augmented problem, H represents the set of all possible histories,while A and H correspond to the action space and the horizon of the original bandit problem.
Thetransition probabilities T and reward function R are given by:
[Equation]
) with the marginal reward probabilities:
[Equation]
The policy that maximizes the sum of rewards in the BAMDP corresponds to the Bayes-optimalpolicy for the original bandit problem.
2.2 Meta-Reinforcement Learning
While the BAMDP formalism provides a precise recipe for deriving a Bayes-optimal policy, findingan analytical expression of this policy is typically not possible.
Recent work on meta-reinforcementlearning, however, has shown that it is possible to learn an approximation to it
[Wang et al., 2016,
Ortega et al., 2019, Zintgraf et al., 2019].
Duan et al.
[2016] refer to this approach as RL 2 because ituses a traditional reinforcement learning algorithm to meta-learn another free-standing reinforcementlearning algorithm.
RL 2 parametrizes the to-be-learned reinforcement learning algorithm with a general-purpose functionapproximator.
Typically, this function approximator takes the form of a recurrent neural network thatreceives the last action and reward as inputs, uses them to update its hidden state, and produces a policythat is conditioned on the new hidden state.
Let W denote the parameters of this recurrent neuralnetwork.
In an outer-loop meta-learning process, the network is then trained on the prior distributionover bandit problems p ( ω ) to find the history-dependent policy π ( a t | h t , W ) that maximizes the sumof obtained rewards.
If the meta-learning procedure has successfully converged to its optimum, RL 2implements a free-standing reinforcement learning algorithm that mimics the Bayes-optimal policy.
Importantly, learning at this stage is fully implemented through the forward dynamics of the recurrentneural network and no further updating of its parameters is required.
2.3 Resource-Rational RL 2
We transform RL 2 into a resource-rational algorithm by augmenting its meta-learning objectivewith an information-theoretic constraint and refer to this resource-rational variant as RR-RL 2 .
Moreprecisely, RR-RL 2 is obtained by solving the following optimization problem:
[Equation]
The first component of Equation 5 corresponds to the standard meta-reinforcement learning objective,while the second component ensures that the Kullback–Leibler (KL) divergence between a stochasticparameter encoding q ( W | Λ ) and a prior p ( W ) remains smaller than some constant C .
The KL termcan be interpreted as the description length of neural network parameters, i.e., the number of bitsrequired to store them.
1 RR-RL 2 therefore optimally trades-off performance against the number ofbits required to implement the emerging reinforcement learning algorithm.
Note that the objectivefrom Equation 5 can also be motivated by a PAC-Bayes bound on generalization performance tounseen tasks [Yin et al., 2019, Rothfuss et al., 2020, Jose and Simeone, 2020].
While we focus on theresource-rational interpretation in the present article, we believe that both of these perspectives arecomplementary.
In practice, we solve a sample-based approximation of the optimization problem in Equation
usinga standard on-policy actor-critic algorithm
[Mnih et al., 2016, Wu et al., 2017].
We rely on a dualgradient ascent procedure [Haarnoja et al., 2018] to ensure that the constraint is satisfied.
Appendix Acontains a complete description of the network architecture, choices of prior and encoding distribution,and the meta-learning procedure.
[Figure]
D i m e n s i o n
Humans
RR-RL
Modeling Human Exploration
We now demonstrate that RR-RL 2 explains human choices on both a qualitative and quantitativelevel.
We first show that varying its description length leads to a set of diverse exploration strategies,allowing us to capture individual differences in human decision-making.
When reanalyzing data froma two-armed bandit benchmark
[Gershman, 2018], we furthermore find that RR-RL 2 beats previouslyconsidered algorithms in terms of fitting human behavior by a large margin.
Experimental Design: The behavioral data-set of Gershman [2018] contains records of 44 participants who each played 20 two-armed bandit problems with an episode length of H = 10 .
The meanreward for each arm a was drawn from p ( ω a ) =
N (0 , 10) at the beginning of the task and the rewardin each time-step from p ( r t | a t , ω ) =
N ( ω a t , 1) .
Analysis: To analyze the set of emerging exploration strategies, we adopted a method proposed by
Gershman [2018].
He assumed that an agent uses Bayes’ rule as described in Equation 1 to update itsbeliefs over unobserved parameters.
If prior and reward are both normally distributed, the posteriorwill also be normally distributed and the corresponding updating rule is given by the Kalman filteringequations.
Let p ( ω a | h t ) =
N ( µ a,t , σ a,t ) be the posterior distribution at time-step t .
Based on theparameters of this posterior distribution, he then defined the following probit regression model:
[Equation]
with Φ denoting the cumulative distribution function of a standard normal distribution.
Equation 6is also referred to as the hybrid model as it contains several known exploration strategies as specialcases.
We can recover a Boltzmann-like exploration strategy for w =
[ w 1 , 0 , 0] , a variant of the UCBalgorithm for w =
[ w 1 , w 2 , 0] , and Thompson sampling for w =
Fitting the coefficients of the hybrid model to behavioral data allows us to inspect how much a givenagent relied on a particular exploration strategy.
Previously, Gershman [2018] has applied this formof analysis to human data, which revealed that people rely on a mixture of directed and randomexploration.
In this article, we extend this approach to artificial data generated by RR-RL 2 .
[Figure]
Results: We trained RR-RL 2 with a targeted description length of { 1 , 2 , . .
. , 10000 } nats on the samedistribution used in the original experimental study and examined how the description length of thesemodels influences their exploration behavior using the previously described probit regression analysis.
Figure 1 (a) illustrates the results of this analysis.
We find that RR-RL 2 implements a Boltzmann-likeexploration strategy for description lengths between 1 and 100 nats.
Note that behavior at this stageis quite noisy as indicated by the small probit regression weights (average regression coefficient of0 . 15 ).
Beginning from 100 nats, we observe a rise of the factor corresponding to Thompson sampling,which continues to rise until the limit of 10000 nats.
Between 100 and 1000 nats, we additionallyfind minor influences of a Boltzmann-like exploration strategy (average regression coefficient of0 . 19 ).
For a description length of 1000 nats and larger, Boltzmann-like exploration diminishes and isreplaced with minor influences of a UCB-based strategy (average regression coefficient of 0 . 23 ).
Having established that different styles of exploration emerge in RR-RL 2 depending on its descriptionlength, we next set out to test how well it explains human choices.
In order to do so, we conducteda Bayesian model comparison [Bishop, 2006].
A detailed summary of our comparison procedureis provided in Appendix B. We used the Bayesian information criterion (BIC, Schwarz, 1978) asan approximation to the model evidence.
The resulting BIC values for each candidate model areillustrated in Figure 2 (a).
We find that the BIC value for RR-RL 2 is substantially lower comparedto that of the hybrid model ( 5562 .
63 against 6158 .
91 ) when aggregated across all participants; allother models provide a less adequate fit to human choices.
The majority of participants ( n = 32 )was best described by RR-RL 2 and the protected exceedance probability (PXP), which measuresthe probability that a particular model is the most frequent within a set of alternatives [Rigoux et al.
,2014], also favored RR-RL 2 decisively (PXP ≈ 1 ).
We provide a detailed illustration of the posteriorprobabilities for each model and participant in Figure 2 (b).
Finally, we compared the probit regression coefficients of human participants to the ones of RR-RL 2 .
Figure 1 (b) shows a two-dimensional UMAP embedding [McInnes et al., 2018] of these coefficients.
The figure reveals a set of diverse exploration strategies within the human population and confirmsthat RR-RL 2 captures the overall variability in human exploration appropriately.
Manipulating Computational Resources
RR-RL 2 also makes precise predictions about what should happen if computational resources areactively manipulated.
Do these predictions align with the actual behavior of people?
Providinganswers to this question is non-trivial because we cannot simply ask a person to use an algorithm
[Figure]
Humans
[Figure]
C h
o i c e P r o [Figure] a b ili t y low
[Figure]
high
[Figure]
KL=
{+KL=
+}
[Figure]
C h
o i
c e P r o b a b ili t y low-risk high-risk with a shorter or longer description length.
There are, however, two types of studies that can provideinsights.
The first type consists of lesion studies that compare the behavior of healthy participants tothat of participants suffering from brain damage, whereas the second type consists of developmentalstudies that investigate how behavior evolves during cognitive development.
We next take a look atan example of each of them and demonstrate that RR-RL 2 reproduces their key findings.
4.1 Damage to Ventromedial Prefrontal Cortex
There has been a long history of analyzing people with brain lesions in cognitive neuroscience[Damasio and Damasio, 1989].
We focused on a particular study conducted by Bechara et al.
[1994]for the purpose of this article and predicted that reducing the description length of RR-RL 2 shouldcorrespond to the behavior of brain-lesioned patients.
Experimental Design: To probe decision-making in clinical populations, Bechara et al.
[1994]introduced an experimental paradigm called the Iowa Gambling Task (IGT).
The IGT involves 100choices in a single four-armed bandit problem.
Two of the arms are high-risk arms, while the othertwo are low-risk arms.
High-risk arms result in a constant positive reward of 100 but have a chanceto yield a noisy penalty with an expected value of 125 .
Low-risk arms result in a constant positivereward of 50 but have a chance to yield a noisy penalty with an expected value of 25 .
A complete listof trials is printed in Table D1.
Bechara et al.
[1994] used the IGT to compare the decision-making ofhealthy participants to that of participants with ventromedial prefrontal cortex (vmPFC) damage.
Analysis: The focus of our analysis was the proportion of selected low- and high-risk arms across theentire experiment.
High-risk arms cause an average loss of 25 points per trial, while low-risk armsprovide an average payoff of 25 points per trial.
We should therefore expect an agent to select thesuperior low-risk arms with higher frequency.
Healthy participants are indeed able to learn about thestructure of the task and will after a while start to sample the superior low-risk arms.
Participants withvmPFC damage, however, continue to sample to the inferior high-risk as illustrated in Figure 3 (a).
This pattern is striking because performance in these subjects remains worse than chance regardlessof how often they interact with the task.
Results: RR-RL 2 requires a distribution over bandit problems for meta-learning, but participantsin the IGT only encountered a single bandit task.
Therefore, we cannot directly use the task of theoriginal study for meta-learning as we have done in the previous example.
We instead constructed adistribution over bandit problems that maintains the key characteristics of the IGT:
[Figure]
• The positive reward component was independently sampled for each arm from a uniform
distribution with a minimum value of 0 and a maximum value of 150 .
• The mean across all trials of the negative reward component was also sampled from a uniform distribution with a minimum value of 0 and a maximum value of 150 .
• The negative reward component had an occurrence probability sampled randomly from auniform distribution with a minimum value of 0 .
05 and a maximum value of 0 .
• We furthermore added additive noise sampled from a zero-mean normal distribution with astandard deviation of 10 to the negative reward component in each time-step.
We trained RR-RL 2 with a targeted description length of { 100 , 200 , . . .
, 10000 } nats on the previouslydescribed distribution.
When tested on the IGT, we find that RR-RL 2 replicates the pattern reported by
Bechara et al.
Models with a high description length successfully solve the task by selectinglow-risk arms in the majority of time-steps.
If description length is however sufficiently reduced,
RR-RL 2 predominately samples high-risk arms.
We illustrate this behavior for two example modelsin Figure 3 (b).
Figure D2 provides a more detailed picture of how description length mediates choicebehavior.
In summary, our analysis sheds light on why brain-lesioned patients display below-chance performance in the IGT.
Intuitively, any resource-limited agent must primarily devote its computationalresources to things that are easy to estimate.
In the IGT, the deterministic positive reward component is easier to estimate than the noisy negative component.
An agent with significantly restrictedresources will thus focus on the positive component while ignoring the negative.
In turn, the agentwill assign higher estimated payoffs to the inferior high-risk arms and therefore select them morefrequently.
We found that RR-RL 2 implements this behavior and that reducing its description lengthcaptured participants with lesioned vmPFC.
4.2 Developmental Trajectories
People are not born with fully-developed cognitive abilities but instead develop them during theirlifetime.
In this section, we tested whether increasing the description length of RR-RL 2 matchesthe behavioral trajectories of people as they grow up.
To test this hypothesis, we reanalyzed datacollected by Somerville et al.
[2017], who studied changes in exploration behavior between earlyadolescence and adulthood.
Experimental Design:
In their study, Somerville et al.
[2017] made use of an experimental paradigmknown as the horizon task [Wilson et al., 2014].
Each task was based on a two-armed bandit problem and involved four forced-choice trials, followed by either one or six free-choice trials.
Participantswere aware of the number of remaining choices and could use this information to guide their behavior.
The mean reward of one of the arms was drawn randomly from { 40 , 60 } , while the mean rewardfor the other was determined by sampling the difference to the first arm from { 4 , 8 , 12 , 20 , 30 } .
Thearrangement of arms as well as the sign of their difference was randomized.
In each time-step, theobserved reward was sampled from a normal distribution with the corresponding mean value anda standard deviation of 8 .
The addition of forced-choice trials allowed to control the amount ofinformation that was available to participants.
They either provided an equal amount of informationfor both arms (i.e., two observations each) or an unequal amount of information (i.e., a singleobservation from one arm, three from the other).
In total, Somerville et al.
[2017] collected data for147 participants between the ages of 12 . 08 and 28 , completing 160 bandit tasks each.
Analysis: Following Somerville et al.
[2017], we used the decision in the first free-choice trial todistinguish between different types of exploration.
In the unequal information condition, a choicewas classified as directed exploration if it corresponded to the option that was observed fewer timesduring the forced-choice trials.
In the equal information condition, a choice was classified as randomexploration if it corresponded to the option with the lower estimated mean.
We refer to an explorationbehavior as strategic if it occurs more frequently in the long horizon tasks compared to the shorthorizon tasks.
Somerville et al.
[2017] found – as shown in Figure 4 (a) – that strategic directedexploration emerges during adolescence, whereas strategic random exploration is age-invariant.
To quantify these effects, they fitted two independent linear regression models, using the probabilityof engaging in directed and random exploration as dependent variables.
Both models used age, thecorresponding horizon, and the interaction between the two as regressors.
They found a significanteffect of horizon in both conditions, indicating that participants engaged more in both directed andrandom exploration in tasks with a longer horizon.
Furthermore, they found a significant interactioneffect between horizon and age , confirmingthat strategic directed exploration increases during cognitive development, while strategic randomexploration remains constant over time.
Results: We trained RR-RL 2 with a targeted description length of { 100 , 200 , . .
. , 10000 } nats onthe same distribution used in the original experimental study.
Figure 4 (b) visualizes how strategicdirected and random exploration change as the description length of RR-RL 2 increases.
Matchingthe main result of the experimental study, we find that strategic directed exploration increases withdescription length, while strategic random exploration remains unaffected.
We repeated the previouslydescribed regression analysis on data simulated by RR-RL 2 to quantify this conclusion (replacingage as a regressor with description length).
The outcome of this analysis mirrored the results ofthe original study.
We found a significant effect of horizon on both directed ( F 1 , 194 = 56 .
50 , p <0 .
001 , η 2 = 0 . 20 ) and random exploration ( F 1 , 194 = 6 .
80 , p = 0 .
01 , η 2 = 0 .
This meansthat RR-RL 2 made more exploratory decisions of both types if it was beneficial to do so.
We alsofound a significant interaction effect between horizon and description length on directed exploration( F 1 , 194 = 17 .
48 , p < 0 .
001 , η 2 = 0 .
06 ) but not on random exploration ( F 1 , 194 = 0 .
02 , p = 0 . 89 ).
These results confirm that description length and age have comparable qualitative effects on thedevelopment of strategic exploration.
However, when comparing the effect sizes of our analysis to those from the experimental study,we find that the interaction effect between description length and horizon on directed explorationonly amounts to around half of the effect between age and horizon.
We speculated that part of thisdifference comes from a mismatch between the distribution used to train our models and what kindof tasks people expect in the experiment.
People, for instance, might assume that task rewards arenoisier than they are, which would require more exploratory choices, and, in turn, lead to strongereffects.
We tested this hypothesis by retraining RR-RL 2 on the same distribution but with the standarddeviation of the reward noise increased by 50% .
While this modification increased the effect size ofthe interaction effect on directed exploration, it did not close the gap entirely, suggesting that there areadditional – currently undiscovered – factors that contribute to the development of strategic directedexploration during adolescence.
General Discussion
The exploration-exploitation dilemma is one of the core challenges in reinforcement learning.
Howdo humans arbitrate between exploration and exploitation, and which kind of exploration strategiesdo they engage in?
We have put forward the hypothesis that people tackle this problem in a resourcerational manner.
To test this hypothesis, we proposed a method for meta-learning reinforcementlearning algorithms with limited description length.
The resulting class of models – which we coined
RR-RL 2 – makes precise predictions about how people make decisions.
We have put these predictionsto a rigorous test by comparing our model to data from three psychological studies.
RR-RL 2 displayedkey elements of human decision-making in all three of them:
It captured human exploration in a two-armed bandit task on both a qualitative and quantitative level.
Reducing its description length aligned with decision-making in brain-lesioned patients.3.
Increasing its description length reflected changes in exploration behavior attributed cognitive development.
In summary, our results demonstrate that it is possible to meta-learn resource-rational reinforcementlearning algorithms and that human exploration is well-characterized by these very algorithms.
5.1 Limitations and Future Work
We have focused on comparing RR-RL 2 to human exploration in the simple multi-armed banditsetting.
In the real world, however, people face much more sophisticated challenges that call for aricher repertoire of exploration strategies [Schulz et al., 2019, Brändle et al., 2021].
This criticism isnot necessarily a shortcoming of the proposed model, which could in principle be applied to morecomplex tasks, but rather one regarding the experimental research in cognitive psychology, which haspredominately focused on multi-armed bandit problems.
In future work, we intend to develop newexperimental paradigms that allow us to compare RR-RL 2 against human behavior in more complexsettings.
Potential paradigms may include contextual bandits, grid world problems, and multi-agentscenarios.
RR-RL 2 also places a constraint on a particular type of computational resource: the descriptionlength of the reinforcement learning algorithm in use.
People, on the other hand, are subject to avariety of additional computational constraints.
They can, for instance, only run algorithms with finitecomputation time or only store a restricted amount of chunks in their short-term memory [Collinset al., 2014].
Future work should aim to unify all of these constraints in a common framework.
5.2 Conclusion
Many applications could benefit from the availability of human-like agents.
Having access to suchagents may be especially valuable in cooperative self-play scenarios, where training with them iscrucial for successful coordination with people [Carroll et al., 2019, Strouse et al., 2021].
Thetraditional path towards constructing agents that learn and think like people is to take inspirationfrom the cognitive processes of the human mind and incorporate them into existing systems [Lakeet al., 2017].
In this article, we have pursued a different approach.
Instead of hard-coding cognitiveprocesses directly into our agents, we have identified two computational principles – meta-learningand resource rationality – that give rise to many aspects of human behavior.
The presented approachis very general, easy to adapt to new domains, and can be scaled to more complex problem settingswithout major modifications.
Finally, we want to emphasize that low description lengths might notonly be a biological necessity, but also a feature [Gigerenzer et al., 1999, Zador, 2019].
Implementingan algorithm in just a few bits acts as a strong form of regularization and could, in turn, produceexploration strategies that are applicable across domains.
Hence, we believe that constructing artificialsystems with such constraints could lead us towards more generally capable agents.
A Meta-Learning DetailsA.1 Meta-Learning Procedure
We employed a standard on-policy actor-critic procedure [Mnih et al., 2016, Wu et al., 2017] tooptimize a sample-based approximation of Equation 5 using the A DAM optimizer with a learning rateof 3 × 10 − 4 .
We simultaneously updated a dual parameter to satisfy the constraint on descriptionlength as suggested in prior work [Haarnoja et al., 2018, Eysenbach et al., 2021].
Models were trainedon one million batches of size 32 .
By the end of meta-learning, all models have converged.
Weadditionally scaled rewards to roughly fall within the range of [ − 1 , 1] to further stabilize training.
Pseudocode for the meta-learning procedure is provided in Algorithm 1.
Initialize Λ and β for n = 1 to N do ω ∼ p ( ω )
W ∼ q ( W | Λ ) a 0 , r 0 , h 0 = model.init() for t = 1 to H do h t , V t , π ( a t | h t , W ) =
model.forward ( a t − 1 , r t − 1 , h t − 1 ) a t ∼ π ( a t | h t , W ) r t ∼ p ( r t | a t , ω ) end for
L dual ← − β ( KL [ q ( W | Λ ) || p ( W )] − C )
L critic ←
L actor ←for t = 1 to H do
L critic ← L critic + ( r t + V t +1 − V t )
L actor ← L actor − ( r t + V t +1 − V t ) log π ( a t | h t , W ) end for Λ ← Λ − α ∇ Λ (cid:0) H − 1 ( L critic + L actor ) + β KL [ q ( W | Λ ) || p ( W )] (cid:1)
β ← β − α ∇ β L dual end for
A.2 Model Architecture
The model architecture consists of a single gated recurrent unit (GRU, Cho et al., 2014) layer with ahidden size of 128 .
Inputs to this GRU layer correspond to the action and reward from the previoustime-step.
Its outputs were then transformed by two linear layers, projecting to the policy and valueestimate respectively.
A.3 Prior and Encoding Distributions
The prior over network weights corresponds to a variational dropout prior [Kingma et al., 2015].
The encoding distribution is parametrized by a set of independent normal distributions over networkweights.
We adopted the approximation suggested by Molchanov et al.
[2017] to estimate the KLdivergence between the encoding and prior distribution and obtained gradients with respect to Λusing the reparametrization trick [Kingma and Welling, 2013].
B Bayesian Model Comparison
In the main text, we conducted a Bayesian model comparison to evaluate how well each model fittedthe behavioral data from the two-armed bandit task.
For this comparison, we computed the posteriorprobability that participant i with corresponding data D i used model m via Bayes’ theorem:
[Equation]
We assumed a uniform prior over models and approximated the model evidence with the Bayesianinformation criterion:
[Equation]
where N is the total number of tasks, H is the number of trials within each task, and | θ | is the numberof fitted parameters.
We use a i,nt and h i,nt to denote the action chosen and the history observed byparticipant i in task n and trial t .
The policy of our baseline models is directly given by Equation 6.
For RL 2 and RR-RL 2 , we additionally assumed an ϵ -greedy error model:
[Equation]
Furthermore, we approximated the integral over neural network weights with a Monte Carlo approximation of S = 10 samples:
[Equation]
In addition to the two meta-learning models, we considered several baseline algorithms within ourmodel comparison procedure.
These include the hybrid model from Equation 6, Thompson sampling,an upper confidence bound algorithm, and a Boltzmann-like exploration strategy.
Note that ourbaseline algorithms can be obtained by restricting a sub-set of parameters in the hybrid model to bezero.
Table B1 specifies fitted parameters θ for each model and their corresponding search domains.
We applied a simple grid search procedure over the values given in Table B1 to obtain the maximumlikelihood estimate of parameters for RL 2 and RR-RL 2 .
Parameters of the baseline models wereoptimized using a Newton-Raphson algorithm.
[Table]
C Two-Armed Bandit Task
Models were trained as described in Appendix A. Figure C1 (a) confirms that performance improvesas description length is increased.
Figure C1 (b) verifies that our models achieved their targeteddescription length.
[Figure]
C.1 Model Comparison based on AIC
We complemented our Bayesian model comparison from the main text with a model comparison thatrelies on the Akaike information criterion (AIC, Akaike, 1974):
[Equation]
This measure offers a simple approximation to the out-of-sample predictive accuracy of a model[Gelman et al., 1995].
Figure C2 visualizes the AIC values for each model on the aggregated data ofall participants.
The results of this analysis are fully consistent with the results based on BIC reportedin the main text.
[Figure]
D Iowa Gambling Task
[Figure]
Models were trained as described in Appendix A.
In addition, we assumed a discount factor ofγ = 0 .
9 for this set of experiments.
A geometric discount factor can be interpreted as a modificationto the task dynamics such that an agent believes to reach a terminal state with probability 1 − γ
Levine [2018].
We used this interpretation of the discount factor to model that participants in the IGTare not informed about the duration of the task.
Figure D1 (a) confirms that performance improves as description length is increased.
Figure D1 (b)verifies that our models achieved their targeted description length.
Table D1 contains a trial-wiseoverview of the IGT.
Figure D2 depicts the probability of selecting low- and high-risks arms for alldescription lengths.
[Figure] [Figure]
+ + + + + + +100       −
+
+ +
+ +
+ +
+ +
+50 − − − −− 50+ + + + + + + + + +50        −
[Figure]
c e P r o b a b ili t y
-risk
-risk
E Horizon Task
Models were trained as described in Appendix A.
In addition, models received a binary valueencoding the horizon of the current task.
Like the humans in the experimental study, they coulduse this information to guide their exploration behavior.
We unrolled the network during the forcedchoice trials by providing it with the externally specified actions and rewards.
We did not use theforced-choice trials to update the parameters of the network.
Figure E1 (a) confirms that performance improves as description length is increased.
Figure E1 (b)verifies that our models achieved their targeted description length.
[Figure]