Modeling Human Exploration Through Resource-Rational Reinforcement Learning
Abstract
Equipping artificial agents with useful exploration mechanisms remains a challenge to this day.
Humans, on the other hand, seem to manage the trade-off between exploration and exploitation effortlessly.
In the present article, we put forward the hypothesis that they accomplish this by making optimal use of limited computational resources.
We study this hypothesis by meta-learning reinforcement learning algorithms that sacrifice performance for a shorter description length (defined as the number of bits required to implement the given algorithm).
The emerging class of models captures human exploration behavior better than previously considered approaches, such as Boltzmann exploration, upper confidence bound algorithms, and
Thompson sampling.
We additionally demonstrate that changing the description length in our class of models produces the intended effects: reducing description length captures the behavior of brain-lesioned patients while increasing it mirrors cognitive development during adolescence.
Introduction
Knowing how to efficiently balance between exploring unfamiliar parts of an environment and exploiting currently available knowledge is an essential ingredient of any intelligent organism.
In theory, it is possible to obtain a Bayes-optimal solution to this exploration-exploitation dilemma by solving an augmented problem known as a Bayes-adaptive Markov decision process (BAMDP,
Duff, 2003).
However, BAMDPs are intractable to solve in general and analytical solutions are only available for a few special cases [Gittins, 1979].
The intractability of this optimal solution prompted researchers to develop a body of heuristic strategies [Auer et al., 2002, Kaufmann et al., 2012, Russo et al., 2017, Russo and Van Roy, 2014].
Most of these heuristics can be divided into two broad categories: directed and random exploration strategies [Wilson et al., 2014, Schulz and Gershman, 2019].
Directed exploration strategies provide bonus rewards that encourage the agent to visit parts of the environment that ought to be explored, whereas random exploration strategies inject some form of stochasticity into the policy.
Having access to a vast amount of existing exploration strategies leads to the question: which of them should we use when building artificial agents?
To answer this question, we may take inspiration from how people approach the exploration-exploitation dilemma.
Human exploration has been studied extensively in the multi-armed bandit setting [Mehlhorn et al., 2015, Wilson et al., 2021, Brändle et al., 2021].
Prior work indicates that people substantially deviate from the Bayes-optimal strategy even for the simplest bandit problems [Steyvers et al., 2009, Zhang and Angela, 2013, Binz and Endres, 2019].
They, however, use uncertainty estimates to intelligently guide their choices through a combination of both directed and random exploration [Wilson et al., 2014, Gershman, 2018].
The question of when and why individuals rely on a particular exploration strategy has been under-explored so far.
We take the first steps towards answering these questions by looking at human exploration from a resource-rational perspective [Gershman et al., 2015, Lieder and Griffiths, 2020, Binz et al., 2022].
More specifically, we investigate the hypothesis that people solve the exploration-exploitation dilemma by making optimal use of limited computational resources.
To test this hypothesis, we devise a family of resource-rational reinforcement learning algorithms by combining ideas from meta-learning [Bengio et al., 1991, Schmidhuber et al., 1996] and information theory [Hinton and Van Camp, 1993].
The resulting model – which we call resource-rational RL 2 (RR-RL 2 ) – implements a free-standing reinforcement learning algorithm that achieves optimal behavior subject to the constraint that it can be implemented with a given number of bits.
We demonstrate that RR-RL 2 captures many aspects of human exploration by reanalyzing data from three previously conducted psychological studies.
First, we show that it explains human choices in a two-armed bandit task better than traditional approaches, such as Thompson sampling [Thompson, 1933], upper confidence bound (UCB) algorithms [Kaufmann et al., 2012], and mixtures thereof [Gershman, 2018].
We then verify that the manipulation of computational resources in our class of models matches the manipulation of resources in human subjects in two different contexts.
Taken together, these results enrich our understanding of human exploration and provide insights into how to improve the exploratory capabilities of artificial agents.
Methods
We start by describing the general problem setting considered in this article and its optimal solution, followed by a brief summary of the meta-reinforcement learning framework.
We then show how to augment the standard meta-reinforcement learning objective with an information-theoretic constraint, allowing us to construct reinforcement learning algorithms that trade-off performance against the number of bits required to implement them.
2.1 Notation and Preliminaries
Each task considered in this article can be interpreted as a multi-armed bandit problem.
In a k -armed bandit problem, an agent repeatedly interacts with k slot machines that are associated with a reward distribution p ( r t | a t , ω ) with unknown parameters ω .
In each time-step, the agent selects an action a t and is provided a reward r t based on the associated reward distribution.
The goal of an agent is to select actions such that the sum of rewards over a finite horizon H is maximized.
We assume that the agent additionally has access to a prior distribution p ( ω ) over bandit problems that it may encounter, which can be updated after observing a history of observations h t :=
( a 1 , r 1 , . .
. , a t − 1 , r t − 1 ) by applying Bayes’ rule:
[Equation]
The policy that optimally trades-off exploration and exploitation can be obtained by reasoning how the agent’s beliefs about reward functions evolve with future observations [Martin, 1967].
Formally, this is accomplished by transforming the original bandit problem into a corresponding BAMDP defined by the tuple ( H , A , H, T, R ) .
In this augmented problem, H represents the set of all possible histories, while A and H correspond to the action space and the horizon of the original bandit problem.
The transition probabilities T and reward function R are given by:
[Equation]
with the marginal reward probabilities:
[Equation]
The policy that maximizes the sum of rewards in the BAMDP corresponds to the Bayes-optimal policy for the original bandit problem.
2.2 Meta-Reinforcement Learning
While the BAMDP formalism provides a precise recipe for deriving a Bayes-optimal policy, finding an analytical expression of this policy is typically not possible.
Recent work on meta-reinforcement learning, however, has shown that it is possible to learn an approximation to it
[Wang et al., 2016,
Ortega et al., 2019, Zintgraf et al., 2019].
Duan et al.
[2016] refer to this approach as RL 2 because it uses a traditional reinforcement learning algorithm to meta-learn another free-standing reinforcement learning algorithm.
RL 2 parametrizes the to-be-learned reinforcement learning algorithm with a general-purpose function approximator.
Typically, this function approximator takes the form of a recurrent neural network that receives the last action and reward as inputs, uses them to update its hidden state, and produces a policy that is conditioned on the new hidden state.
Let W denote the parameters of this recurrent neural network.
In an outer-loop meta-learning process, the network is then trained on the prior distribution over bandit problems p ( ω ) to find the history-dependent policy π ( a t | h t , W ) that maximizes the sum of obtained rewards.
If the meta-learning procedure has successfully converged to its optimum, RLimplements a free-standing reinforcement learning algorithm that mimics the Bayes-optimal policy.
Importantly, learning at this stage is fully implemented through the forward dynamics of the recurrent neural network and no further updating of its parameters is required.
2.3 Resource-Rational RL
We transform RL 2 into a resource-rational algorithm by augmenting its meta-learning objective with an information-theoretic constraint and refer to this resource-rational variant as RR-RL 2 .
More precisely, RR-RL 2 is obtained by solving the following optimization problem:
[Equation]
The first component of Equation 5 corresponds to the standard meta-reinforcement learning objective, while the second component ensures that the Kullback–Leibler (KL) divergence between a stochastic parameter encoding q ( W | Λ ) and a prior p ( W ) remains smaller than some constant C .
The KL term can be interpreted as the description length of neural network parameters, i.e., the number of bits required to store them.
1 RR-RL 2 therefore optimally trades-off performance against the number of bits required to implement the emerging reinforcement learning algorithm.
Note that the objective from Equation 5 can also be motivated by a PAC-Bayes bound on generalization performance to unseen tasks [Yin et al., 2019, Rothfuss et al., 2020, Jose and Simeone, 2020].
While we focus on the resource-rational interpretation in the present article, we believe that both of these perspectives are complementary.
In practice, we solve a sample-based approximation of the optimization problem in Equation [-
 usinga-]5 using a standard on-policy actor-critic algorithm
[Mnih et al., 2016, Wu et al., 2017].
We rely on a dual gradient ascent procedure [Haarnoja et al., 2018] to ensure that the constraint is satisfied.
Appendix A contains a complete description of the network architecture, choices of prior and encoding distribution, and the meta-learning procedure.
A public implementation of our model and the following analyses is available under https://github.com/marcelbinz/ resource-rational-reinforcement-learning .
Which exploration strategies RR-RL 2 implements will partially depend on its available computational resources.
Models with limited resources must implement strategies that rely on simple computations, such as computing average rewards or noisy estimates thereof.
Models with access to more resources, on the other hand, can spend some of them to compute more complex statistics, such as uncertainty estimates, and incorporate these into their decision-making process.
Moreover, what strategies are resource-rational not only depends on the computational resources of the decision-maker but also on the characteristics of the particular problem under consideration.
Problems with a short task horizon, for instance, require less exploration and thereby make exploitation more appealing, while problems with a longer horizon allow for the application of more sophisticated exploration strategies. [Figure] Exploration Strategies [Figure]
UCB
Thompson
Dimension
D i m e n s i o n
Humans
RR-RL
Modeling Human Exploration
We now demonstrate that RR-RL 2 explains human choices on both a qualitative and quantitative level.
We first show that varying its description length leads to a set of diverse exploration strategies, allowing us to capture individual differences in human decision-making.
When reanalyzing data from a two-armed bandit benchmark [Gershman, 2018], we furthermore find that RR-RL 2 beats previously considered algorithms in terms of fitting human behavior by a large margin.
Experimental Design: The behavioral data-set of Gershman [2018] contains records of 44 participants who each played 20 two-armed bandit problems with an episode length of H = 10 .
The mean reward for each arm a was drawn from p ( ω a ) =
N (0 , 10) at the beginning of the task and the reward in each time-step from p ( r t | a t , ω ) =
N ( ω a t , 1) .
Analysis: To analyze the set of emerging exploration strategies, we adopted a method proposed by
Gershman [2018].
He assumed that an agent uses Bayes’ rule as described in Equation 1 to update its beliefs over unobserved parameters.
If prior and reward are both normally distributed, the posterior will also be normally distributed and the corresponding updating rule is given by the Kalman filtering equations.
Let p ( ω a | h t ) =
N ( µ a,t , σ a,t ) be the posterior distribution at time-step t .
Based on the parameters of this posterior distribution, he then defined the following probit regression model:
[Equation]
with Φ denoting the cumulative distribution function of a standard normal distribution.
Equationis also referred to as the hybrid model as it contains several known exploration strategies as special cases.
We can recover a Boltzmann-like exploration strategy for w =
[ w 1 , 0 , 0] , a variant of the UCB algorithm for w =
[ w 1 , w 2 , 0] , and Thompson sampling for w =
Fitting the coefficients of the hybrid model to behavioral data allows us to inspect how much a given agent relied on a particular exploration strategy.
Previously, Gershman [2018] has applied this form of analysis to human data, which revealed that people rely on a mixture of directed and random exploration.
In this article, we extend this approach to artificial data generated by RR-RL 2 .
[Figure]
Results: We trained RR-RL 2 with a targeted description length of { 1 , 2 , . .
. , 10000 } nats on the same distribution used in the original experimental study and examined how the description length of these models influences their exploration behavior using the previously described probit regression analysis.
Figure 1 (a) illustrates the results of this analysis.
We find that RR-RL 2 implements a Boltzmann-like exploration strategy for description lengths between 1 and 100 nats.
Note that behavior at this stage is quite noisy as indicated by the small probit regression weights (average regression coefficient of 0 . 15 ).
Beginning from 100 nats, we observe a rise of the factor corresponding to Thompson sampling, which continues to rise until the limit of 10000 nats.
Between 100 and 1000 nats, we additionally find minor influences of a Boltzmann-like exploration strategy (average regression coefficient of 0 . 19 ).
For a description length of 1000 nats and larger, Boltzmann-like exploration diminishes and is replaced with minor influences of a UCB-based strategy (average regression coefficient of 0 . 23 ).
Having established that different styles of exploration emerge in RR-RL 2 depending on its description length, we next set out to test how well it explains human choices.
In order to do so, we conducted a Bayesian model comparison [Bishop, 2006].
A detailed summary of our comparison procedure is provided in Appendix B. We used the Bayesian information criterion (BIC, Schwarz, 1978) as an approximation to the model evidence.
The resulting BIC values for each candidate model are illustrated in Figure 2 (a).
We find that the BIC value for RR-RL 2 is substantially lower compared to that of the hybrid model ( 5562 .
63 against 6158 .
91 ) when aggregated across all participants; all other models provide a less adequate fit to human choices.
The majority of participants ( n = 32 ) was best described by RR-RL 2 and the protected exceedance probability (PXP), which measures the probability that a particular model is the most frequent within a set of alternatives [Rigoux et al., 2014], also favored RR-RL 2 decisively (PXP ≈ 1 ).
We provide a detailed illustration of the posterior probabilities for each model and participant in Figure 2 (b).
Finally, we compared the probit regression coefficients of human participants to the ones of RR-RL 2 .
Figure 1 (b) shows a two-dimensional UMAP embedding [McInnes et al., 2018] of these coefficients.
The figure reveals a set of diverse exploration strategies within the human population and confirms that RR-RL 2 captures the overall variability in human exploration appropriately.
Manipulating Computational Resources
RR-RL 2 also makes precise predictions about what should happen if computational resources are actively manipulated.
Do these predictions align with the actual behavior of people?
Providing answers to this question is non-trivial because we cannot simply ask a person to use an algorithm with a shorter or longer description length.
There are, however, two types of studies that can provide insights.
[-high [Figure] KL= KL=
-]The first type consists of lesion studies that compare the behavior of healthy participants to that of participants suffering from brain damage, whereas the second type consists of developmental
[Figure]
studies that investigate how behavior evolves during cognitive development.
We next take a look at an example of each of them and demonstrate that RR-RL 2 reproduces their key findings.
4.1 Damage to Ventromedial Prefrontal Cortex
There has been a long history of analyzing people with brain lesions in cognitive neuroscience [Damasio and Damasio, 1989].
We focused on a particular study conducted by Bechara et al.
[1994] for the purpose of this article and predicted that reducing the description length of RR-RL 2 should correspond to the behavior of brain-lesioned patients.
Experimental Design: To probe decision-making in clinical populations, Bechara et al.
[1994] introduced an experimental paradigm called the Iowa Gambling Task (IGT).
The IGT involveschoices in a single four-armed bandit problem.
Two of the arms are high-risk arms, while the other two are low-risk arms.
High-risk arms result in a constant positive reward of 100 but have a chance to yield a noisy penalty with an expected value of 125 .
Low-risk arms result in a constant positive reward of 50 but have a chance to yield a noisy penalty with an expected value of 25 .
A complete list of trials is printed in Table D1.
Bechara et al.
[1994] used the IGT to compare the decision-making of healthy participants to that of participants with ventromedial prefrontal cortex (vmPFC) damage.
Analysis: The focus of our analysis was the proportion of selected low- and high-risk arms across the entire experiment.
High-risk arms cause an average loss of 25 points per trial, while low-risk arms provide an average payoff of 25 points per trial.
We should therefore expect an agent to select the superior low-risk arms with higher frequency.
Healthy participants are indeed able to learn about the structure of the task and will after a while start to sample the superior low-risk arms.
Participants with vmPFC damage, however, continue to sample to the inferior high-risk as illustrated in Figure 3 (a).
This pattern is striking because performance in these subjects remains worse than chance regardless of how often they interact with the task.
Results: RR-RL 2 requires a distribution over bandit problems for meta-learning, but participants in the IGT only encountered a single bandit task.
Therefore, we cannot directly use the task of the original study for meta-learning as we have done in the previous example.
We instead constructed a distribution over bandit problems that maintains the key characteristics of the IGT:
• The positive reward component was independently sampled for each arm from a uniform distribution with a minimum value of 0 and a maximum value of 150 .
• The mean across all trials of the negative reward component was also sampled from a uniform distribution with a minimum value of 0 and a maximum value of 150 . [Figure] [Figure]
p l o r [Figure] [Figure] p ( e x p l o r e )
[Figure] • The negative reward component had an occurrence probability sampled randomly from a uniform distribution with a minimum value of 0 .
05 and a maximum value of 0 .
• We furthermore added additive noise sampled from a zero-mean normal distribution with a standard deviation of 10 to the negative reward component in each time-step.
We trained RR-RL 2 with a targeted description length of { 100 , 200 , . . .
, 10000 } nats on the previously described distribution.
When tested on the IGT, we find that RR-RL 2 replicates the pattern reported by
Bechara et al.
Models with a high description length successfully solve the task by selecting low-risk arms in the majority of time-steps.
If description length is however sufficiently reduced,
RR-RL 2 predominately samples high-risk arms.
We illustrate this behavior for two example models in Figure 3 (b).
Figure D2 provides a more detailed picture of how description length mediates choice behavior.
In summary, our analysis sheds light on why brain-lesioned patients display below-chance performance in the IGT.
Intuitively, any resource-limited agent must primarily devote its computational resources to things that are easy to estimate.
In the IGT, the deterministic positive reward component is easier to estimate than the noisy negative component.
An agent with significantly restricted resources will thus focus on the positive component while ignoring the negative.
In turn, the agent will assign higher estimated payoffs to the inferior high-risk arms and therefore select them more frequently.
We found that RR-RL 2 implements this behavior and that reducing its description length captured participants with lesioned vmPFC.
4.2 Developmental Trajectories
People are not born with fully-developed cognitive abilities but instead develop them during their lifetime.
In this section, we tested whether increasing the description length of RR-RL 2 matches the behavioral trajectories of people as they grow up.
To test this hypothesis, we reanalyzed data collected by Somerville et al.
[2017], who studied changes in exploration behavior between early adolescence and adulthood.
Experimental Design:
In their study, Somerville et al.
[2017] made use of an experimental paradigm known as the horizon task [Wilson et al., 2014].
Each task was based on a two-armed bandit problem and involved four forced-choice trials, followed by either one or six free-choice trials.
Participants were aware of the number of remaining choices and could use this information to guide their behavior.
The mean reward of one of the arms was drawn randomly from { 40 , 60 } , while the mean reward for the other was determined by sampling the difference to the first arm from { 4 , 8 , 12 , 20 , 30 } .
The arrangement of arms as well as the sign of their difference was randomized.
In each time-step, the observed reward was sampled from a normal distribution with the corresponding mean value and a standard deviation of 8 .
The addition of forced-choice trials allowed to control the amount of information that was available to participants.
They either provided an equal amount of information for both arms (i.e., two observations each) or an unequal amount of information (i.e., a single observation from one arm, three from the other).
In total, Somerville et al.
[2017] collected data for 147 participants between the ages of 12 . 08 and 28 , completing 160 bandit tasks each.
Analysis: Following Somerville et al.
[2017], we used the decision in the first free-choice trial to distinguish between different types of exploration.
In the unequal information condition, a choice was classified as directed exploration if it corresponded to the option that was observed fewer times during the forced-choice trials.
In the equal information condition, a choice was classified as random exploration if it corresponded to the option with the lower estimated mean.
We refer to an exploration behavior as strategic if it occurs more frequently in the long horizon tasks compared to the short horizon tasks.
Somerville et al.
[2017] found – as shown in Figure 4 (a) – that strategic directed exploration emerges during adolescence, whereas strategic random exploration is age-invariant.
To quantify these effects, they fitted two independent linear regression models, using the probability of engaging in directed and random exploration as dependent variables.
Both models used age, the corresponding horizon, and the interaction between the two as regressors.
They found a significant effect of horizon in both conditions, indicating that participants engaged more in both directed and random exploration in tasks with a longer horizon.
Furthermore, they found a significant interaction effect between horizon and age for directed exploration but not for random exploration, confirming that strategic directed exploration increases during cognitive development, while strategic random exploration remains constant over time.
Results: We trained RR-RL 2 with a targeted description length of { 100 , 200 , . .
. , 10000 } nats on the same distribution used in the original experimental study.
Figure 4 (b) visualizes how strategic directed and random exploration change as the description length of RR-RL 2 increases.
Matching the main result of the experimental study, we find that strategic directed exploration increases with description length, while strategic random exploration remains unaffected.
We repeated the previously described regression analysis on data simulated by RR-RL 2 to quantify this conclusion (replacing age as a regressor with description length).
The outcome of this analysis mirrored the results of the original study.
We found a significant effect of horizon on both directed ( F 1 , 194 = 56 .
50 , p < 0 .
001 , η 2 = 0 . 20 ) and random exploration ( F 1 , 194 = 6 .
80 , p = 0 .
01 , η 2 = 0 .
This means that RR-RL 2 made more exploratory decisions of both types if it was beneficial to do so.
We also found a significant interaction effect between horizon and description length on directed exploration ( F 1 , 194 = 17 .
48 , p < 0 .
001 , η 2 = 0 .
06 ) but not on random exploration ( F 1 , 194 = 0 .
02 , p = 0 . 89 ).
These results confirm that description length and age have comparable qualitative effects on the development of strategic exploration.
However, when comparing the effect sizes of our analysis to those from the experimental study, we find that the interaction effect between description length and horizon on directed exploration only amounts to around half of the effect between age and horizon.
We speculated that part of this difference comes from a mismatch between the distribution used to train our models and what kind of tasks people expect in the experiment.
People, for instance, might assume that task rewards are noisier than they are, which would require more exploratory choices, and, in turn, lead to stronger effects.
We tested this hypothesis by retraining RR-RL 2 on the same distribution but with the standard deviation of the reward noise increased by 50% .
While this modification increased the effect size of the interaction effect on directed exploration, it did not close the gap entirely, suggesting that there are additional – currently undiscovered – factors that contribute to the development of strategic directed exploration during adolescence.
Furthermore, we found that, while strategic random exploration was unaffected by both age and description length, humans had a higher base rate of engaging in random exploration than our models.
We, however, believe that this issue could be resolved by adding an ε -greedy error model to our models as doing so would naturally increase the base rate of random exploration.
General Discussion
The exploration-exploitation dilemma is one of the core challenges in reinforcement learning.
How do humans arbitrate between exploration and exploitation, and which kind of exploration strategies do they engage in?
We have put forward the hypothesis that people tackle this problem in a resourcerational manner.
To test this hypothesis, we proposed a method for meta-learning reinforcement learning algorithms with limited description length.
The resulting class of models – which we coined
RR-RL 2 – makes precise predictions about how people make decisions.
We have put these predictions to a rigorous test by comparing our model to data from three psychological studies.
RR-RL 2 displayed key elements of human decision-making in all three of them:
It captured human exploration in a two-armed bandit task on both a qualitative and quantitative level.
Reducing its description length aligned with decision-making in brain-lesioned patients.
Increasing its description length reflected changes in exploration behavior attributed cognitive development.
In summary, our results demonstrate that it is possible to meta-learn resource-rational reinforcement learning algorithms and that human exploration is well-characterized by these very algorithms.
5.1 Limitations and Future Work
We have focused on comparing RR-RL 2 to human exploration in the simple multi-armed bandit setting.
In the real world, however, people face much more sophisticated challenges that call for a richer repertoire of exploration strategies [Schulz et al., 2019, Brändle et al., 2021].
This criticism is not necessarily a shortcoming of the proposed model, which could in principle be applied to more complex tasks, but rather one regarding the experimental research in cognitive psychology, which has predominately focused on multi-armed bandit problems.
In future work, we intend to develop new experimental paradigms that allow us to compare RR-RL 2 against human behavior in more complex settings.
Potential paradigms may include contextual bandits, grid world problems, and multi-agent scenarios.
RR-RL 2 also places a constraint on a particular type of computational resource: the description length of the reinforcement learning algorithm in use.
People, on the other hand, are subject to a variety of additional computational constraints.
They can, for instance, only run algorithms with finite computation time or only store a restricted amount of chunks in their short-term memory [Collins et al., 2014].
Future work should aim to unify all of these constraints in a common framework.
It would also be interesting to investigate how RR-RL 2 could be implemented in a biologically plausible way.
We believe that replacing the standard recurrent neural networks used in this article with spiking neural networks could provide one promising path towards this goal.
Earlier work of
Bellec et al. [2018] showed that it is indeed possible to train spiking neural networks in a meta-learning setting and would therefore provide an excellent starting point for this endeavor. Finally, there are many alternative ways of how one could apply meta-learning to our setting. It is, for example, possible to meta-learn hyperparameters of standard reinforcement learning algorithms [Doya, 2002, Luksys et al., 2009], or to use model-agnostic meta-learning to find optimal weight initializations for neural networks that are trained by gradient descent [Finn et al., 2017].
Future work should compare these and other models against each other.
For this to be effective, we believe that a set of commonly-accepted benchmarks needs to be established.
5.2 Conclusion
Many applications could benefit from the availability of human-like agents.
Having access to such agents may be especially valuable in cooperative self-play scenarios, where training with them is crucial for successful coordination with people [Carroll et al., 2019, Strouse et al., 2021].
The traditional path towards constructing agents that learn and think like people is to take inspiration from the cognitive processes of the human mind and incorporate them into existing systems [Lake et al., 2017].
In this article, we have pursued a different approach.
Instead of hard-coding cognitive processes directly into our agents, we have identified two computational principles – meta-learning and resource rationality – that give rise to many aspects of human behavior.
The presented approach is very general, easy to adapt to new domains, and can be scaled to more complex problem settings without major modifications.
Finally, we want to emphasize that low description lengths might not only be a biological necessity, but also a feature [Gigerenzer et al., 1999, Zador, 2019].
Implementing an algorithm in just a few bits acts as a strong form of regularization and could, in turn, produce exploration strategies that are applicable across domains.
Hence, we believe that constructing artificial systems with such constraints could lead us towards more generally capable agents.
Acknowledgments and Disclosure of Funding
This work was funded by the Max Planck Society, the Volkswagen Foundation, as well as the
Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy–EXC2064/1–390727645.
A Meta-Learning Details A.1 Meta-Learning Procedure
We employed a standard on-policy actor-critic procedure [Mnih et al., 2016, Wu et al., 2017] to optimize a sample-based approximation of Equation 5 using the A DAM optimizer with a learning rate of 3 × 10 − 4 .
We simultaneously updated a dual parameter to satisfy the constraint on description length as suggested in prior work [Haarnoja et al., 2018, Eysenbach et al., 2021].
Models were trained on one million batches of size 32 .
By the end of meta-learning, all models have converged.
We additionally scaled rewards to roughly fall within the range of [ − 1 , 1] to further stabilize training.
Pseudocode for the meta-learning procedure is provided in Algorithm 1.
Initialize Λ and β for n = 1 to N do ω ∼ p ( ω )
W ∼ q ( W | Λ ) a 0 , r 0 , h 0 = model.init() for t = 1 to H do h t , V t , π ( a t | h t , W ) =
model.forward ( a t − 1 , r t − 1 , h t − 1 ) a t ∼ π ( a t | h t , W ) r t ∼ p ( r t | a t , ω ) end for
L dual ← − β ( KL [ q ( W | Λ ) || p ( W )] − C )
L critic ←
L actor ←for t = 1 to H do
L critic ← L critic + ( r t + V t +1 − V t )
L actor ← L actor − ( r t + V t +1 − V t ) log π ( a t | h t , W ) end for Λ ← Λ − α ∇ Λ (cid:0) H − 1 ( L critic + L actor ) + β KL [ q ( W | Λ ) || p ( W )] (cid:1)
β ← β − α ∇ β L dual end for
A.2 Model Architecture
The model architecture consists of a single gated recurrent unit (GRU, Cho et al., 2014) layer with a hidden size of 128 .
Inputs to this GRU layer correspond to the action and reward from the previous time-step.
Its outputs were then transformed by two linear layers, projecting to the policy and value estimate respectively.
A.3 Prior and Encoding Distributions
The prior over network weights corresponds to a variational dropout prior [Kingma et al., 2015].
The encoding distribution is parametrized by a set of independent normal distributions over network weights.
We adopted the approximation suggested by Molchanov et al.
[2017] to estimate the KL divergence between the encoding and prior distribution and obtained gradients with respect to Λ using the reparametrization trick [Kingma and Welling, 2013].
B Bayesian Model Comparison
In the main text, we conducted a Bayesian model comparison to evaluate how well each model fitted the behavioral data from the two-armed bandit task.
For this comparison, we computed the posterior probability that participant i with corresponding data D i used model m via Bayes’ theorem:
[Equation]
We assumed a uniform prior over models and approximated the model evidence with the Bayesian information criterion:
[Equation]
where N is the total number of tasks, H is the number of trials within each task, and | θ | is the number of fitted parameters.
We use a i,nt and h i,nt to denote the action chosen and the history observed by participant i in task n and trial t .
The policy of our baseline models is directly given by Equation 6.
For RL 2 and RR-RL 2 , we additionally assumed an ε -greedy error model:
[Equation]
Furthermore, we approximated the integral over neural network weights with a Monte Carlo approximation of S = 10 samples:
[Equation]
In addition to the two meta-learning models, we considered several baseline algorithms within our model comparison procedure.
These include the hybrid model from Equation 6, Thompson sampling, an upper confidence bound algorithm, and a Boltzmann-like exploration strategy.
Note that our baseline algorithms can be obtained by restricting a sub-set of parameters in the hybrid model to be zero.
Table B1 specifies fitted parameters θ for each model and their corresponding search domains.
We applied a simple grid search procedure over the values given in Table B1 to obtain the maximum likelihood estimate of parameters for RL 2 and RR-RL 2 .
Parameters of the baseline models were optimized using a Newton-Raphson algorithm.
[Table]
C Two-Armed Bandit Task
Models were trained as described in Appendix A. Figure C1 (a) confirms that performance improves as description length is increased.
Figure C1 (b) verifies that our models achieved their targeted description length.
[Figure]
C.1 Model Comparison Based on AIC
We complemented our Bayesian model comparison from the main text with a model comparison that relies on the Akaike information criterion (AIC, Akaike, 1974):
[Equation]
This measure offers a simple approximation to the out-of-sample predictive accuracy of a model [Gelman et al., 1995].
Figure C2 visualizes the AIC values for each model on the aggregated data of all participants.
The results of this analysis are fully consistent with the results based on BIC reported in the main text.
[Figure]
D Iowa Gambling Task
Models were trained as described in Appendix A.
In addition, we assumed a discount factor of γ = 0 .
9 for this set of experiments.
A geometric discount factor can be interpreted as a modification to the task dynamics such that an agent believes to reach a terminal state with probability 1 − γ
Levine [2018].
We used this interpretation of the discount factor to model that participants in the IGT are not informed about the duration of the task.
Figure D1 (a) confirms that performance improves as description length is increased.
Figure D1 (b) verifies that our models achieved their targeted description length.
Table D1 contains a trial-wise overview of the IGT.
Figure D2 depicts the probability of selecting low- and high-risks arms for all description lengths.
[Figure] [Figure]
[Figure] + + [-+ + + + +100       −
-][Figure]
+
[Figure] + + [Figure]
+
[Figure]
+
[Figure]
+
[Figure]
+
[Figure]
+
[Figure]
+
[Figure]
+
[Figure]
+100       −0+ + + + + + + + + +50 − − − −− 50+ + + + + + + + + +50        −
[Figure]
C h o i c e P r o b a b ili t y
E Horizon Task
Models were trained as described in Appendix A.
In addition, models received a binary value encoding the horizon of the current task.
Like the humans in the experimental study, they could use this information to guide their exploration behavior.
We unrolled the network during the forcedchoice trials by providing it with the externally specified actions and rewards.
We did not use the forced-choice trials to update the parameters of the network.
Figure E1 (a) confirms that performance improves as description length is increased.
Figure E1 (b) verifies that our models achieved their targeted description length.
[Figure]