Action-modulated midbrain dopamine activity arises from distributed control policies
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Animal behavior is driven by multiple brain regions working in parallel with distinct control policies.
We present a biologically plausible model of off-policyreinforcement learning in the basal ganglia, which enables learning in such anarchitecture.
The model accounts for action-related modulation of dopamine activity that is not captured by previous models that implement on-policy algorithms.
In particular, the model predicts that dopamine activity signals a combination ofreward prediction error (as in classic models) and “action surprise," a measure ofhow unexpected an action is relative to the basal ganglia’s current policy.
In thepresence of the action surprise term, the model implements an approximate formof Q -learning.
On benchmark navigation and reaching tasks, we show empiricallythat this model is capable of learning from data driven completely or in part byother policies (e.g. from other brain regions).
By contrast, models without theaction surprise term suffer in the presence of additional policies, and are incapableof learning at all from behavior that is completely externally driven.
The model provides a computational account for numerous experimental ﬁndings about dopamineactivity that cannot be explained by classic models of reinforcement learning inthe basal ganglia.
These include differing levels of action surprise signals in dorsaland ventral striatum, decreasing amounts movement-modulated dopamine activitywith practice, and representations of action initiation and kinematics in dopamineactivity.
It also provides further predictions that can be tested with recordings ofstriatal dopamine activity.
Introduction
An extensive body of work has sought to account for the function of the basal ganglia using thecomputational framework of reinforcement learning (RL), in particular RL algorithms for actionselection and value learning [26, 37, 25, 49].
Striatal neurons within the basal ganglia integratediverse inputs, including projections from across the cerebral cortex.
The activity of these neurons, inparticular neurons in the dorsal striatum plays a key role in action selection [39].
On the other hand,neurons in the ventral striatum have been shown to encode the learned value of stimuli [8, 36, 38].
The phasic activity of midbrain dopamine neurons projecting to the striatum gates plasticity at corticostriatal synapses [53, 7, 9] and may therefore modulate the learning of actions and values.
Indeed,in many RL algorithms, the learning of state-to-action mappings (“policies”) and value estimates ismodulated by a scalar factor known as the “advantage” or “reward prediction error” (RPE), whichmeasures deviations in attained reward from expectations based on learned value estimates.
Numerousexperiments have shown that striatal dopamine activity encodes an RPE-like signal [42, 36, 22].
Collectively, these ﬁndings suggest a model in which the basal ganglia implements an online actorcritic RL algorithm, where ventral and dorsal striatal subregions play the roles of critic and actor, respectively, and dopaminergic activity encodes the advantage (RPE) signal [26].
This model hasbeen extended in a variety of ways to incorporate more biological detail [47, 48, 9, 6, 46], ideas frommodel-based RL [13], distributional RL [11], and meta-RL [51].
Despite these promising links, there remain challenges to the view of basal ganglia as implementingan actor-critic RL algorithm.
In this work, we address two such challenges.
First, dopamine activityin the basal ganglia is observed to encode other information beside RPEs [15], particularly signalsrelating to movement initiation and vigor [10, 56].
Second, the classic actor-critic model of the basalganglia is an on-policy RL algorithm—it is designed to learn online from experiences driven by thebasal ganglia’s policy.
However, motor control in biological systems is distributed, with multiplebrain regions including the motor cortex and cerebellum exerting inﬂuence on behavior independentof the basal ganglia [16, 54, 2, 44, 5].
We show that classic actor-critic models of the basal gangliafail in this scenario and argue that the ability to learn from off-policy experience is essential to modelsof basal ganglia learning.
We present a biologically plausible model of off-policy RL in continuous action spaces.
Our modeldiffers from the classic actor-critic model of the basal ganglia by adding an additional action-relatedterm to dopamine activity.
We call this term “action surprise," as it measures deviation of the actionan agent takes in a particular state from the typical action that the output of the basal ganglia wouldselect in that state.
We show mathematically that, with this addition, the algorithm implements anapproximate form of Q -learning in continuous action spaces proposed by [19].
Using simulations,we show that action surprise is essential to effective learning when controllers other than the basalganglia also contribute to behavior.
Thus, action-related activity need not be understood as anindependent function of midbrain dopamine neurons separate from their role in learning, but ratheras a necessary component of an algorithm that enables off-policy RL.
This action surprise modelaccounts for several experimental ﬁndings about movement-related dopamine activity in the basalganglia.
We also provide an interpretation of the presence of greater levels of movement-relatedactivity in dorsal-projecting dopamine neurons, showing that such an asymmetry can approximate asupervised learning signal that accelerates the consolidation of other controllers’ policies.
Finally, wedescribe predictions of our model regarding dopamine activity.
Background: Connection between policy gradient algorithms andcortico-striatal plasticity
Here we brieﬂy introduce the connection between on-policy policy gradient algorithms and synapticplasticity rules in the striatum.
In what follows, we assume familiarity with standard notationand concepts in RL; for a brief review, see Appendix A.1.
Policy gradient algorithms update theparameters ✓ P of the policy according to
[Equation]
(1) where  t is an estimate of the advantage function A ( s t , a t ) =
Q ( s t , a t )  V ( s t ) , which will bediscussed in more detail below.
Throughout this work we assume that policies ⇡ ( a | s ) are parameterized by Gaussian distributionsa i ⇠ N ( µ i ( s ) ,  2 i ) .
For simplicity, in what follows we assume  i =
for all i ; however, our resultsare easily extended to the case of  i varying across action dimensions.
Letting ✓ µ be the parametersof µ ( s ) , we have
[Equation]
Supposing that our policy is parameterized by a linear map from some feature representation of s t ,µ ( s t ) =
W µ  ( s t ) , this becomes:
[Equation]
This is a “three-factor” learning rule [17] for synapse W ij obtained by multiplying presynapticactivations  j ( s t ) , a postsynaptic term ( a t ) i  µ i ( s t ) measuring the deviation of the sampled actionfrom the typical action in this state, and a third factor  t .
RL models of the basal ganglia assume that( a t ) i  µ i ( s t ) is available to the postsynaptic neuron and that  t is signaled by dopamine releasein the striatum [49, 37].
Consistent with this model, experimental observations have shown that a coincidence of dopamine release and pre and post-synaptic neural activity is necessary for plasticityat cortico-striatal synapses [53, 7, 9].
Speciﬁc biological implementations of this learning rule arediscussed further in Appendix A.2.
In actor-critic models the  t factor is
[Equation]
often referred to as “reward prediction error” (RPE).
Here, ˆ V is an estimate of the value functionoutput by a “critic” network, separate from the policy network, which learns its parameters ✓ V usingtemporal difference (TD) learning:
[Equation]
In models of the basal ganglia, the ventral striatum is often assigned the role of the critic, as it isimplicated in value-learning but less so in controlling actions [36, 37].
Importantly, TD learning usesthe same quantity  t for learning the value function as for policy learning.
Hence, a scalar  t signalmeasuring RPE and broadcast across the striatum supports learning for both the actor and critic.
RPEcaptures key experimental features of striatal dopamine activity—responses to unexpected reward orreward-predictive cues, no signiﬁcant response to expected reward, and suppression in response tounexpected lack of reward [42, 36, 22].
Off-policy RL through action surprise signals in dopamine activity
The actor-critic algorithm described above is an on-policy algorithm.
Hence, the algorithm mayperform suboptimally, or fail altogether, if the actions used for learning are not (or not always)sampled from the learned policy.
Off-policy algorithms are often used to enable learning from areplay buffer of past experiences, expert demonstrations, and/or a separate exploration policy [1].
Ina biological context, we argue that the fact that other brain regions can exert control over behaviorindependently of the basal ganglia further motivates off-policy RL.
Indeed, later we show empiricallythat standard on-policy algorithms suffer when additional controllers, corresponding to other brainregions, exert (partial) control over an agent’s behavior.
3.1 Action-sensitive dopamine activity arises from parameterized Q -learning
While there are a variety of approaches to off-policy RL, most require learning an estimate of the
Q -function Q ( s , a ) rather than the state-value function V ( s ) .
Q -learning iteratively minimizes thefollowing loss function:
[Equation]
Computing the quantity max a Q ( s , a ) directly is intractable in high-dimensional, continuous actionspaces.
A variety of approaches to this problem have been proposed.
Here we focus on an approachadopted by [19], which involves restricting the form of the Q -function estimate to a family offunctions whose maximum is easy to compute (we discuss alternatives which have proven to beeffective in deep RL, but are difﬁcult to map to biological implementations, in Appendix A.3).
We parameterize the Q -function as follows:
[Equation]
For now we treat the the scaling factor  as a ﬁxed hyperparameter; however, it can also be learnedonline if desired (see Appendix A.4).
This parameterization can be considered a special case of theparameterization of [19].
A primary insight of our work is the observation that, under the parameterization of Eq. 8, gradientupdates of the loss function of Eq.
6 yield a biologically plausible actor-critic algorithm with actionsensitive dopamine activity.
In particular, taking the gradient of the Q -learning loss function withrespect to the parameters ✓ V and ✓ µ of ˆ V and µ , respectively, yields the following learning updates:
[Equation] [Equation]
where
[Equation]
These update equations are the same as those of the standard on-policy advantage actor-criticalgorithm, but with one additional term added to the dopaminergic signal.
Now the dopamine factor + represents a the sum of the classic RPE  and || a  µ ( s ) || 2 , which measures the deviation of thesampled action from the action that would be most likely to be chosen by the actor.
We refer to thisterm as “action surprise.”
We note that our model is agnostic to whether action surprise is encoded by the same neurons as
RPE.
It may be encoded by separate neurons as long as they release dopamine in the same areasas RPE-signaling dopamine.
We also note that the update equation (Eq. 9) for the actor uses theaction a t taken by the agent.
Biologically, this requires an efferent copy of the agent’s action (takinginto account the inﬂuence of other controllers) be sent to the striatal projection neurons representingaction in the basal ganglia.
This architecture is consistent with the presence of pathways from motorcortex, thalamus, and cerebellum to striatal projection neurons [32, 21, 4, 31].
A schematic of theconnections involved in our model is depicted in Figure 1.
[Figure]
critic module performs RL with weight updates modulated by  + t , which is a combination of both RPE and action surprise.
Dashed lines indicate architectural features of the action surprise model that are not present in the classic actor-critic model.
3.2 Differential action surprise signals in dorsal and ventral striatum
The formulation above predicts that the same dopamine signal  + is broadcast to both the actor andcritic.
However, the ratio of movement-related to reward-related modulation of dopamine activityappears to vary across the striatum, with more movement-related activity in dorsal regions [23].
Inour modeling framework, this corresponds to the action surprise term being weighted more stronglyin the actor (dorsal striatum) than the critic (ventral striatum).
Such an asymmetry is produced byadding an additional term to the weight updates of the actor, proportional to
[Equation]
This update is aligned with the gradient with respect to ✓ µ of the following loss function:
[Equation]
(14) with the learning rate itself governed by the magnitude of action surprise.
Hence, these updatesencourage the basal ganglia policy to imitate the agent’s behavioral policy on steps with large actionsurprise.
Heuristically, steps with large action surprise are more likely to have been driven by anexternal controller.
An additional action surprise contribution to actor-modulating dopamine activitycan therefore be interpreted as an approximate supervised learning signal encouraging imitation ofthe external controller’s policy.
We show in simulations that this effect helps the basal ganglia morerapidly consolidate expert policies of other controllers.
Experimental Setup
Tasks .
We simulated two simple continuous control tasks to demonstrate the role of the action surpriseterm in off-policy learning.
See Figure 2A for illustrations.
Open-ﬁeld navigation: An agent is positioned in a two-dimensional square environment with continuous coordinates.
On each trial, the initial position of the agent and a goal location are randomlysampled.
The agent controls its x and y acceleration.
Two-joint: An agent controls a two-joint arm with two equal-length segments.
The arm position anda target location are randomly sampled at the beginning of each trial.
The agent outputs torques ateach of the joints in order to move the most peripheral point of the arm toward the target.
In both tasks the cost incurred by the agent at each time step is proportional to the inverse of itssquared distance from the goal location g at each time step, plus penalties for its squared velocity andacceleration (summed across joints in the two-joint arm case)
[Equation]
Model architecture .
For both tasks we used a shallow neural network architecture with a singlehidden layer.
The state inputs to the network consist of the agent’s (angular) position and (angular)velocity, as well as the position of the target location, a total of six scalar variables.
Each of thesevariables is represented as a one-hot vector by discretizing its domain into 10 equally spaced bins.
These vectors are concatenated and the result used as the network input.
The hidden layer of thenetwork has size 256 and uses the ReLU nonlinearity, and the input weights to the hidden layer areﬁxed at their random (Kaiming uniform) initialization.
We ﬁxed these weights to avoid the complexityof biological implementations of backpropagation, since this is not the focus of our work (however,we note that the action surprise model can be applied to deep networks by backpropagating gradientsthrough additional layers).
Thus, the hidden layer activations of the network serve as a ﬁxed featurerepresentation  ( s ) of the environment state, and all learning occurs in output weights W µ and w Vwhich output actions µ = W µ  ( s ) and value estimates ˆ V = w V ·  ( s ) , respectively.
We refer tothis network as the basal ganglia network.
External controllers and training protocol .
To model the inﬂuence of other brain regions on behavior,we introduced an additional neural network that also exerted control over the agent’s actions (Figure1, orange).
To vary the performance of this controller’s policy, we trained the controller on the taskwith backpropagation for different numbers of steps: 0 (random policy), 2,000 (intermediate policy),or 100,000 (expert policy).
We also varied the degree to which the external controller inﬂuencesbehavior.
In the fully on-policy case, the external controller is not used.
In the fully off-policycase, the external controller entirely drives the behavior.
In the case of partial control, each actionis sampled from the basal ganglia’s network or the external controller with probability 0.5 each.
Additionally, we implemented an alternative partial control mechanism, in which the average of thebasal ganglia network output and the external controller output is used.
Throughout training, Gaussian exploration noise is added to the output of both networks at each step.
In all cases, for the action surprise model, the action a t used for the actor update in Eq.
9 is the actiontaken by the agent, taking into account the contributions of both controllers and the exploration noise.
On-policy baseline models .
We compared the action surprise model with two baselines in whichdopamine neurons signal pure RPE.
The ﬁrst is described by Eqs.
2 and 5, where the action used theactor update is simply the action a t taken by the agent, as in the action surprise model.
As discussedabove, this approach is potentially unstable in the presence of other controllers due to the mismatchbetween the behavioral policy from which actions are sampled and the basal ganglia network’s policy.
As a second, potentially more competitive baseline, we instead used the output of the basal ganglianetwork alone, including exploration noise, in place of a t in Eq.
We refer to this as the “no efferentcopy” model since the basal ganglia is blind to the action ultimately taken by the agent.
In thisversion, the inﬂuence of the external controller is effectively treated as part of the environment.
Wenote that this approach has no ability even in principle to learn from fully off-policy data, but canlearn when the basal ganglia network exerts partial control over actions.
Hyperparameters .
For all models we optimized the learning rate and magnitude of exploration noiseas hyperparameters.
For the action surprise model we treated the coefﬁcient 1  2 of the action surprise term as a hyperparameter.
Since the choice of  affects the overall magnitude of updates in the actor,to avoid biasing results we allowed the actor and critic learning rates to be optimized as separatehyperparameters for the RPE-only models.
See Appendix A.5 for details.
Experimental Results
100k
Episodes
C o s t 100k
Episodes
C o s t 100k
Episodes C o s t 100k Episodes C o s t 100k
Episodes
C o s t 100k
Episodes
C o s t 100k
Episodes
C o s t 100k
Episodes
C o s t 100k
Episodes
C o s t
A
B goal
[Table]
Open
[Table]
navigation
C
Two
[Table]
Open
[Table]
Open
[Table]
100k
Episodes
C o s t 100k
Episodes
C o s t 100k
Episodes
{+C o s t 
+}
100k
Episodes
C o s t 100k
Episodes
C o s t 100k
Episodes
{+C o s t 
+}
100k
Episodes
C o s t 100k
Episodes
C o s t 100k
Episodes
C o s t
Open
[Table]
field 100k 100k
Episodes
Episodes
[Table]
[Table]
[Table] [Table] [Figure] We ﬁrst analyzed performance for the case of actions partially driven by the basal ganglia network and partially driven by the external controller, intended as a representative model of behavioral control distributed across brain regions.
Fig.
2C shows that the basal ganglia policy attained superiorlearning speed and performance than RPE-only models (shown here is a representative case usingthe intermediate-level external controller).
Fig.
2B shows example trials in which this performanceadvantage manifests very clearly, with the action surprise model successfully reaching the targetwhile the other models fail completely.
The action surprise model remained advantageous when we varied the level of expertise of the external controller (Fig.
3A).
The same was true when we assessedthe combined performance of the basal ganglia network and the external controller (Fig.
3B), ratherthan the performance of the basal ganglia network by itself.
Notably, the RPE-only model withefferent copy failed entirely to learn in some cases.
We found similar results when the contributionsof the basal ganglia network and external controller were averaged rather than being combined by sampling (Appendix A.6).
We next examined the fully off-policy case.
The RPE-only model with no efferent copy failed tolearn from off-policy data in all cases, as expected, as the action term used in the update for thismodel is uncorrelated with the actions taken by the external controller.
Interestingly, the RPE-onlymodel with efferent copy also failed to learn in all cases, in fact even more catastrophically.
Theaction surprise model, by contrast, successfully learned both tasks regardless of the quality of theexternal controller’s policy (Figure 3C).
[Figure]
t I n
[Figure]
m
[Figure]
ndo
[Figure]
[Figure]
A potential concern is that the action surprise model’s improvement for off-policy learning comesat the expense of on-policy learning performance.
However, we found that even in the case of fullyon-policy learning, with no external controller, the action surprise model matched and indeed slightlyoutperformed the RPE-only model (Figure 3D).
Note that there is no distinction between the efferentcopy/no efferent copy variants in the on-policy case.
We also tested the variant of the action surprise model in which an additional update (Eq. 13) isapplied to the actor, corresponding to a higher action surprise coefﬁcient in the actor (dorsal striatum)than in the critic (ventral striatum).
We found that this addition improves learning in the presence ofexpert external controllers (Figure 3E), consistent with accelerated consolidation of the expert policy.
Biological implications of the action surprise model
The action surprise model explains several features of midbrain dopamine activity and also makesseveral testable predictions, which we outline below.
Nonspeciﬁc encoding of action .
A central property of the model is that the action surprise term|| a t  µ ( s t ) || 2 is not action-speciﬁc—it reﬂects only scalar information about the agent’s action, 240 even in high-dimensional action spaces, and does not distinguish between two equally surprisingactions.
This contrasts with the representation in striatal projection neurons forming the output of theactor network, which specify movement commands.
Indeed, experimental recordings have founddetailed encoding of kinematics in striatal projection neurons [14] but only coarse movement-relatedsignals in dopamine activity that do not reliably distinguish between movement types [10, 33].
Thisdistinction is consistent with our model and inconsistent with models that explain action-modulateddopamine activity in terms of speciﬁc motor commands.
Decrease in action-modulated dopamine activity with learning .
The action surprise model predictsthat movement-related dopamine activity is lower when the basal ganglia’s policy more closelymatches the agent’s actions.
Before the basal ganglia has learned an effective policy, actions drivenby an expert external controller will typically provoke large action surprise.
Once the basal ganglianetwork has learned an expert policy itself, action surprise will typically be lower. Thus, we predicta reduction in the magnitude of action surprise signals over the course of learning. We observed A
[Table]
Episodes
[Table]
a g n i t u
d e o f a c t
i o n s u r p r
i s e
B 0.
[Table]
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
C
[Table]
n -f i e l d n av i g a t i on
[Table]
w
[Table]
o
[Table]
n
[Table]
m
[Table]
Episodes 0. 0 a g n i t u d e o f a c t i o n s u r p r
i s e
In all these this phenomenon in both simulated tasks (Figure 4A).
This prediction is supported by a ﬁnding thatdopamine activity coinciding with lever-press movements decreases with repeated task practice [10].
Context-dependent correlation between dopamine activity and acceleration .
Several studies haveshown that striatal dopamine activity correlates with movement speed and acceleration [50, 41, 3].
Others [23, 10, 33] have shown that activity in movement-responsive midbrain dopamine neuronsis modulated primarily movement at initiation (and in some cases at movement offset) but less soduring ongoing movements.
Our model gives rise to a complex relationship between movementand action-driven dopamine signals, with the two appearing correlated on some trials but not others(Figure 4B).
On average, action-driven dopamine activity is highest at trial onset (Figure 4C), andit is positively correlated with the agent’s acceleration (Figure 4D).
Our results suggest that therepresentations of movement initiation and kinematic information attributed to dopamine activity inprior work may actually emerge as byproducts of an action surprise computation.
Neural circuits underlying the computation of action surprise .
The action surprise model requiresinformation about the agent’s action be available to midbrain dopamine neurons for computation ofthe action surprise term.
This is consistent with the presence of pathways from motor cortex andcerebellum to midbrain dopamine neurons [52, 34].
We note that the speciﬁc form of the actionsurprise term in our model, || a  µ ( s ) || 2 , is somewhat arbitrary, and other measures of distance 270 between a and µ ( s ) may be equally suitable.
We also note that our model is agnostic as to whetherthe action surprise and RPE components of dopamine activity are represented in the same individualneurons.
Empirically, RPE-signalling and movement-signaling dopamine neurons appear to comprisedistinct but partially overlapping populations in the same striatal subregions [10], though data on thisquestion is limited.
Discussion
Our work demonstrates a new link between action-related midbrain dopamine activity and reinforcement learning in the basal ganglia.
While each of these topics has received extensive treatment in theneuroscience literature, they have typically been studied separately.
Midbrain dopamine activity isknown to causally affect movement initiation and invigoration in real time [10, 40].
Consequently, thefunction of such activity is often regarded as a motivational signal [55, 35] separate from dopamine’srole in reinforcement learning.
However, action and RPE-related dopamine release coincide inthe same striatal subregions, often even in the same neurons, and with comparable magnitude[50, 41, 10, 33, 15].
Thus, it is likely that both action-related and RPE-related dopamine activityimpact cortico-striatal synaptic plasticity and learning.
Our results show that such an inﬂuence is anecessary component of an architecture capable of off-policy learning.
The effects of action-related dopamine activity, which is not a feature of classic RL models of thebasal ganglia, may be more easily overlooked in the context of simple tasks in which animals learn toassociate a small set of cues or actions with immediate reward.
In such tasks, standard RPE-basedmodels are adequate to achieve good performance.
The failure of such models that we observe, whicharises from a mismatch between the value functions associated with the behavioral and basal gangliapolicies, only emerges in tasks where reward is obtained through an extended sequence of actions.
Our model suggests that distributed control of behavior by many regions plays an important role inlearning in the basal ganglia.
A number of experiments have observed differential contributions frommultiple brain regions during learning.
For instance, some motor skills are observed to recruit themotor cortex early in learning before being consolidated into the basal ganglia [27, 24].
These resultssuggest that regions other than the basal ganglia may be more adept at ﬂexibly adapting to noveltasks, while the basal ganglia specializes in consolidating well-practiced skills.
Other studies haveobserved parallel contributions of model-based and model-free reinforcement learning strategies inthe same task [45, 18], revealing arbitration mechanisms that transfer control between them in orderto leverage the advantages of each [29, 28].
The ﬂexibility afforded by off-policy RL algorithms,such as the action surprise model, enables the basal ganglia to beneﬁt from complementary learningand control strategies adopted by other neural circuits.
Exploring how off-policy RL algorithms canbest leverage these diverse sources of expertise is a fruitful avenue for extensions to our model.
Forinstance, action surprise signals may be modulated by the conﬁdence of external controllers in orderto more efﬁciently learn from expert behavior.
Predictions of the action surprise model can be tested by experimental recordings of striatal projectionneurons, midbrain dopamine neurons, or other motor areas.
Our work also has implications forreinforcement learning.
The introduction of a biologically plausible off-policy reinforcement learningalgorithm, involving local learning rules and a single modulatory factor, enables the deploymentof off-policy RL on plasticity-enabled neuromorphic hardware [12].
Moreover, we anticipate thatinsights into how neural circuits learn from off-policy behavior governed by distributed and diversecontrollers will provide useful inspiration for RL algorithms.