Action-modulated midbrain dopamine activity arises from distributed control policies
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Animal behavior is driven by multiple brain regions working in parallel with distinct control policies.
We present a biologically plausible model of off-policyreinforcement learning in the basal ganglia, which enables learning in such anarchitecture.
The model accounts for action-related modulation of dopamine activity that is not captured by previous models that implement on-policy algorithms.
In particular, the model predicts that dopamine activity signals a combination ofreward prediction error (as in classic models) and “action surprise," a measure ofhow unexpected an action is relative to the basal ganglia’s current policy.
In thepresence of the action surprise term, the model implements an approximate formof Q -learning.
On benchmark navigation and reaching tasks, we show empiricallythat this model is capable of learning from data driven completely or in part byother policies (e.g. from other brain regions).
By contrast, models without theaction surprise term suffer in the presence of additional policies, and are incapableof learning at all from behavior that is completely externally driven.
The model provides a computational account for numerous experimental ﬁndings about dopamineactivity that cannot be explained by classic models of reinforcement learning in thebasal ganglia.
These include differing levels of action surprise signals in dorsal andventral striatum, decreasing amounts of movement-modulated dopamine activitywith practice, and representations of action initiation and kinematics in dopamineactivity.
It also provides further predictions that can be tested with recordings ofstriatal dopamine activity.
Introduction
An extensive body of work has sought to account for the function of the basal ganglia using thecomputational framework of reinforcement learning (RL), in particular RL algorithms for actionselection and value learning [29, 43, 28, 57].
Striatal neurons within the basal ganglia integratediverse inputs, including projections from across the cerebral cortex.
The activity of these neurons, inparticular neurons in the dorsal striatum plays a key role in action selection [45].
On the other hand,neurons in the ventral striatum have been shown to encode the learned value of stimuli [10, 42, 44].
The phasic activity of midbrain dopamine neurons projecting to the striatum gates plasticity at corticostriatal synapses [62, 9, 11] and may therefore modulate the learning of actions and values.
Indeed,in many RL algorithms, the learning of state-to-action mappings (“policies”) and value estimates ismodulated by a scalar factor known as the “advantage” or “reward prediction error” (RPE), whichmeasures deviations in attained reward from expectations based on learned value estimates.
Numerousexperiments have shown that striatal dopamine activity encodes an RPE-like signal [49, 42, 25].
Collectively, these ﬁndings suggest a model in which the basal ganglia implements an online actorcritic RL algorithm, where ventral and dorsal striatal subregions play the roles of critic and actor,respectively, and dopaminergic activity encodes the advantage (RPE) signal [29].
This model hasbeen extended in a variety of ways to incorporate more biological detail [55, 56, 11, 8, 54], ideasfrom model-based RL [15], distributional RL [13], and meta-RL [60].
Despite these promising links, there remain challenges to the view of basal ganglia as implementingan actor-critic RL algorithm.
In this work, we address two such challenges.
First, dopamine activityin the basal ganglia is observed to encode other information beside RPEs [18], particularly signalsrelating to movement initiation and vigor [12, 66].
Second, the classic actor-critic model of the basalganglia is an on-policy RL algorithm—it is designed to learn from experiences driven by the basalganglia’s policy.
However, motor control in the brain is distributed, with multiple brain regionsincluding the motor cortex and cerebellum exerting inﬂuence on behavior [19, 63, 3, 51, 6].
We showthat classic actor-critic models of the basal ganglia fail in this scenario and argue that the ability tolearn from off-policy experience is essential to models of basal ganglia learning.
We present a biologically plausible model of off-policy RL in continuous action spaces.
Our modeldiffers from the classic actor-critic model of the basal ganglia by adding an additional action-relatedterm to dopamine activity.
We call this term “action surprise," as it measures deviation of the actionan agent takes in a particular state from the typical action that the output of the basal ganglia wouldselect in that state.
We show mathematically that, with this addition, the algorithm implements anapproximate form of Q -learning in continuous action spaces proposed by [22].
Using simulations, weshow that action surprise is essential to effective learning when controllers other than the basal gangliaalso contribute to behavior.
Thus, action-related activity need not be understood as an independentfunction of midbrain dopamine neurons separate from their role in learning, but rather as a necessarycomponent of an algorithm that enables off-policy RL.
This action surprise model accounts for severalexperimental ﬁndings about movement-related dopamine activity in the basal ganglia.
Finally, wedescribe predictions of our model regarding dopamine activity.
Background: Connection between policy gradient algorithms andcortico-striatal plasticity
Here we brieﬂy introduce the connection between on-policy policy gradient algorithms and synapticplasticity rules in the striatum.
In what follows, we assume familiarity with standard notationand concepts in RL; for a brief review, see Appendix A.1.
Policy gradient algorithms update theparameters ✓ P of the policy according to
[Equation]
(1) where  t is an estimate of the advantage function A ( s t , a t ) =
Q ( s t , a t )  V ( s t ) .
Throughout this work we assume that policies ⇡ ( a | s ) are parameterized by Gaussian distributionsa i ⇠ N ( µ i ( s ) ,  2 i ) .
For simplicity, we will assume  i =
for all i ; however, our results are easilyextended to the case of heterogeneous  i .
Letting ✓ µ be the parameters of µ ( s ) , we have
[Equation]
Supposing our policy is parameterized by a linear map from a feature representation of s t , µ ( s t ) =
W µ  ( s t ) , this becomes:
[Equation]
This is a “three-factor” learning rule [20] for synapse W ij obtained by multiplying presynapticactivity  j ( s t ) , a postsynaptic term ( a t ) i  µ i ( s t ) , measuring the deviation of the sampled actionfrom the typical action in this state, and a third factor  t .
RL models of the basal ganglia assume that( a t ) i  µ i ( s t ) is available to the postsynaptic neuron and that  t is signaled by dopamine release inthe striatum [57, 43].
Experiments have indeed shown that a coincidence of dopamine release and preand post-synaptic neural activity enables plasticity at cortico-striatal synapses [62, 9, 11].
Biologicalimplementations of this learning rule are discussed further in Appendix A.2.
In actor-critic models the  t factor is
[Equation]
often referred to as “reward prediction error” (RPE).
Here, ˆ V is an estimate of the value functionoutput by a “critic” network, separate from the policy network, which learns its parameters ✓ V usingtemporal difference (TD) learning:
[Equation]
In models of the basal ganglia, the ventral striatum is often assigned the role of the critic, as it isimplicated in value-learning but less so in controlling actions [42, 43].
Importantly, TD learning usesthe same quantity  t for learning the value function as for policy learning.
Hence, a scalar  t signalmeasuring RPE and broadcast across the striatum supports learning for both the actor and critic.
RPEcaptures key experimental features of striatal dopamine activity—responses to unexpected reward orreward-predictive cues, no signiﬁcant response to expected reward, and suppression in response tounexpected lack of reward [49, 42, 25].
Off-policy RL through action surprise signals in dopamine activity
The actor-critic algorithm described above is an on-policy algorithm.
Hence, the algorithm mayperform suboptimally, or fail altogether, if the actions used for learning are not (always) sampledfrom the learned policy.
Off-policy algorithms are often used to enable learning from a replay bufferof past experiences, expert demonstrations, and/or a separate exploration policy [2].
In a biologicalcontext, we argue that the fact that other brain regions can exert control over behavior independentlyof the basal ganglia further motivates off-policy RL.
Indeed, we will show empirically that standardon-policy algorithms suffer when other controllers exert (partial) control over an agent’s behavior.
3.1 Action-sensitive dopamine activity arises from parameterized Q -learning
While there are a variety of approaches to off-policy RL, many require learning an estimate of the
Q -function Q ( s , a ) rather than V ( s ) .
Q -learning iteratively minimizes the following loss function:
[Equation]
Computing the quantity max a Q ( s , a ) directly is intractable in high-dimensional, continuous actionspaces.
A variety of approaches to this problem have been proposed.
Here we focus on an approachadopted by [22], which involves restricting the form of the Q -function estimate to a family offunctions whose maximum is easy to compute (we discuss other techniques for continuous Q-learningand off-policy RL, which we argue are less biologically realistic, in Appendix A.3).
We parameterize the Q -function as follows:
[Equation]
For now we treat the the scaling factor  as a ﬁxed hyperparameter; however, it can also be learnedonline (see Appendix A.4).
This parameterization is a special case of the one proposed in [22].
A primary insight of our work is the observation that, under the parameterization of Eq. 8, gradientupdates of the loss function of Eq.
6 yield a biologically plausible actor-critic algorithm with actionsensitive dopamine activity.
In particular, taking the gradient of the Q -learning loss function withrespect to the parameters ✓ V and ✓ µ of ˆ V and µ , respectively, yields the following learning updates:
[Equation] [Equation]
10) where
[Equation]
These update equations are the same as those of the on-policy advantage actor-critic algorithm, butwith one additional term added to the dopaminergic signal.
Now this signal  + represents a the sumof classic RPE  and k a  µ ( s ) k 2 , which measures the deviation of the sampled action from theaction most likely to be chosen by the actor.
We call this term “action surprise.”
It can be interpretedas a reduction in the predicted value Q ( s t , a t ) of actions that deviate from the basal ganglia’s policy.
We note that our model is agnostic to whether action surprise is encoded by the same neurons as
RPE.
It may be encoded by separate neurons as long as they release dopamine in the same areasas RPE-signaling dopamine.
We also note that the update equation (Eq. 9) for the actor uses theaction a t taken by the agent.
Biologically, this requires an efferent copy of the agent’s action (takinginto account the inﬂuence of other controllers) be sent to the striatal projection neurons representingaction in the basal ganglia.
This architecture is consistent with the presence of pathways from motorcortex, thalamus, and cerebellum to striatal projection neurons [36, 24, 5, 35].
A schematic of theconnections involved in our model is depicted in Fig.
It is easy to misinterpret the effect of the dopaminergic action surprise as encouraging the basalganglia’s policy to imitate those of other controllers.
This is in fact not the case.
Later, we show empirically that this model robustly enables off-policy learning even in the presence of poorly performingexternal controllers.
Below, however, we show how imitation learning can be implemented.
[Figure]
Both systems receive state information via a feature representation  ( s projection neuron activity).
The actor-critic module performs RL with weight updates modulated by  + t , which is a combination of both RPE and action surprise.
indicate architectural features of the action surprise model that are not present in the classic actor-critic model.
3.2 Differential action surprise signals in dorsal and ventral striatum
The formulation above predicts that the same dopamine signal  + is broadcast to both the actor andcritic.
However, the ratio of movement-related to reward-related modulation of dopamine activityappears to vary across the striatum, with more movement-related activity in dorsal regions [26].
Inour modeling framework, this corresponds to the action surprise term being weighted more stronglyin the actor (dorsal striatum) than the critic (ventral striatum).
Such an asymmetry is produced byadding an additional term to the weight updates of the actor, proportional to
[Equation]
This update is aligned with the gradient with respect to ✓ µ of the following loss function:
[Equation]
[Equation] with the learning rate itself governed by the magnitude of action surprise.
Hence, these updatesencourage the basal ganglia policy to imitate the agent’s behavioral policy on steps with large actionsurprise.
Heuristically, steps with large action surprise are more likely to have been driven by anexternal controller.
An additional action surprise contribution to actor-modulating dopamine activity can therefore be interpreted as an approximate supervised learning signal encouraging imitation ofthe external controller’s policy.
We show in simulations that this effect helps the basal ganglia morerapidly consolidate expert policies of other controllers.
Experimental Setup
Tasks .
We simulated two simple continuous control tasks to demonstrate the role of the action surpriseterm in off-policy learning.
See Fig. 2A for illustrations.
Open-ﬁeld navigation: An agent is positioned in a two-dimensional square environment with continuous coordinates.
On each trial, the initial position of the agent and a goal location are randomlysampled.
The agent controls its x and y acceleration.
Two-joint: An agent controls a two-joint arm with two equal-length segments.
The arm position anda target location are randomly sampled at the beginning of each trial.
The agent outputs torques ateach of the joints in order to move the most peripheral point of the arm toward the target.
In both tasks the cost incurred by the agent at each time step is proportional to the inverse of itssquared distance from the goal location g at each time step, plus penalties for its squared velocity andacceleration (summed across joints in the two-joint arm case) [Equation]
[Equation]
We also experimented with a version of the loss function in which rewards for reaching the goallocation are sparse and binary, rather than continuous, which gave similar results (see Appendix A.7).
Model architecture .
For both tasks we used a neural network architecture with a single hiddenlayer.
The state inputs to the network consisted of the agent’s (angular) position and (angular)velocity, as well as the position of the target location, a total of six scalar variables.
Each variablewas represented as a one-hot vector by discretizing its domain into 10 equally spaced bins.
Thesevectors were concatenated and used as the network input.
The hidden layer had size 256 and used ReLU nonlinearities. Input weights to the hidden layer were ﬁxed at random (Kaiming uniform)initializations.
We ﬁxed these weights to avoid the complexity of biological implementations ofbackpropagation, since this is not the focus of our work (however, we note that the action surprisemodel can be applied to deep networks by backpropagating gradients through additional layers).
Thus, the hidden layer activations served as a ﬁxed feature representation  ( s ) of the environmentstate, and all learning occurs in output weights W µ and w V which output actions µ = W µ  ( s ) andvalue estimates ˆ V = w V ·  ( s ) , respectively.
We refer to this network as the basal ganglia network.
External controllers and training protocol .
To model the inﬂuence of other brain regions on behavior,we introduced an additional neural network that also exerted control over the agent’s actions (Fig. 1,orange).
To vary the performance of this controller’s policy, we trained the controller on the taskwith backpropagation for different numbers of steps: 0 (random policy), 2,000 (intermediate policy),or 100,000 (expert policy).
We also varied the degree to which the external controller inﬂuencesbehavior.
In the fully on-policy case, the external controller is not used.
In the fully off-policycase, the external controller entirely drives the behavior.
In the case of partial control, each actionis sampled from the basal ganglia’s network or the external controller with probability 0.5 each.
Additionally, we implemented an alternative partial control mechanism, in which the average of thebasal ganglia network output and the external controller output is used.
Throughout training, Gaussian exploration noise is added to the output of both networks at each step.
In all cases, for the action surprise model, the action a t used for the actor update in Eq.
9 is the actiontaken by the agent, taking into account the contributions of both controllers and the exploration noise.
On-policy baseline models .
We compared the action surprise model with two baselines in whichdopamine neurons signal pure RPE.
The ﬁrst is described by Eqs.
2 and 5, where the action used theactor update is simply the action a t taken by the agent, as in the action surprise model.
As discussedabove, this approach is potentially unstable in the presence of other controllers due to the mismatchbetween the behavioral policy from which actions are sampled and the basal ganglia network’s policy.
As a second, potentially more competitive baseline, we instead used the output of the basal ganglianetwork alone, including exploration noise, in place of a t in Eq.
We refer to this as the “no efferent copy” model since the basal ganglia is blind to the action ultimately taken by the agent.
In thisversion, the inﬂuence of the external controller is effectively treated as part of the environment.
Wenote that this approach has no ability even in principle to learn from fully off-policy data, but canlearn when the basal ganglia network exerts partial control over actions.
Hyperparameters .
For all models we optimized the learning rate and magnitude of exploration noiseas hyperparameters.
For the action surprise model we optimized the coefﬁcient 1  2 of the actionsurprise term as a hyperparameter.
To ensure strong baselines we allowed the actor and critic learningrates to be optimized separately for the RPE-only models.
See Appendix A.5 for details.
Experimental Results
[-C o s t 
-]
[-C o s t 
-]
[Figure]
We ﬁrst analyzed performance for the case of actions partially driven by the basal ganglia networkand partially driven by the external controller, intended as a representative model of behavioralcontrol distributed across brain regions.
Fig.
2C shows that the basal ganglia policy attained superiorlearning speed and performance than RPE-only models (shown here is a representative case usingthe intermediate-level external controller).
Fig.
2B shows example trials in which this performanceadvantage manifests very clearly, with the action surprise model successfully reaching the targetwhile the other models fail completely.
The action surprise model remained advantageous whenwe varied the expertise of the external controller (Fig.
3A).
The same was true when we assessedcombined performance of the basal ganglia network and the external controller (Fig.
3B), rather thanperformance of the basal ganglia network by itself.
Notably, the RPE-only model with efferent copyfailed to learn in some cases.
We found similar results when the contributions of the basal ganglianetwork and external controller were averaged rather than combined by sampling (Appendix A.6).
We next examined the fully off-policy case.
The RPE-only model with no efferent copy failed to learnfrom off-policy data in all cases, as expected, as the action term used in the update for this modelis uncorrelated with the actions taken by the external controller.
Interestingly, the RPE-only modelwith efferent copy also failed (catastrophically) to learn in all cases.
The action surprise model, bycontrast, learned both tasks regardless of the quality of the external controller’s policy (Fig. 3C).
A potential concern is that the action surprise model’s improvement for off-policy learning comesat the expense of on-policy learning performance.
However, we found that even in the case of fully
[Figure]
ndo m
I n t
[Figure]
r m
[Figure]
d i a t e
[Figure]
x
[Figure]
r t A
[Figure] surprise on-policy learning, with no external controller, the action surprise model matched and indeed slightlyoutperformed the RPE-only model (Fig.
3D).
We also tested the variant of the action surprise model in which an additional update (Eq. 13) isapplied to the actor, corresponding to a higher action surprise coefﬁcient in the actor (dorsal striatum)than in the critic (ventral striatum).
We found that this addition improves learning in the presence ofexpert external controllers (Fig. 3E), consistent with accelerated consolidation of the expert policy.
Biological implications of the action surprise model
The action surprise model explains several features of midbrain dopamine activity and also makesseveral testable predictions, which we outline below.
Nonspeciﬁc encoding of action .
A central property of the model is that the action surprise termk a t  µ ( s t ) k 2 is not action-speciﬁc—it reﬂects only scalar information about the agent’s action,even in high-dimensional action spaces, and does not distinguish between two equally surprisingactions.
This contrasts with the representation in striatal projection neurons forming the output of theactor network, which specify movement commands.
Indeed, experimental recordings have founddetailed encoding of kinematics in striatal projection neurons [16] but only coarse movement-relatedsignals in dopamine activity that do not reliably distinguish between movement types [12, 37].
Thisdistinction is consistent with our model and inconsistent with models that explain action-modulateddopamine activity in terms of speciﬁc motor commands.
Decrease in action-modulated dopamine activity with learning .
The action surprise model predictsthat movement-related dopamine activity is lower when the basal ganglia’s policy more closelymatches the agent’s actions.
Before the basal ganglia has learned an effective policy, actions drivenby an expert external controller will typically provoke large action surprise.
Once the basal ganglia
[Figure]
agn i t ude
[Figure]
i
[Figure]
agn i t ude o f a c t i on
[Figure]
r
[Figure]
s
[Figure]
A
B C
D
[Figure]
deviation of action surprise term in the dopamine response over the course of learning in the case the basal ganglia shares partial control with an expert controller.
all these examples network has learned an expert policy itself, action surprise will typically be lower. Thus, we predicta reduction in the magnitude of action surprise signals over the course of learning.
We observedthis phenomenon in both simulated tasks (Fig. 4A).
This prediction is supported by a ﬁnding thatdopamine activity coinciding with lever-press movements decreases with repeated task practice [12].
Context-dependent correlation between dopamine activity and acceleration .
Several studies haveshown that striatal dopamine activity correlates with movement speed and acceleration [59, 48, 4].
Others [26, 12, 37] have shown that activity in movement-responsive midbrain dopamine neurons ismodulated primarily movement at initiation (and in some cases at movement offset) but less so duringongoing movements.
Our model gives rise to a complex relationship between movement and actiondriven dopamine signals, with the two appearing correlated on some trials but not others (Fig. 4B).
On average, action-driven dopamine activity is highest at trial onset (Fig. 4C), and it is positivelycorrelated with the agent’s acceleration (Fig. 4D).
Our results suggest that the representations ofmovement initiation and kinematic information attributed to dopamine activity in prior work mayactually emerge as byproducts of an action surprise computation.
Neural circuits underlying the computation of action surprise .
The action surprise model requiresinformation about the agent’s action be available to midbrain dopamine neurons for computation ofthe action surprise term.
This is consistent with the presence of pathways from motor cortex andcerebellum to midbrain dopamine neurons [61, 38].
It is also consistent with perturbation experimentsﬁnding a causal effect of cortical perturbations and lesions on striatal dopamine activity [32, 53, 1, 7].
We note that the speciﬁc form of the action surprise term in our model, k a  µ ( s ) k 2 , is somewhatarbitrary, and other measures of distance between a and µ ( s ) may be equally suitable.
We also notethat our model is agnostic as to whether the action surprise and RPE components of dopamine activity are represented in the same individual neurons.
Empirically, RPE-signalling and movement-signalingdopamine neurons appear to comprise distinct but partially overlapping populations in the samestriatal subregions [12], though data on this question is limited. Considerations for future experimental work .
We intend this work to motivate experimental workthat can support, challenge, or reﬁne our model.
Doing so requires careful experimental design.
First,the effects of action-related dopamine activity on learning are easily overlooked in the context ofsimple tasks in which animals learn to associate cues or actions with immediate reward.
In such tasks,standard RPE-based models may be adequate to achieve good performance.
Second, disentangling“action surprise” as we deﬁne it from other related quantities (initiation, acceleration, motivation, etc.)requires a rich action space, while many animal RL studies involve a small, discrete set of actions(e.g. binary choice). Hence, motor tasks that require sequences of movements involving continuouscontrol [30, 40] are most appropriate for our purposes. Testing predictions of the model also involvesdisentangling the behavioral contributions of the basal ganglia and other brain regions and analyzingthese in concert with simultaneously recorded dopamine activity and behavior.
While the technologyexists to conduct such simultaneous recordings, no such experiment has been performed.
Tests ofthe model would also beneﬁt from the ability to causally modulate the surprisingness of actions, forinstance using catch trials with different target movements.
Discussion
Our work demonstrates a new link between action-related midbrain dopamine activity and reinforcement learning in the basal ganglia.
While each of these topics has received extensive treatment in theneuroscience literature, they have typically been studied separately.
Midbrain dopamine activity isknown to causally affect movement initiation and invigoration in real time [12, 46].
Consequently,the function of such activity is often regarded as a motivational signal [64, 41], or as a contributorto action selection [17], separate from dopamine’s role in reinforcement learning.
However, actionand RPE-related dopamine release coincide in the same striatal subregions, often even in the sameneurons, and with comparable magnitude [59, 48, 12, 37, 18].
Thus, it is likely that both actionrelated and RPE-related dopamine activity impact cortico-striatal synaptic plasticity and learning.
Ourresults show that such an inﬂuence is a necessary component of an architecture capable of off-policylearning.
We note that our proposal does not preclude alternative roles for dopamine activity. Indeed,action-related dopamine signals are observed to both precede and follow movement [12]. Earlyresponses may participate in motivation and action selection while lingering responses responses(extending into the critical plasticity window for cortico-striatal synapses [65]) may aid in learning.
Whether similar signals can achieve both functions is an important direction for future modeling work.
Prior work [17] has suggested that dopaminergic action selection signals may correspond to imagined RPEs of simulated action outcomes according to a learned model of the environment, which maycorrelate strongly with action surprise if actions are partially controlled by a model-based system.
Our model suggests that distributed control of behavior by many regions plays an important rolein learning in the basal ganglia.
A number of experiments have observed differential contributionsfrom multiple brain regions during learning.
For instance, some motor skills are observed to recruitthe motor cortex early in learning before being consolidated into the basal ganglia [30, 27].
Theseresults suggest that regions other than the basal ganglia may be more adept at ﬂexibly adaptingto novel tasks, while the basal ganglia specializes in consolidating well-practiced skills.
Otherstudies have observed parallel contributions of model-based and model-free reinforcement learningstrategies in the same task [52, 21], revealing arbitration mechanisms leverage the advantages of each[33, 31].
The ﬂexibility afforded by off-policy RL algorithms enables the basal ganglia to beneﬁtfrom complementary learning and control strategies adopted by other neural circuits.
Exploring howoff-policy RL algorithms can best leverage these diverse sources of expertise is a fruitful avenue forextensions to our model.
For instance, action surprise signals may be modulated by the conﬁdence ofexternal controllers in order to more efﬁciently learn from expert behavior.
Our work also has implications for reinforcement learning.
The introduction of a biologicallyplausible off-policy reinforcement learning algorithm, involving local learning rules and a singlemodulatory factor, enables the deployment of off-policy RL on plasticity-enabled neuromorphichardware [14].
Moreover, we anticipate that insights into how neural circuits learn from off-policybehavior governed by distributed and diverse controllers will provide inspiration for RL algorithms.