Neuroprospecting with DeepRL agents
Abstract
A virtuous cycle between neuroscience and deep reinforcement learning is emerging and the AI community can do much to enable and accelerate it.
Introduction: There is a rich history of cross-pollination between the technological pursuit of developing artiﬁcial intelligence (AI) and the scientiﬁc endeavor to understand biological intelligence [1, 2, 3, 4].
Deep Neural Networks (DNNs), which are now ubiquitous in AI systems, were initially inspired by the brain.
Coming full circle, DNNs now form the basis of the leading models of neuronal systems (e.g. perceptual, cognitive, motor control) and have eclipsed traditional mathematical models in their ability to explain experimental data [5].
While these studies have predominantly used supervised learning to train their networks, a small but steadily growing set of recent studies investigating complex naturalistic behavior and high-level cognitive functions such as meta-learning, decision-making and control [6, 7, 8, 9, 10, 11] have instead used used deep reinforcement learning (DRL) [12, 13] to train their neural network models in the agent setting.
Recent perspectives, written for an audience with a background in neurobiology, discuss the opportunities that this alternative framework offers to neuroscience [14, 15].
Here we describe some of the current technological and algorithmic challenges in this emerging niche that AI researchers could help address, and also highlight some potential opportunities for cross-pollination with AI.
Design and analysis of DeepRL agents: Like many other scientiﬁc domains in recent years neuroscience has embraced the use of DNNs to learn models that mimic the function of a speciﬁc neuronal system.
This trend was accelerated by the availability of cheaper computing power, large labeled training datasets (such as in computer vision or NLP) and user-friendly deep-learning frameworks.
Similarly, recent neuroscientiﬁc studies using DRL are beneﬁting from the ongoing development and standardization of DRL training frameworks [16] and physics simulators [17, 18, 19].
However, much work remains to be done to realize the true potential of this new paradigm.
As summarized in Figure 1a, we believe that there are three dimensions along which the currently
AI-focused DRL ecosystem can be developed further in the aid of such neuroscientiﬁc studies.
First, biophysical complexity can be incorporated into DRL agents across several modeling levels.
Unlike current studies in decision making that assume action spaces that directly imply decisions, more realistic motor-control could emerge by embodying agents in a biomechanical body model [20, 21, 22, 23].
Artiﬁcial neural networks (ANNs) are coarse approximations of real biological circuits that do not account for information stored in individual neuronal spikes or on the timescale of spike timing differences.
Using spiking neural networks, along with emerging techniques for reward-based training [24, 25, 26] could lead to more biologically plausible models.
Furthermore, constraining network wiring using connectomes from model organisms [27, 28, 29] could provide a ﬁne-grained correspondence between biological circuits and trained ANNs.
Second, tasks and simulators tailored to neuroscience could produce agents with more biologically plausible neural computations and richer behavioral repertoires.
Unlike most current studies that focus on one sensory modality, providing agents with multiple senses in ﬂexible or procedural high-ﬁdelity 3D virtual-reality environments [30, 31] could further capture further biological complexity and simulate more realistic perception.
Similarly, while current studies train networks on individual tasks [32, 33, 34, 35], training agents on multiple tasks [36, 21] or using auxiliary objectives that might not
Model grounding & complexity
Hypotheses
Artificial neural network agent
Real-world experiment
Algorithmic + Training algorithms + Grounding to experimental data + Auxiliary rewards + Reverse engineering neural computation + Comparing agents
Biophysical details + Sensory modalities + Joints/anatomy + Network modularity + Spiking networks + Timescales & neuromodulation
Simulation + Multiple tasks + Bio-inspired curricula + Multisensory+ Simulation fidelity + Generative and procedural models
[Figure]
be aligned with the agent’s primary task (such as minimizing metabolic cost [37, 34]), could produce agents with not just a richer repertoire of behaviors, but also more complex neural activity structures with shared task-representations, shared task-dynamics and task-speciﬁc adaptations [38, 39, 40, 41].
Third, the core algorithms and strategies for training agents could also be adapted to the goals of this scientiﬁc niche.
Learning algorithms that respect biological constraints (e.g. excitationinhibition balance, Dale’s law) have received attention in the supervised-learning setting [42, 43] but not in the reinforcement-learning setting.
Other directions for methodological innovation in agent training include using meta-algorithms like evolution [44], exploiting environmental symmetries or other regularities in training [45], using ofﬂine methods to directly ground agent behavior in recorded animal behavior [46], using algorithms constrained by morphology [47], and using complex biologically-inspired training curricula [48, 49, 50].
Neuroscientists train task-solving DNNs to generate hypotheses for how neural systems implementing distributed computation solve problems in perception, cognition or other psychological capabilities [2].
Trained task-solving ANNs and agents often exhibit behavioral features or neural dynamics that resemble their biological counterparts.
This makes them useful as tools for generating hypotheses of behavior and neural computation in conditions where neural recording, or experiments might be challenging or expensive.
Such hypotheses could guide future real-world experiments (Figure 1b).
See [51, 52] for striking recent examples of such artiﬁcial-network inspired neuroscience experiments.
Moreover, the process of designing an optimization objective for training such networks provides insights into systems at an abstract level that descriptive approaches typically cannot [53, 54].
Currently popular methods for neural network understanding [55, 56] are focused on comparing the responses of feedforward networks to a library of stimuli, and, by design, do not take into account the order or amount of stimuli previously seen by the network.
Though alternative approaches developed for use with RNNs do take stimulus history into consideration [57, 58, 59], they do not account for the network being deployed in a closed-loop agent setting where the network’s current state determines it’s next action, which in turn affects the next observation seen by it.
New theoretical development is required for comparing RNN-based agents, speciﬁcally to deal with small differences in how agents react to stimuli, that result in compounding differences over behavior trajectories [60].
Neuroprospecting: The well-established practice of searching in nature for molecules, microorganisms, plants, and other species containing biochemical or genetic information of commercial value is known as bioprospecting.
In a similar spirit, we use the term neuroprospecting to refer to the search for mechanisms, architectures and algorithms implemented in neurobiological systems that could inspire new approaches in AI.
As artiﬁcial agent designs and analyses become increasingly sophisticated, more are more opportunities arise for extracting biological middleware that could inspire advances in artiﬁcial intelligence.
This could potentially allow artiﬁcial systems to behave more like biological creatures, that are known to be born with sophisticated innate capabilities [62], and be more vastly more robust and swift to adapt in complex real-world environments [63, 64].
A virtuous cycle between real-world experiments and increasingly complex yet biologically grounded artiﬁcial models could beneﬁt both the neuroscience and machine intelligence communities [65, 66, 4, 67].