Bioinspired Dynamic Control of Amphibious Articulated Creatures with Spiking Neural Networks
A BSTRACT
We present a biologically plausible, compact spiking neural network 2 for controlling the crawling and swimming behaviors of amphibiouscreatures with articulated skeletons.
Prior methods for learning effi- 4 cient control policies for such creatures are resource-greedy, both interms of computational time and energy requirements due to the high 6 number of degrees of freedom introduced by the many joints presentin the creature skeleton.
Our approach takes a radical departure from 8 prior work and exploits the physiology of amphibious creatures.
Specifically, we emulate experimentally identified biological con- 10 trollers for amphibious creatures with a network of spiking neurons,which alleviates the need for training altogether and can potentially 12 provide the additional benefit of utilizing minimal resources in termsof energy.
Our approach is robust and allows the amphibious crea- 14 ture to avoid both static and dynamic obstacles when exhibitingdifferent movement patterns, and also adaptively control its swim- 16 ming speed.
Moreover, we show that the creature can seamlesslytransition between crawling and swimming behaviors as it moves 18 from land to water or vice-versa, similar to its real-world counterpart.
Our approach presents an efficient and scalable alternative for 20 producing vivid and lively motion, as we demonstrate through acomplex scene where multiple amphibious creatures interact with 22 each other, successfully avoiding collisions while moving across apool of water.
Our approach is generalizable to other creatures also, 24 as we show through the design of a controller for a quadruped.
1 I NTRODUCTION
Character animation is an integral part of computer graphics [27, 29 62,68].
Beyond the immense success in the VFX industry [31,32],methods for controlling characters are finding increasing use cases 31 in robotics for applications such as housekeeping [23], autonomousdriving [69], meal assistance [47], etc.
This development is partly 33 motivated by the promise that robots may some day completelyreplace humans in dangerous or undesirable tasks, such as rescue 35 missions [48] and planetary [54] or underwater explorations [52].
While wheeled robots can effectively address simple tasks in struc- 37 tured environments, the increasing variability in unstructured environments demands articulated robots with multiple degrees of 39 freedom (DOFs).
However, this additional complexity also makes itmore challenging to control their behaviors.
The most common and successful method for training the 42 behaviors of articulated characters is Reinforcement Learning(RL)
RL algorithms have provided remarkable results 44 in a wide range of tasks spanning from high-level cognition, such as path planning [72], to low-level control of crawling [6], walk46 ing [39], and swimming [46,57] behaviors.
However, RL methods 47 also suffer from some crucial drawbacks.
First, their training phase 48 requires tens of thousands of steps, even for simple tasks such as 49 an inverted pendulum [20], resulting in considerably long training 50 times.
Second, the training time also scales with the number of 51 DOFs of the characters [64], rendering the training of such agents 52 computationally challenging.
Third, the prolonged training results 53 in significant power consumption [17,41].
Decreasing the training time of articulated characters/robots, both 55 algorithmically and computationally, and lowering power require56 ments has been a major goal for the research community, and has led 57 to the development of step-skipping algorithms [35] and specialized 58 hardware accelerators [59].
59 Alternatively, researchers have drawn inspiration from biologi60 cal agents that effortlessly give rise to complicated behaviors and 61 have attempted to replicate them in bioinspired algorithms for char62 acter control [9,38,49].
Especially for repetitive behaviors, such 63 as locomotion and swimming, numerous studies have replicated 64 the function of the specialized biological networks called Central 65 Pattern Generators (CPG) [30, 57] that are known to drive such 66 behaviors.
These networks give rise to periodic activity patterns 67 that drive the limbs and muscles of biological agents for generat68 ing the motion [25].
The periodic patterns, which are the core of 69 such networks, have been traditionally approximated with oscilla70 tors defined as dynamical systems, alleviating the need for training 71 altogether [29,30].
However, arbitrary sets of ordinary differential 72 equations (ODEs) that simulate oscillators require precise numerical 73 operations.
As a result, they are not inherently compatible with edge 74 computing hardware, such as neuromorphic processors [14,43], that 75 could also alleviate the power requirements of the control.
Even 76 when explicitly designed for deployment on such platforms to ben77 efit from their energy efficiency, ODE-based methods introduce 78 additional resource requirements [22,60].
In this paper, we exploit the identified biological principles of 80 neuronal CPG networks that give rise to crawling and swimming 81 behaviors in amphibious animals, such as lizards, salamanders, and 82 crocodiles.
We design a compact network of spiking neurons, which 83 emulates the individual biological neurons and their connectivity in 84 CPG networks to control the motions of an articulated amphibious 85 creature without the need for any training.
Our control method 86 produces biologically realistic motion for one crawling and two 87 swimming patterns, including the ability to seamlessly transition 88 between these behaviors.
Additionally, we demonstrate that the 89 amphibious creature can adapt its speed to external control signals.
90 We also incorporate sensory feedback to allow the creature to avoid 91 collisions with static and dynamic obstacles that may be present in 92 the surrounding environment.
In summary, our main contributions 93 are as follows:
• A continuous-valued oscillator that utilizes the aggregate pop-
ulation dynamics of groups of bursting neurons,
• Three novel, bioinspired architectures based on spiking neural
networks (SNN) that use populations of bursting neurons as
building blocks for controlling two different swimming pat-
terns and one crawling pattern of an articulated amphibiouscreature, and seamlessly transitioning between them, and• A closed-loop controller that responds to external stimuli andmodifies the creature’s speed and direction of motion to navigate in unstructured environments with static and dynamicobstacles
Our bioinspired control method requires no training the under- 106 standing of the biological networks that drive the swimming behaviors in animals and some empirical tuning of its hyperparameters in 108 exchange for a prolonged training that is conventionally necessaryfor driving the amphibious creature.
Our work hints that biolog- 110 ically plausible control of low-level behaviors with networks ofspiking neurons could mitigate the computational burden of train- 112 ing multi-DOF agents with RL and allow for parallel employmentof the available computational resources for performing high-level 114 cognitive tasks.
2 R ELATED W ORK
Simulating biologically plausible motion of articulated characters 117 is an active area of research in computer graphics [11,12,18].
Itsprimary goal is the demonstration of physically realistic behaviors, 119 which can preserve the robustness of their biological counterpartsto perturbation and which can adapt to control inputs.
While biped 121 locomotion has been extensively studied for understanding the emergence of human gait patterns [37,38], and also their recovery [56] 123 as well as enrichment with fine motor skills [42], the walking motions of multi-legged animals has also attracted considerable in- 125 terest [19, 44, 51].
Researchers have also explored swimming ofhumanoids [57] and aquatic animals [46], and flying in bird-like 127 characters [34,67].
The common ground in all these works is theexploration of the underlying control algorithms that can give rise to 129 these fine and complex movement patterns.
Increasing the number of DOFs in characters and the complexity 131 of the surrounding environment results in high-dimensional observation and action spaces, which cannot be easily captured in predefined 133 datasets of reasonable size.
This highlights the applicability of RLalgorithms that are based on trial-and-error for accumulating rewards 135 and punishments [4,28].
The inclusion of deep network architectures has provided remarkable generalization to diverse agents and 137 tasks, such as the generalization of policies from biped humanoidsto dinosaur-resembling quadrupeds [49], or the control of diverse 139 soft-bodied animals spanning from starfishes to octopuses and fromstingrays to squids [46].
Beyond the generalization to different 141 agents, RL algorithms can also address various tasks.
For example,the deep networks in [49] are not limited to walking control and 143 can also replicate dancing, flipping, and even throwing behaviors.
However, this versatility comes at a significant computational cost.
Even for small and simple tasks such as an inverted pendulum, thetraining requires tens of thousands of steps [20].
The training time 147 deteriorates with increase in the number of DOFs [64], as hundredsof thousands of samples are required to achieve the required gen- 149 eralization.
Moreover, long training processes need to be repeatedeven when only slight changes are made to the task or the character’s 151 skeleton [21].
In addition to compute time, long training sessionsare considerably greedy in terms of power consumption [17, 41], 153 which is an essential concern for mobile and resource-constrainedagents [63].
To mitigate the computational burden of RL training, a line of 156 research sought inspiration in the effortless emergence of repetitivebehaviors such as locomotion, swimming, and flying in biological 158 agents.
Typically, such methods approximate the biological CPGnetworks with oscillator circuits [30,71].
The resulting controllers 160 require minimal training for effectively controlling the character’sjoints and give rise to several motion patterns.
The output of such 162 networks is a sinusoid that could, theoretically, be replaced by asimple function generator.
However, a static sinusoidal function is
[Figure]
insufficient to provide effective control with real-time adaptation
[Figure]
(frequency, phase, continuity) to dynamic environments. Conversely, [Figure] the dynamical systems can adapt to external signals [30] and can also
[Figure]
incorporate learning to fine-tune their behavior [57].
However, they 168 also introduce some new limitations to the control problem.
First, the 169 generation of the oscillatory behavior relies on systems of high-order, [Figure] non-linear differential equations [57,66], whose solution requires 171 high-order numerical methods and high precision, which introduces [Figure]
computational overhead. More importantly, the use of ODE systems 173 limits their applicability to accelerated and energy-efficient edge 174 devices, such as neuromorphic processors.
This incompatibility is [Figure] due to the biological inspiration of these methods being limited to 176 the phenomenological level of the oscillatory activity.
A handful of 177 works [2,40,50] attempt to go deeper and emulate the characterized [Figure] underlying neuronal connectome of the CPG networks that drives [Figure]
the periodic activity [25].
These methods benefit from the compact
[Figure] and explainable connectivity of the biological networks to provide [Figure] effective and efficient control, but this area is still in its infancy.
[Figure] Our work follows this last direction and alleviates the need for 183 training when controlling the crawling and swimming motions of 184 articulated amphibious creatures.
Emulating the experimentally 185 identified connectome of the CPG that drives these behaviors in 186 animals such as salamanders and lizards, our compact SNN produces 187 different behaviors and naturally transitions between them.
Using a 188 small number of spiking neurons and a simplified model for each
[Figure]
of them, we reduce the computational requirements and provide a
[Figure]
controller that can be directly deployed on different neuromorphic
[Figure]
platforms, allowing for energy-efficient character control.
[Figure]
3 B URSTING N EURON M ODEL
[Figure]
Neurons in artificial neural networks sum continuous-valued inputs,
[Figure]
apply threshold functions and provide continuous-valued outputs [7]
[Figure]
Therefore, they only loosely approximate biological neurons, which
[Figure]
integrate their synaptic inputs in their continuous membrane voltage.
[Figure]
When this voltage exceeds a threshold value, the neuron emits a
[Figure]
binary, all-or-none output called a spike and resets its membrane
[Figure]
voltage.
Adding more functionality to their networks, different types
[Figure]
of spiking neurons have distinct firing modes, with a prominent one
[Figure]
being the bursting neuron [36].
The cumulative activity of a population of sufficiently many bursting neurons (B) can closely approximate a continuous
Unlike regular-firing neurons that emit solitary spikes and re- 203 set, bursting neurons fire a packet of spikes called a burst whenexceeding their threshold and reset for a more extended period of 205 quiescence (see Figure 1A).
This alternation between spiking andquiescence, which is not typical in neurons with other firing modes, 207 makes the bursting neuron a great candidate for approximating theperiodic fluctuation of an oscillator.
In fact, a single bursting neuron 209 has been used to drive the discrete, periodic movements (such as, elevation and lowering, forward and backward motion) of a hexapod’s 211 limbs and control its locomotion [50].
However, the activity of a single bursting neuron resembles a 213 binary oscillator that toggles between two discrete states.
While thisbinary nature is sufficient for driving motion patterns that consist 215 of discrete components (such as, leg elevation/lowering), it resultsin abrupt movements and discontinuous behaviors.
Consequently, 217 it hinders the applicability of the bursting neuron for continuouscontrol tasks that require intermediate position values over a range of 219 the discrete extremes.
To address this issue, we propose a continuousvalued oscillator that utilizes the population dynamics of groups of 221 bursting neurons, as shown in Figure 1B.
4 P OPULATION OF B URSTING N EURONS
Before describing our approach for simulating the spiking activity 224 of a population of bursting neurons, we explain our modeling of asingle bursting neuron.
We used the simplified neuron model in [33] 226 which is governed by the following set of equations:
[Equation]
with the voltage being reset based on the following condition:if v ≥ 30 mV , then v ← c , u ← u + d ,
[Equation]
where v is the neuron’s voltage, u is an internal state variable, ˙ v , ˙ u 229 denote time derivatives, I ex is an external input current, s j ∈ { 0 , 1 }denotes the spike of the j th presynaptic neuron, and w j is the weight 231 of the connection for the j th out of the M presynaptic neurons.
Thepostsynaptic potential v psp (set to 1 . 5 mV) is the increase in thepostsynaptic voltage membrane due to a single presynaptic spike, 234 while ε ∼ U ( − 0 .
2 , 0 . 4 ) is a uniformly distributed noise factor used 235 for emulating the variability of the population dynamics that emerge 236 from weight mismatch, synaptic failures, etc [1].
The parameters 237 a , b , c , and d define the neuron’s firing mode.
We set a to 0.02, b 238 to 0.2, c to -65, and d to 2 as suggested in [33] for simulating all 239 neurons.
For simulating the slow bursting behavior (see Figure 1A), 240 we set a to 0 . 00125 to slow down the decay of u , and c to − 47 to 241 reset v at a higher value.
We integrate equations (1) and (2) using 242 forward Euler with a time step of 0 .
5 ms.
Next, we aggregate the activity of multiple bursting neurons and 244 observe their population dynamics, as shown by the motor neurons 245 in Figure 1B. Although small populations of neurons (e.g., N = 10) 246 introduce intermediate levels of oscillatory activity, increasing the 247 population size (e.g., N ∈ { 100 , 1000 } ) improves the approxima248 tion of a continuous-valued oscillator, as shown in Figure 1C.
Even 249 in its most resource-greedy version (1000 neurons per population 250 for 12 bursting neurons, 12 motor neurons, and 2 sensing popula251 tions, 26K in total), our controller, can still be deployed on a single 252 neuromorphic processor (Intel’s Loihi includes 128 neurocores of 253 1024 neurons each [14]).
In fact, that size of our SNN is compara254 ble to similar controller networks proposed before (38 neurocores 255 in [60]), whose power consumption is about 10mW.
As a result, 256 the biological plausibility of our network and its compatibility with 257 edge-computing hardware provide significant energy efficiency for 258 the control task in a dynamic environment.
5 B IOINSPIRED M OTION C ONTROL
Using a population of bursting neurons as a building block, we pro261 pose three novel, bioinspired architectures based on spiking neural 262 networks (SNN) to control an amphibious articulated creature: (a) 263 the sequential alternation network (Figure 2A) that gives rise to al264 ternating, synchronized contractions of the right and left sides of the 265 creature’s body, (b) the chained oscillator network (Figure 2B) that 266 gives rise to an undulatory movement of the spine by propagating 267 the activation of the joints along the spine, and (c) a subnetwork that 268 we use in conjunction with the sequential alternation controller to 269 move the creature’s legs (Figure 2C) during crawling.
5.1 Sequential Alternation Controller for Swimming
This controller draws inspiration from the alternating flaps of the 272 tails of a fish when swimming [24].
Two mutually inhibiting burstingneuron populations acting as coupled oscillators form the core of 274 the sequential alternation controller, as shown in Figure 2A. Wedrive these two populations with an external input that dictates their 276 oscillation frequency.
This input corresponds to control signals fromthe midbrain [10] that descends to the spine in the form of tonic 278 excitation, a stable excitatory input [58].
To ensure that the spikingactivity of the two populations will be alternating and not concurrent, 280 we fully connect them with inhibitory synapses whose weights ware set to -1.
In this way, the population that is activated first delays 282 the activation of the other, and so on.
We use two groups of motor neurons for each spinal joint, corre- 284 sponding to the flexor and extensor motor neurons, for translatingthe alternating activation of the two bursting populations to alter- 286 nating contraction of the right and left sides of the creature’s body.
For driving the alternating contractions, we use ipsilateral excitatory 288 connections with a weight w set to 1 from one bursting populationto the flexor motor neuron, and from the other to the extensor motor 290 neuron.
For ensuring a forward swimming movement, we mimic theexperimentally identified connectivity of CPG in fish [25,55] that 292 moves the creature’s head in a direction opposite to that of the rest ofits body.
This is achieved by introducing contralateral connections 294 from the bursting neuron populations to the motor neurons onlyfor that joint.
The spikes of the flexor motor neuron increases the 296 joint’s angle, leading to a left contraction, while the spikes of theextensor motor neuron decreases it, leading to a right contraction.
For controlling the joint angle using the motor neuron spikes, wefirst calculate the normalized, instantaneous motor neuron activity 300 as follows:
[Equation]
where s Fj and s Ej denote the spikes of the flexor and extensor motorneuron populations respectively, and N is the population size.
Then, 303 we use α to modify the joint’s angle θ as follows:
[Equation]
where ⟨·⟩ denotes the clipping of the angle value between its mini- 305 mum and maximum value.
We set θ max to π / 8 to generate biologically realistic and aesthetically pleasing motion sequences.
5.2 Chained Oscillator Controller for Swimming
This controller is inspired by the undulatory movement of the spine 309 in eels, lizards, and snakes [55], which is achieved by contracting(i.e., by flexing or extending) all the spinal joints in the same direc- 311 tion.
However, unlike the sequential alternation controller, thesecontractions are not synchronized but have a phase lag.
To control this movement, we introduce one pair of bursting 314 neuron populations per joint (see Figure 2B).
These populations actas coupled oscillators for controlling each joint independently from 316 the others.
We drive all bursting neuron populations with the sametonic excitation input that dictates their oscillation frequency.
For simulating the required phase lag in the contractions of the 319 spinal joints, we break the synchrony of the different oscillators byintroducing top-down inhibitory connections from the oscillators 321 closer to the creature’s head to the ones closer to its tail, whichare illustrated by the thin lines in Figure 2B.
The weights w of 323 these connections are set to − 0 .
5 / N tdi , where N tdi is the numberof top-down inhibitory inputs that each bursting neuron population 325 receives.
In this way, the activation of the first oscillator inhibits therest, introducing a slight delay to the onset of their activity until their 327 input exceeds the inhibition.
With each bursting neuron populationinhibiting the ones below it, they are activated one after the other 329 and the oscillations exhibit the desired phase lag. [Figure]
For translating these orchestrated oscillations of the bursting neu- [Figure] ron populations to joint contractions, we introduce flexor and exten- [Figure]
sor motor neuron populations, as described in Section 5.1. We use 333 one bursting neuron population of each mutually inhibiting pair to 334 drive a flexor motor neuron population which increases the joint’s 335 angle, and the other to drive an extensor motor neuron population 336 which decreases it. For enforcing the contractions of all joints in the [Figure]
same direction, we only introduce ipsilateral connections between [Figure] the bursing neuron and the motor neuron populations. [Figure]
5.3 Leg Controller for Crawling
[Figure] For moving the creature forward when it is on the ground, we utilize 341 the rotational motors for its legs in conjunction with the sequentially [Figure] alternating spine movement (as described in Section 5.1) during 343 crawling.
To drive the forward crawling, we group the four legs into 344 two pairs.
The left front and right hind legs form one pair, and the [Figure] right front and left hind legs form the other pair. The legs in each
[Figure]
pair are synchronously rotated forward and we introduce a constant [Figure] phase lag of a half-cycle to enforce the alternating movement of [Figure]
the two pairs. To control each leg, we employed two motor neuron [Figure] populations similar to the control of the spinal joints, as shown in
[Figure] Figure 2C. To ensure the synchronization of each leg pair with the [Figure] spine contraction in one direction, we connected the two bursting
[Figure]
neuron populations of the sequential alternating spine controller
[Figure]
to the two motor neuron populations that control the legs.
We 354 set the weights of these excitatory connections to 1.
To achieve 355 forward crawling, we rotate each leg only in the forward direction [Figure] by translating the spikes of both motor neuron populations to an [Figure]
increase in the joint angle.
We set θ max to π for the leg joints. Note
[Figure] that the angle of the leg joints was not clipped using equation (5) to [Figure] allow for continuous forward rotation and crawling movement.
[Figure]
5.4 Transition between Motion Patterns
For transitioning between the crawling and swimming patterns, or 362 vice-versa, we use the creature’s position in the surrounding envi-
[Figure]
ronment as provided by a GPS sensor. When it is no longer on [Table]
the ground and sufficiently submerged into the water based on a 365 user-specified tolerance, we stop the movement of its legs and force [Table] them to point backwards for minimal resistance while swimming. [Table]
Similarly, we use the leg controller when the GPS sensor indicates [Table] that the creature has moved sufficiently out of the water. [Table]
6 S ENSORS AND F EEDBACK N ETWORKS
With a few exceptions [3,61], SNN-based CPG are open-loop con- [Table] trollers that do not modify their behavior in response to sensory [Table]
inputs [16,26].
While some of them can adapt their oscillation fre- [Table] quency to external signals [50], they still disregard sensory feedback, [Table]
hindering their applicability to dynamic environments where the
[Table]
agent needs adapt its behavior.
[Table] To remedy this, we introduce two feedback loops to our SNN- [Table] based CPG to allow the amphibious creature to interact with its
[Table]
surrounding environment. The first feedback loop utilizes a GPS [Table] sensor, from which we read the creature’s position and swimming
[Table]
speed in real-time.
This allows us to select the used motion pattern 381 as a function of its position.
The second feedback loop employs two
[Table] distance sensors that are positioned on the sides of the creature’s [Table] head, resembling a pair of eyes (see Figure 2) and allowing it to
[Table]
detect and avoid obstacles in its environment.
Each distance sensor 385 is equipped with six rays arranged equidistantly on the surface of a [Table] cone, whose angle is 45 degrees, with a maximum sensing distance [Table]
d max = 0 . 7 m, and provides a value d that is an estimate of the [Table] distance between the creature’s head and the closest object.
[Table]
6.1 Distance Sensory Feedback Network
To incorporate the distance sensory feedback, we translate the read- 391 out values d r and d ℓ of the right and left sensor, respectively, intotwo excitatory current values as follows:
[Equation]
where A is a scaling factor that is set to 70 to rescale the current 394 value to the operating range of the neuron model.
Next, we usethese excitatory currents to drive two sensory neuron populations 396 (see Figures 2A and B), whose spiking activity encodes the distanceof objects from the left and right sides of the creature’s head.
Subsequently, we use the spiking activity of these two distance 399 sensory neuron populations to modify the output of the CPG network.
In the sequential alternation controller, we decrease the contraction 401 of the spinal joints in the direction of an obstacle by connecting thesensory neuron to the flexor and the extensor motor neuron popula- 403 tions of the spinal joints with inhibitory synapses, whose weight ware set to − 1.
In this way, when an obstacle is on the left (right) side 405 of the creature’s head, the left (right) distance sensor provides a readout value d ℓ < d max ( d r < d max ).
This increases the current value I rds( I ℓ ds ) that stimulates the corresponding sensory neuron populations 408 (see equations (6) and (7)) and increases the sensory neuron activity.
The sensory neuron spikes inhibit the motor neuron populations 410 that induce left (right) contractions, turning the creature to the right(left) and avoiding the obstacle.
We achieve a similar result with 412 the chained oscillator controller by connecting the sensory neuronpopulations directly to the bursting neuron populations to inhibit 414 their activity and decrease the contractions in the given direction, asdescribed above.
Note that in the sequential alternating controller, 416 we connect the sensory neuron populations directly to the motorneuron populations, not the busting neuron populations.
This was 418 done so as to avoid interference of the sensory feedback networkwith the leg movement network since the bursting neuron activity in 420 that controller directly reflects on the frequency and range of the legmovements.
7 G ENERALIZATION TO QUADRUPED LOCOMOTION
[Figure]
To demonstrate the appli- 424 cability of our approach toagents beyond the afore- 426 mentioned articulated creature, we sought to use our 428 method to control the locomotion of a quadruped.
As 430 an example, we focused onthe Ant-v2 agent from Mu- 432 JoCo OpenAI gym that iswidely used as a test bed 434 for RL algorithms.
Similarly to the salamander con- 436 troller, a pair of mutuallyinhibiting bursting popula- 438 tions constituted the coreof the quadruped controller 440 and its specific connectivitywas then dictated by the desired locomotion pattern.
More specif- 442 ically, the legs of the quadruped were grouped in two pairs (frontleft with hind right and front right with hind left), with sequentially 444 alternating half-cycles: The first half-cycle consisted of the liftingand forward motion of the first pair, and the concurrent lowering and 446 backward motion of the second pair, while in the second half cyclethe order of the pairs switched.
To recreate this behavior, we utilized 448 one pair of mutually inhibiting BN populations, whose activity setthe pace of the sequential alternation (Fig. 3).
Each BN population(e.g., left) drove a pair of MN groups (e.g., top-left and bottom-right). [Figure]
Each MN group lifted and moved forward the corresponding leg, [Figure] while lowering and moving backward a leg of the opposite pair. To
[Figure] compare the performance of our SNN controller, we used a RL base- [Figure] line and trained the quadruped to walk using the well-established
[Figure]
Proximal Policy Optimization (PPO) algorithm
We mea- [Figure] sured the training time (number of episodes) it required to reach the
[Figure] SNN performance, and compared the reward accumulated by the [Figure] SNN controller and PPO in the early episodes of the training.
[Figure]
8 S IMULATION R ESULTS
[Figure]
We used the Webots framework [45] for our simulations, an open- [Figure] source mobile robot simulation software developed by Cyberbotics [Figure]
Ltd.
We chose this platform because it has been successfully used [Figure] by other research groups in prior work for simulating swimming [Figure]
and amphibious robots [8,13,70].
We used the salamander robot [Figure] prototype [30] and environment that are provided with this software [Figure]
package. Upon acceptance, accompanying this article, we will open- [Figure] source our code for running 3D simulations of amphibious creatures
[Figure]
with our proposed bioinspired controller.
[Figure]
In this section, we demonstrate the applicability of our proposed [Figure] method for controlling the behavior of an amphibious articulated
[Figure]
creature. Our method can effectively generate two swimming pat- [Figure] terns, as described in Sections 8.2 and 8.4, and one crawling pattern
[Figure]
(see Section 8.3) for the amphibious creature.
Moreover, our con-
[Figure] troller is robust enough to adapt the creature’s speed to external [Figure] control signals, as described in Section 8.6, and drive it to avoid
[Figure]
collisions with static and dynamic obstacles in its surrounding envi-
[Figure]
ronment (see Section 8.7).
We first present how our spiking neural [Figure] network can control the contractions of a single spinal joint
[Figure]
8.1 Single Joint Control
[Figure]
We control the alternating contractions of a
[Figure]
single spinal joint in a simplified version of
[Figure]
the amphibious creature with the core mod-
[Figure]
ule of our spiking neural network, as shown
[Figure]
in Figure 4.
The external tonic excitation in-
[Figure]
put induces bursting activity in both bursting
[Figure]
neuron populations, and their mutual inhi-
[Figure]
bition ensures the alternation of their burst-
[Figure]
ing activities.
With each bursting neuron
[Figure]
population exciting a corresponding motor
[Figure]
neuron population, the activity of the two
[Figure]
motor neuron populations is also alternating [Figure] (see Figure 5A).
Translating the motor neu-
[Figure]
ron activity to changes in the joint’s angle [Figure] (equations (4)–(5)) results in an oscillation [Figure]
between the minimum and maximum angle
[Figure]
values, as shown in Figure 5C.
This oscilla-
[Figure] tion of the joint angle translates to an alter- [Figure] nating contraction of the right and left sides of the creature’s body,
[Figure]
as illustrated in Figure 5B. As these alternating contractions of the
[Figure]
two sides of the body constitute the basis of swimming movements,
[Figure]
we move on to examine the emergence of such swimming patterns
[Figure]
with our full-scale controllers.
[Figure]
8.2 Sequential Alternation Swimming Pattern
[Figure] We extended the core module of our spiking neural network to con- [Figure] trol all six spinal joints of the amphibious creature for generating
[Figure]
a swimming pattern with alternating contractions of the right and
[Figure]
left sides of its body, as shown in Figure 6.
Our controller (see
[Figure]
Figure 2A) drives each of the six joints as described in Section 8.1, [Figure] giving rise to similar oscillations of their angles between the mini- [Figure]
mum and maximum values.
The oscillations of different joints are
[Figure]
synchronized and give rise to synchronous contractions of the left
[Figure]
side of the body that alternate with synchronous contractions of the right side (see the bottom row in Figure 6).
This synchronization 514 emerges from using a single pair of mutually inhibiting burstingneuron populations that act as a coupled oscillator and dictate the 516 pace of the contractions for the whole spine.
While synchronizedwith the rest of the spinal joints, the contractions of the first joint 518 are anti-phasic (see top row in Figure 6).
This constant phase lag of180 degrees is due to the contralateral connections of the oscillatory 520 bursting neuron populations to the flexor and extensor motor neuronpopulations of the first joint, as shown in Figure 2A. Functionally, 522 this phase reversal moves the creature’s head in the opposite direction to the rest of its body and creates the necessary propulsion to 524 move in the water.
Note that the short, synchronized joint activationin the first few milliseconds of the movement (see Figure 6) is an 526 artifact due to the initialization of the neuron stimulation.
8.3 Crawling Pattern
To allow the creature to move on the ground, we combine the al- 529 ternating contractions of its spine with the rotation of its legs viaa subnetwork that actuates the legs, as shown in Figure 2C.
The 531 movement of the legs is pairwise synchronized: the right front legand the left hind leg rotate simultaneously and, consequently, always 533 point in the same direction (see Figure 7A).
The other two legs aresimilarly synchronized.
The synchronization of the leg rotations 535 with the sequential alternation of the spine contractions gives riseto the crawling pattern.
More specifically, the two pairs of legs
537 complete a half-rotation with each spine contraction.
In this way,the right front and left hind legs always point forwards when the 539 spine contracts to the left, as shown in Figure 7A1, while the othertwo legs point backwards, and vice-versa.
This synchronization of 541 the leg rotations and the spine contractions is due to the commondrive of their corresponding motor neuron populations by the same 543 oscillatory core, i.e., the pair of mutually inhibiting bursting neuronpopulations, as shown in Figures 2A and C.
The crawling motion 545 allows the creature to initially move on the ground before enteringthe water (see Figure 7B).
Its legs keep rotating as long as the water 547 is shallow and then stop and are kept to the side of the body whenonly a swimming movement is required.
This transition from the 549 crawling to the swimming motion is dictated by the estimate of the
[Figure]
creature’s position, as provided by the GPS sensor.
[Figure]
8.4 Undulatory Swimming Pattern
[Figure]
In this section, we demonstrate how our chained oscillator controller,
[Figure]
as shown in Figure 2B, gives rise to the characteristic undulatory
[Figure]
movement of various amphibious animals, such as salamanders,
[Figure]
crocodiles, and lizards.
Similar to the swimming pattern with se-
[Figure]
quentially alternating contractions (see Figure 6), our spiking neural
[Figure]
network induces oscillations for all six spinal joint angles between
[Figure]
their maximum and minimum values, as shown in Figure 8.
How-
[Figure]
ever, these oscillations are not synchronized.
Instead, there is a
[Figure]
phase lag between the oscillations of consecutive joints (shown by [Figure] the dashed line in Figure 8).
This phase lag is due to the top-down
[Figure]
inhibition, as described in Section 8.4.
Functionally, this results in
[Figure]
the propagation of contraction initiation along the spine and gives 564 rise to the undulatory movement, as shown in the bottom row of 565 Figure 8.
Notably, the magnitude of this phase lag adapts to the 566 oscillation frequency, allowing for undulatory swimming at differ567 ent velocities without any changes to the network architecture or 568 configuration.
In agreement with experimental findings [5,65], we 569 diminish the phase lag at low and high oscillation frequencies, break570 ing the swimming pattern (see also Section 9).
We also highlight 571 that the orchestrated joint contractions generate the propulsion to 572 move the creature, without requiring an anti-phasic joint oscillation 573 as used in the sequential alternating controller (see Section 8.2).
The 574 short, synchronized joint activation in the first few milliseconds of 575 the movement (see Figure 8) is an artifact due to the initialization of 576 the neuron stimulation.
8.5 Timings
We examined the applicability of our method to real-time character 579 control by measuring the execution time of our SNN controller.
A 580 stripped-down implementation of our sequential alternation network
(see Section 5.1) simulated 1s of real-time in 689 , 63 ± 0 .
38 ms, 582 while the larger chained oscillator network (see Section 5.2) required829 , 07 ± 0 .
72 ms.
These results are averaged over 100 iterations 584 running on an Intel i5-1035G1 CPU with 8 cores at 1 GHz.
[Figure]
8.6 Speed Adaptation
[Figure]
The periodic nature of the oscillations raises the question whether [Figure] its possible to control their frequency and, thus, the creature’s swim-
[Figure]
ming speed. To investigate this we focus on the undulatory swim- [Figure] ming pattern, which is utilized by animals when fast swimming is
[Figure]
required, as it can produce high thrust forces [55].
To modify the
[Figure]
oscillation frequency, we provide different levels of tonic excitation [Figure] to the mutually inhibiting bursting neuron populations shown in [Figure]
Figure 2.
More specifically, we select six levels of input current
[Figure] I ex , which cover the operating range of the simplified neuron model, [Figure] and drive the bursting neuron populations.
For each input level, we
[Figure] ran ten simulation experiments, where we measured the oscillation [Figure] frequency of the bursting neuron populations and the corresponding
[Figure]
swimming speed, as shown in Figure 9.
Our results demonstrate that
[Figure] the increase in the tonic excitation input induces a gradual increase [Figure] in the oscillation frequency and, in turn, an increase in the creature’s
[Figure]
swimming speed.
Interestingly, the oscillation frequency and the
[Figure] swimming speed exhibit a linear relationship across a range, which [Figure] is typical for undulatory swimming across species [55].
Additionally, [Figure] the swimming speed saturates at low and high oscillation frequen- [Figure]
cies.
The reason for this is two-fold: First, those frequencies distort [Figure] the phase lag between consecutive joints, which is necessary for [Figure]
undulatory swimming. Second, at high frequencies, the mechani- [Figure] cal properties of the joints and motors do not allow for full-range
[Figure]
movements of the joints.
[Figure]
8.7 Obstacle Avoidance
The amphibious creature can not only adapt its swimming speed 612 to external inputs, but also adjust its swimming direction to avoid [Figure] collisions with obstacles in its environment.
To demonstrate this,
[Table] we utilized the distance sensors and our controllers’ corresponding
[Table]
feedback network, as described in Section 6.
We also chose the
[Table]
undulatory swimming pattern for its higher movement flexibility.
617 Focusing on static obstacles first, we simulated a complex envi618 ronment with various objects, as shown in Figure 10A. We initially oriented the creature such that its straight swimming motion would 620 have driven it into the rightmost barrel and allowed it to swim freely.
While initially swimming in a straight line, the feedback from the 622 distance sensors modified the spine contractions and turned the creature to the right.
After avoiding the rightmost barrel, its swimming 624 direction pointed towards the middle barrel.
Again, the creaturesensed the new obstacle and turned to the left to avoid colliding with 626 it.
In a similar fashion, it also avoided the left barrel to navigatesuccessfully through the complex environment.
Next, focusing on dynamic obstacles, we introduced an identical 629 swimming creature (left) that was swimming in the pond usingthe sequential alternating controller, as shown in Figure 10B.
We 631 selected this pattern for the left creature because its slower swimmingspeed allowed for the detection of its thin skeleton by the distance 633 sensors of the right creature.
We initially oriented the two swimmingcreatures so that their straight trajectories would result in a direct 635 collision and allowed them to swim freely.
While the one on the rightinitially swam in a straight line, it modified its spinal contractions 637 as soon as it detected the left creature, and turned right to avoid thecollision.
In fact, its higher swimming speed due to its undulatory 639 swimming pattern allowed it to swim past the left creature andcontinue swimming along its updated trajectory.
8.8 Method generalization and comparison
Finally, we examined whether our method could be used for the locomotion control of 644 agents beyond the salamander creature and how 646 its performance comparedagainst traditional
RL 648 methods.
For this, we 649 used the adaptation of ourcontrol method presented 651 in Section 7 to drive thelocomotion of a quadruped 653 and compared its performance against PPO.
Since 655 our SNN controller doesnot require training due 657 to its biologically inspiredarchitecture, it accumulates 659 rewards right away, while
PPO performs poorly in the first episodes of its training (see
[Figure]
Figure 12).
While PPO eventually outperformed our method as
[Figure]
expected, it took 118,337 episodes to match and exceed the SNN
[Figure]
performance. Moreover, our SNN gave rise to a consistently realistic
[Figure]
walking pattern, unlike PPO, which sacrificed motion realism to
[Figure]
further increase the covered distance (and, therefore, the rewards). [Figure] Although the comparison of our method against RL is necessary [Figure]
from an evaluation perspective, it is not perfectly fair.
Our control
[Figure]
architecture does not require thousands of training episodes to learn
[Figure]
the behavior from scratch.
However, it requires the domain-expert
[Figure]
knowledge of neuroscientists that have identified the biological
[Figure]
networks providing these behaviors in animals.
In addition, it also
[Figure]
requires some manual tuning of hyperparameters to replicate the
[Figure]
biological function.
[Figure]
8.9 Multi-agent Simulations
[Figure]
Our method scales well and allows for the simulation of multiple
[Figure]
amphibious creatures in a complex environment.
Specifically, we
[Figure]
simulated seven creatures that were driven by our spiking neural net-
[Figure]
work controllers, as shown in Figure 13.
The creatures navigated the
[Figure]
pond avoiding collisions with each other while moving along
their
[Figure]
individual trajectories.
The creatures would move along a straight
[Figure]
line from their initial positions ( T = 0 s), but their trajectories were
[Figure]
modified to avoid collisions with other creatures or due to terrain
[Figure]
irregularities.
Two of the creatures were initially on the ground
[Figure]
(index 1) or in shallow water (index 7) and utilized the crawling
[Table]
pattern to move before transitioning to swimming.
[Table]
9 L IMITATIONS
[Table]
We proposed a bioinspired method for controlling amphibious crea-
[Table]
tures with multiple DOFs that addresses some drawbacks of well-
[Table]
established methods, such as the long training times and high com-
[Table]
putational overhead, and also has the potential to reduce power
[Table]
requirements.
However, it is also challenged by its own limitations.
[Table] While our controller gives rise to three different behaviors, sepa- [Table] rate subnetworks drive each of them.
Although universal controllers
[Table]
are challenging even for mature training methods, the requirement to [Table] design a specialized subnetwork for each behavior limits the general- [Table]
ity of our approach.
Beyond its task-specific design, our architecture [Table] is also tailored to the number of DOFs in the creature’s skeleton. [Table]
While extending the spiking neural network to control additional [Table] joints in the spine is straightforward, completely different repetitive [Table]
behaviors require manual intervention. The generalization to tasks [Table] for different agents (e.g. biped locomotion) that require balancing
[Table]
based on extensive feedback would introduce additional challenges
[Table]
to our method and require further refinement.
[Table]
The separation of subnetworks that give rise to the three motion
[Table]
patterns introduces some discontinuity when transitioning between
[Table] behaviors. Specifically, the leg rotations are synchronized with the 707 alternating contractions of the spine during crawling, as discussedin Section 8.3, providing a smooth transition to the sequentially 709 alternating swimming pattern.
However, when transitioning to theundulatory swimming pattern, the transition is abrupt.
It would be 711 interesting to investigate control strategies that can allow the creatureto smoothly ease in and out between different motion patterns.
Although our method is designed to mitigate the computational 714 requirements of other approaches, computation can still pose asa potential limitation.
Our biologically plausible spiking neural 716 network inherits the compact structure of its biological counterpart.
However, the size of the neuron populations is decisive for the 718 elegance of the resulting motions.
In the current context, we wereable to effectively control the amphibious creature with as few as 720 20 neurons per population, and their total number would scale uplinearly with the addition of extra joints.
However, increasing the 722 number of neurons per population to 100 or 1000 could make themotion patterns more smooth and intricate (see Figures 1B and C).
Finally, the number of distance sensors and their positioning on 725 the creature’s head limited the sensing capabilities of our network.
Using a higher number of more sophisticated sensors and integrating 727 them into our feedback loop could further improve the control.
10 C ONCLUSION AND F UTURE W ORK
We presented a bioinspired method for controlling an articulated 730 amphibious creature that demonstrated one crawling motion andtwo swimming motions without the need for any prior training.
Our 732 method allows for closed-loop control with feedback from sensoryinput.
This allowed the creature to transition between different mo- 734 tion patterns, avoid collisions with static and dynamic obstacles, andalso adaptively control its swimming speed.
We demonstrated the 736 scalability of our method by showing multiple amphibious creaturesinteract with each other in a complex environment.
The structure of the creature on which we tailored our method 739 set the limits for the emergent behaviors that we could target.
For
[Figure]
example, with the spinal joints moving only along the horizontal
[Figure]
plane, we could not target three-dimensional swimming motions or [Figure] diving and resurfacing behaviors.
In the future, we plan to investigate [Figure] motion control for creatures with more complex skeletons. Moreover,
[Figure]
we would like to explore whether and how the inclusion of more
[Figure]
sensors of increasing sophistication (inertial units, lidar, etc) would
[Figure]
allow for the generalization of our control methods to other tasks
[Figure]
and agents (e.g. biped locomotion).
[Figure]
More broadly it would be interesting to develop a workflow for
[Figure]
designing controllers for low-level behaviors that would mitigate the
[Figure]
need for domain-expert knowledge and manual design, as mentioned
[Figure]
in Section 9.
We envision training-free controllers for low-level be-
[Figure]
haviors, such as locomotion or swimming, being integrated with
[Figure]
trained controllers for high-level cognitive tasks for providing effi-
[Figure]
cient performance that matches that of biological agents.
[Figure]