V ISUAL HYPERACUITY WITH MOVING SENSOR AND RECURRENT NEURAL COMPUTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Dynamical phenomena, such as recurrent neuronal activity and perpetual motion of the eye, are typically overlooked in models of bottom-up visual perception.
Recent experiments suggest that a tiny inter-saccadic eye motion (“ﬁxational drift”) enhances visual acuity beyond the limit imposed by the density of retinal photoreceptors.
Here we hypothesize that such an enhancement is enabled by recurrent neuronal computations in early visual areas.
Speciﬁcally, we explore a setting involving a low-resolution dynamical sensor that moves with respect to a static scene, with drift-like tiny steps.
This setting mimics a dynamical eye, viewing objects in perceptually-challenging conditions.
The dynamical sensory input is classiﬁed by a convolutional neural network with recurrent connectivity added to its lower layers, in analogy to recurrent connectivity in early visual areas.
Applying our system to CiFAR-10 and CiFAR-100 datasets down-sampled via 8x8 sensor, we found that (i) classiﬁcation accuracy, which is drastically reduced by this down-sampling, is mostly restored to its 32x32 baseline level when using a moving sensor and recurrent connectivity, (ii) in this setting, neurons in the early layers exhibit a wide repertoire of selectivity patterns, spanning the spatio-temporal selectivity space, with neurons preferring different combinations of spatial and temporal patterning and (iii) curved sensor’s trajectories improve visual acuity compared to straight trajectories, echoing recent experimental ﬁndings involving eye-tracking in challenging conditions.
Our work sheds light on the possible role of recurrent connectivity in early vision as well as the roles of ﬁxational drift and temporal-frequency selective cells in the visual system.
It also proposes a solution for artiﬁcial image recognition in settings with limited resolution and multiple time samples, such as in edge AI applications
I NTRODUCTION
Biological vision is known to be a dynamical process.
Two factors contributing to these dynamics are eye motion and recurrent neuronal connections in the brain.
Our eyes move constantly with movements that, kinematically, can be divided into saccades - quick gaze shifts, and drifts - small scanning movements between saccades (often referred to as “ﬁxational drift”)
(Rucci et al., 2018).
These dynamical aspects of vision are reﬂected only partially in contemporary computer vision systems.
Some works addressed large scale shifts in visual attention resembling saccades (Mnih et al., 2014).
Others explored properties and beneﬁts of recurrent top down connections (Nayebi et al., 2018), reminiscent of top-down processing in biological vision (Hochstein & Ahissar, 2002).
Notably, the dynamics of low-level visual processes, occurring early in the bottom-up visual hierarchy and sensitive to the ﬁxational drift (Snodderly et al., 2001; ¨Olveczky et al., 2003; Malevich et al., 2020; Hohl & Lisberger, 2011), remains largely overlooked in models of vision as well as in bio-inspired computer vision systems.
In fact, since the seminal studies by Hubel & Wiesel (1962), selectivity in primary visual cortex has been traditionally described in terms of static spatial ﬁlters (e.g., simple and complex spatial ﬁelds or Gabors of varying frequency and orientation).
In convolutional neural networks (CNNs) (Krizhevsky et al., 2012), which have dominated computer vision over the last decade, features resembling the spatial ﬁlters deduced from biological studies emerge spontaneously over the course of training (Zeiler & Fergus, 2014; Lindsey et al., 2019).
In some cases, remarkable correlations were found between spatial neural representations in CNNs and those identiﬁed in the biological brain (Yamins & DiCarlo, 2016).
On the other hand, temporal dynamics, and sensitivity to temporal features, characterize visual neurons throughout the visual system, from retinal receptors and ganglion cells to thalamic and cortical neurons (Berry et al., 1997; Chichilnisky, 2001; Lee et al., 1981;
Levick et al., 1972; Reinagel &
Reid, 2000; Shimaoka et al., 2018).
Existing evidence suggests that both eye motion (Snodderly et al., 2001; Ahissar & Arieli, 2001; ¨Olveczky et al., 2003; Malevich et al., 2020; Gruber et al., 2021; Hohl & Lisberger, 2011) and recurrent neuronal connectivity (Bejjanki et al., 2011; Samonds et al., 2013) contribute to this temporal dynamics. Furthermore, it was found that recurrent connections improve correlates of artiﬁcial neural networks to neural activity in visual cortical areas (Kar et al., 2019; Kubilius et al., 2019; Kietzmann et al., 2019).
One niche where spatio-temporal computation is probably necessary is the perception of tiny objects.
It is well known that the acuity of biological vision is not limited by the spatial resolution of retinal photoreceptors (“visual hyperacuity”; Westheimer (2009); Barlow (1979)).
Vernier acuity, for example, is dramatically higher than might be expected from pure spatial acuity derived from the photoreceptor density in the retinal mosaic (Westheimer, 2009).
Whether hyperacuity is obtained via spatial, temporal, or spatio-temporal mechanisms is not yet known (Rucci et al., 2018).
In any case, it is evident that the visual processing allowing hyperacuity, or perception of any tiny stimulus, should cope with the ﬁxational drift; if it doesn’t, the drift, whose amplitude is at least two orders of magnitude larger than the smallest perceivable spatial offsets, would impair acuity (Ahissar & Arieli, 2001; Rucci et al., 2018; Ratnam et al., 2017).
The same drift motion could potentially improve acuity if spatio-temporal computations are employed.
Such computations can be based on the emphasis of high-frequency spatial details (Rucci et al., 2007), temporal coding of spatial offsets (Ahissar & Arieli, 2001; 2012), Bayesian inference (Anderson et al., 2020), or on any other derivative of the interactions between ocular motion and the external image.
Furthermore, it is reasonable to attribute such spatio-temporal computations to early visual areas, which are known to exhibit faster dynamics and shorter integration windows compared to regions upstream in the visual processing chain (Gauthier et al., 2012).
Indeed, it had been shown that the recurrent neuronal circuitry in early visual areas could enable countering the blurring from retinal motion (Burak et al., 2010).
Using the information available from over-sampling low-resolution images has an extensive history in computer vision as part of the ﬁeld of super-resolution (Milanfar, 2017).
Multi-image superresolution (MISR) (Farsiu et al., 2004), distinguished from single-image super-resolution (Glasner et al., 2009; Dong et al., 2015), aims to reconstruct high-resolution images from a set of lowresolution ones (Areﬁn et al., 2020;
Ge et al., 2018;
Bhat et al., 2021; Li et al., 2017).
An adjacent ﬁeld of research, low-resolution object recognition, investigates algorithms to maximize the performance on a given task (Xi et al., 2020;
Ge et al., 2018).
Both ﬁelds use low-resolution images as input but differ in the goal of the training and evaluation.
In this paper, we introduce a classiﬁer that exploits spatio-temporal computations in early layers to perceive tiny images.
More speciﬁcally, we trained a convolutional neural network with recurrent connectivity (Areﬁn et al., 2020) introduced to early layers.
The network receives a sequence of low-resolution images generated via sensor motion mimicking ocular drift.
We used high-resolution images to obtain a set of features that were then used to facilitate learning in a teacher-student framework (Hinton et al., 2015).
The outcome is a dynamical classiﬁer that suffers from only a small drop in accuracy when tasked with a signiﬁcant decrease in spatial resolution, a decrease that substantially impairs the accuracy of a comparable static feed-forward classiﬁer.
Using a novel generative model, we found that our dynamical classiﬁer developed features that were primarily sensitive to spatial changes, others that were primarily sensitive to temporal changes, and a majority that exhibited sensitivity to mixed spatio-temporal patterns.
Finally, when examining the correlations between patterns of motion and accuracy of classiﬁcation, we observed that curved trajectories are favorable for recognition, consistent with recent ﬁndings of the curvature of ﬁxational drift trajectories in humans.
(Intoy & Rucci, 2020; Gruber & Ahissar, 2020).
R ESULTS 2. T ASK AND MODELS
To create a synthetic setting reminiscent of ocular drift, we used images from popular CiFAR datasets (Krizhevsky et al., 2009), embedded in a large (200x200 pixel) scene padded by zeros.
Sensor position was deﬁned in units of pixels on the scene and its motion was modeled by a stochastic process that is discussed below.
The sensor’s frames were obtained by cropping a 32x32 pixels window from the scene around the sensor position.
Resolution was then reduced to 8x8 using a standard OpenCV (Bradski, 2000) function (that does not include anti-aliasing ﬁlter) with bi-cubic interpolation (Fig. 1A).
A ResNet50 (He et al., 2016) network pre-trained on Imagenet (Deng et al., 2009), which is available as a part of Keras package (Chollet, 2015), was used as a model of reference.
The model was ﬁnetuned to one of the CiFAR datasets, reaching accuracy of 96.83 and 82.94 percent for CiFARand CiFAR 100, respectively (Table 1, Standard resolution (32x32)).
In order to feed the 32x pix CiFAR images to the network, images were upsampled by factor of 7 to (32x7)x(32x7) pix (i.e.
224x224 pix which are the dimensions of ImageNet images).
In order to verify the generality of our conclusions, we tested another more compact variant of reference CNN with 3M parameter model.
This smaller network that we refer to as Small-net (Table
S7,S8) receives 32x32 pix images as input, therefore, eliminating the need to upsample the CiFAR images before feeding them into the network.
This smaller network also simpliﬁed the analysis of internal representations as explained below.
2.1. T RAINING
We applied a feature learning paradigm (Hinton et al., 2015).
while using our reference network as teachers for the dynamical recurrent classiﬁer (DRC) student.
Typical CNNs perform a series of spatial pooling operations.
Max pooling layers in the reference
CNNs effectively reduce spatial resolution while preserving relevant information about the underlying scene.
To develop our DRC, we exploited this spatial pooling line-up.
We thus took instances of trained CNNs and replaced their bottom layers with recurrent convolutional networks (Fig. 1B).
Speciﬁcally, we used a stack of ConvGRU layers (Ballas et al., 2015; Van Valen et al., 2016) without spatial poolings to replace the original network all the way from the input to the point where the CNN’s spatial resolution is reduced by the desired factor (Table S3).
In our case, the resolution was decreased by factor of 4, therefore the appropriate resolution was achieved after the second max-pooling layer.
At this point, considering the 2 DRC systems we have tested, the resolution of the ResNet50-based teacher is (8x7)x(8x7) (the ’x7’ factor is due to the upsampling of the original 32x32 images by a factor of 7 to (32x7)x(32x7)); while the Small-net-based teacher resolution is 8x8.
We refer to the bottom recurrent part of the DRC as DRC-front-end (DRC-FE).
For the rest of the processing stack we reuse the reference (teacher) network architecture (either ResNet50 or
Small-net).
We refer to this reused part of the DRC as DRC- back-end (DRC-BE) (Fig.
1B).
We trained the DRC in two steps - ﬁrst, the DRC-FE was trained to reproduce features of the teacher network.
Here we used mean-squared loss between the teacher network and the DRC-FE (other optimization goals, such as mean absolute loss and cosine similarity, resulted in very similar performance and are not shown).
Positional data were concatenated with the images time series; see Appendix B.4 for further details.
Next, the DRC-BE was ﬁne-tuned using cross-entropy loss.
Our model was mostly implemented in the Keras package (Chollet, 2015), with the convolutional
GRU layer adapted from the project of (Van Valen et al., 2016) 1 .
2. P ERFORMANCE 2.2. B ASELINE
In order to evaluate the performance improvement which can be attributed to the unique architecture of the DRC, we considered a few baseline solutions.
[Table]
The accuracy of the ResNet50 reference (teacher) network when applied to a single low resolution image, was chosen as a simplistic baseline.
The performance of this network, shown in Table(’Naive training’), demonstrates a large degradation of accuracy in both datasets (See also Table
S4 for network’s architecture).
To facilitate fair comparison, we also checked trained such a naive classiﬁer with feature learning as done in DRC.
The results here did not change signiﬁcantly.
As a more advanced baseline, we considered an averaged prediction (AP) of a feed-forward model over the T sampled frames.
Namely, the estimated probability ˆ p k of a class k is given by ˆ p k =
T (cid:80) Tt
=
ˆ p tk , where ˆ p tk are predictions of the above naive baseline.
The situation here is similar to test time data augmentation (Perez & Wang, 2017) with sensor motion being the augmenter.
Notably, the AP saturated with the number of timesteps while our full system, as described below kept improving (Table 1).
Here application of a teacher slightly improve the accuracy, and in the case of CiFAR-100, the improvement is signiﬁcant ( 1 . 34% on average).
Next, we evaluated models where a convGRU (resp.
GRU) is connected before (resp.
on the top of) the last global average pooling layer of ResNet; we denote it as Resnet+convGRU (resp.
ResNet+GRU)
(Table S6).
At their best, these models achieved accuracy lower by approximately 4% and 7% for CiFAR-10 and CiFAR-100 datasets respectively, compared to 5-step DRC w/o positional information.
The fact that these models and the AP achieve approximately equal performance indicates that trainable recurrent connectivity in top layers has little beneﬁt over-simplistic integration.
This is in contrast to the DRC, where recurrent connectivity is implemented in the low layers.
This result is not surprising since convolutional networks tend to develop invariance to small shifts (Zeiler & Fergus, 2014), such as those that DRC relies on, albeit with important caveats (Azulay &
Weiss, 2018).
Finally, we refer to a recent work (Xi et al., 2020) that uses a generative adversarial network to enhance feature representation in CiFAR-10 task with 8x8 resolution.
This solution performs slightly better than DRC without positional information and ﬁve timesteps but underperforms versus the same DRC setting with ten steps.
Furthermore, no results for CiFAR-100 are available in this work.
[Table]
O UR MODEL
Table 1 Summarizes the performance of the DRC on both datasets.
The simplest version, with ﬁve time-steps and with no positional encoding outperforms the baseline solutions, including the GRUbased one.
It can be clearly seen that adding time steps or increasing the network’s depth is leveraged to higher accuracy in both datasets.
The version with ten time steps achieves accuracy which is just 2-4% inferior to the full resolution setting.
To check whether DRC advantage vs. baselines relies solely on the recurrent computation in the bottom layers, we tested it without motion.
The corresponding result in Table 1 indicates that this is not the case.
Table 2 reports four representative examples of Small-nets trained on CiFAR-10.
Here we see that same trends of performance hold for a shallower and more compact network architecture and for teacher trained from scratch, without transfer learning.
2. R EPRESENTATION
To better understand how our dynamical network extracts high-level features from low-resolution images, we analyzed the activation sensitivity of each of the 64 neurons of the ﬁnal layer of the DRCFE (“feature-neurons”).
We started by using activation maximization with gradient ascent over the input pattern space (Zeiler & Fergus, 2014) and obtained the maximally-activating patterns (AMs) for the teacher network (Fig. S7).
Unfortunately, applying this tool to the spatio-temporal features learned by the student network failed to converge systematically (Fig. S4).
We thus developed a deep generator network (DGN) (Table S9 and Appendix A), partially inspired by (Nguyen et al., 2016), that proved capable of repeatably producing AMs with spatio-temporal patterns, while remaining consistent with the results obtained using gradient ascent in a purely spatial setting (Fig. S7).
For each feature-neuron, we found a speciﬁc series of images that maximized its activity, by allowing the generator to devise unconstrained spatio-temporal patterns.
As previously seen in (Zeiler &
Fergus, 2014), we found that Gabor-like images maximized the activation of many feature-neurons of the teacher network, reminiscent of the sensitivity of neurons in the early visual system to similar stimuli (Carandini et al., 2005).
As expected, we also found that the Gabor-like patterns in our (dynamic) students’ feature-neurons were often reminiscent of those of the (static) teacher.
Importantly, however, the features presented in our student network exhibited dynamics with high spatial and temporal variability, resembling visual receptive ﬁelds for drifting Gabors (Figures 2A, S8, S9).
To isolate the contribution of spatial and temporal variations to the AM of each feature-neuron we explored two settings of constrained maximization (see Fig.
2B for visualization of both constraints).
Speciﬁcally, we tasked the generator with creating either purely spatial input patterns, where all frames must be identical (see middle rows in examples at Fig.
2B), or purely temporal patterns in which differences between frames were allowed, but all the pixels of each frame were identical (e.g., bottom rows in Fig.
2B).
We found features that were more sensitive to temporal dynamics along with others that were more sensitive to spatial dynamics, with the majority of features exhibiting mixed spatio-temporal sensitivity patterns (Fig. 2B).
Interestingly, many features exhibited spatio-temporal AMs that were substantially higher than the corresponding purely spatial and purely temporal AMs, suggesting speciﬁc coding beneﬁts for spatio-temporal ﬁelds in our dynamic network.
This ﬁnding illustrates the importance of studying spatio-temporal receptive ﬁelds in the visual system (DeAngelis et al., 1993; Rust et al., 2005).
2. S ENSOR ’ S TRAJECTORY AND ITS EFFECT ON PERFORMANCE
While ocular drift is considered a diffusive, stochastic process, recent evidence suggests that its high-level properties are controlled by the brain in stimulus or task-dependent manner.
In particular, (Gruber & Ahissar, 2020; Intoy & Rucci, 2020) demonstrated that ocular drifts in human subjects exhibit more curved paths when viewing more informative regions.
We thus examined the effect of our sensor trajectory on recognition accuracy.
We devised a family of stochastic diffusive trajectories with controllable curvature properties.
To do that, we assumed that the random distribution of angles between successive intervals is governed by von Mises distribution (Appendix B for details).
Testing our DRC with varying κ (the parameter which controls the trajectory’s mean curvature; see Appendix B for details)
we found that the recognition performance improved with curvature, providing a possible functional interpretation for the experimental ﬁndings of (Gruber & Ahissar, 2020; Intoy & Rucci, 2020).
Figure 3 shows gradual improvement in accuracy on CiFAR-100 dataset
To distinguish between the properties of the actual trajectory with the generated sequence of points as opposed to the collection of sample points, we performed a shufﬂing experiment.
Fig.
3 shows that randomly shufﬂing points on the trajectory mostly affect trajectories with low curvature.
For trajectories with negative kappa, shufﬂed trajectory visually resembles non-shufﬂed ones and, therefore, it is not of surprise that shufﬂing has little or no effect.
On the other hand, for trajectories with low curvature shufﬂing does have a signiﬁcant effect.
This implies that while the same information is available in both cases, the decoding is compromised in the case of a shufﬂed path.
To leverage the advantage of the trajectory’s curvature further, we devised another family of trajectories for which curvature was explicitly enforced.
We refer to these trajectories as “spirals” (Appendix for details).
As can be seen in 3 the ’spiral’ trajectories resulted in the best performance.
The ’spiral’ ensemble of trajectories is the one that is reported in Tables 1 and 2.
We used the curvature-index deﬁned in Gruber & Ahissar (2020)
(See also Appendix B) in order to compare our synthetic trajectories with real drift trajectories acquired in an experiment (Gruber & Ahissar, 2020).
We found that by using κ values of -1.0 and 0.0 the generated distributions have similar characteristics (see details in Appendix B) to the ones observed in the experiment with the relevant condition- recognizing small 2d shapes.
The empirical distributions in (Gruber & Ahissar, 2020) were obtained by dividing the data to periods where relevant information is projected on the retina, and periods when such stimulus is not available to the retina (compare Fig.
S10 with Figure 3, ’Natural-small’ curvature-index distributions in Gruber & Ahissar (2020)).
These similarities demonstrate that (a) the range of kappas we are using is biologically relevant, and (b) our model can be used for exploring the mechanistic details underlying the biological control preferring such curvatures, as they demonstrate their advantage in recognition.
To conclude, we ﬁnd that a curved motion of the sensor is beneﬁcial for our DRC setting, offering a potential functional interpretation to similar kinematics observed in human vision (Gruber &
Ahissar, 2020; Intoy & Rucci, 2020).
D ISCUSSION
We introduced a dynamical recurrent classiﬁer (DRC), a system that recruits tiny motions of a sensor to compensate for low spatial resolution with temporal over-sampling.
This setting is novel, although some of its components had been used before (Kanazawa et al., 2021; Areﬁn et al., 2020).
We introduced recurrent dynamics to the low layers of the network (the DRC-FE in Fig.
1B), followed by time averaging and by feed forward convolutional layers.
This architecture is reminiscent of the biological brain: the dynamics in early visual areas is faster and the integration windows are shorter than in higher areas.
Therefore, an assumption of static (or slowly varying) representation in high areas is reasonable.
Furthermore, high visual areas (e.g. V4 and IT) exhibit invariance to variety of stimulus distortions (Cadieu et al., 2007; Rust & DiCarlo, 2010), which is lacking in the low areas (V1, V2).
This fact is echoed by our training method which allows the student and the teacher to have different low level representations but request similarity of representations starting at the point along processing hierarchy upstream of which a common architecture is used (DRC-BE in Fig.
1B).
Moving the sensor along the image may be considered as yet another variant of test-time data augmentation.
However, we ﬁnd that the recurrent computation provides an extra beneﬁt compared to the averaged prediction baseline (Table 1) corresponding to such an augmentation.
Importantly, the task of recognition from a series of low-resolution frames differs from the related task of MISR (Farsiu et al., 2004; Areﬁn et al., 2020).
While the latter task requires high-resolution reconstruction of the input scene, the former one does not.
The DRC does not need to learn all the particularities needed to reconstruct a high-resolution image; instead, it focuses on extracting the necessary features for the given task.
Future work may compare the performance of the DRC with standard classiﬁers that use task driven super-resolution (Haris et al., 2018) as a preprocessing step.
These solutions are applicable in a variety of settings such as body worn cameras, UAVs and self-driving cars with the need of performing real-time image recognition tasks, on a stream of visual input images captured from a moving vision sensor of limited quality, e.g. (Desai et al., 2015;
Merenda et al., 2020; Howard et al., 2017; Jiang et al., 2018).
Table 1 suggests that in a setting with
Recently, Anderson et al.
(2020) performed an approximate Bayesian inference to decode features that could account for the improvement in acuity observed in the experiments of Ratnam et al.
In our approach we assume that the primitives are shaped by the stimuli in full resolution – i.e. in the regime of convenience – rather than handcrafted, and are then adapted in a more challenging regime of low resolution.
Furthermore the inference is performed by a trained neuronal agent as opposed to the idealistic Bayesian estimate in (Anderson et al., 2020).
A teacher network assisted our DRC in developing its latent neuronal representation.
In biological terms, this would be analogous to hyperacuity being based on representations developed using regular acuity.
Consistent with this analogy are the ﬁndings that (i) the development of hyperacuity in humans follows the development of regular acuity (tested using Snellen tables) (Skoczenski & Norcia, 2002;
Wang et al., 2009) and (ii) recognizing the smallest Snellen optoypes, which improves with age (Wang et al., 2009), likely requires hyperacuity (Ratnam et al., 2017; Intoy & Rucci, 2020).
Interestingly, there is evidence that the ﬁxational drift contributes to the perception of the small Snellen optotypes (Ratnam et al., 2017; Intoy & Rucci, 2020).
The trajectory along which samples are taken affects the recognition accuracy, echoing experimental ﬁndings (Gruber & Ahissar, 2020; Intoy & Rucci, 2020).
Notably, the sensor trajectories in this work were generated independently of the underlying scene.
This is a possibly sub-optimal situation, and future work may focus on closed loop interaction between sensor trajectory and the perceived scene (Ahissar & Assa, 2016; Gruber et al., 2021).
This could also shed light on the ongoing effort to identify controllable ingredients in the ocular drift motion (Ratnam et al., 2017).
The results of this work can be used when constructing speciﬁc hypotheses about the ways in which the visual system copes with tiny images.
Speciﬁcally, our work suggests that the ocular drift plays a major role in such conditions, supporting earlier hypotheses and empirical ﬁndings (Marshall &
Talbot, 1942; Steinman & Levinson, 1990; Ahissar & Arieli, 2001; Snodderly et al., 2001; Rucci et al., 2007; Ratnam et al., 2017; Anderson et al., 2020; Gruber & Ahissar, 2020; Gruber et al., 2021; Intoy & Rucci, 2020) and that the processing of the drift-derived spatio-temporal information requires recurrent processing in retinal, sub-cortical or cortical visual networks (Burak et al., 2010).
Our results also support the inclusion of eye position signals in such processing (Burak et al., 2010); the accuracy required from such signals likely dictates that they should be derived from retinal signals (e.g., Ahissar et al.
A F EATURE V ISUALIZATION T RAINING P ROCEDURE
As a ﬁrst attempt of feature visualization training we followed the general guidelines of (Erhan et al., 2009; Nguyen et al., 2016).
The code we used was based on an online available code example 2 .
The activation of a feature i in a layer j of our network is denoted by H ij ( θ, x ) , where θ is the network’s parameters, and x is the input.
Using gradient ascent we optimize the input x to maximize the feature activation H ij .
Following Erhan et al.
(2009) x was initialized randomly and the gradient ascent algorithm was applied modifying it pixel by pixel at each iteration.
While this procedure has been shown to work with typical Convolutional Neural Network layers it failed to converge with the DRC-front-end, probably due to its higher dimensional input and complexity of the recurrent layers, (Fig. S4).
Therefore, we designed a new training procedure that includes a deep generative network (DGN) which generates the synthetic input x , and uses gradient ascent to update the DGN weights using the Adam optimizer.
The DGN architecture is shown in Table S9, the architecture was modiﬁed to enable the constraints we enforced over the system, i.e. using 3D,2D and 1D Transpose Convolution for the corresponding spatio-temporal, spatial-only and temporal-only activation maximization.
Supplementary
Activation maximization is usually a non-convex optimization problem, and therefore we do not expect to reach the same global maxima on each run.
We applied a combination of two regularization methods, Batch-normalization over the networks layer output (Ioffe & Szegedy, 2015) and added
Gaussian noise, with µ = 0 and σ = 0 .
05 empirically chosen, to obtain repeatability of AM in an individual feature, while maintaining a signiﬁcant difference between features.
We averaged the results over ten different initialization of the generator to report more consistent results; see (Fig.
S for examples of the convergence behavior of the algorithm.
B I MPLEMENTATION OF DRC TRAJECTORIES B. T RAJECTORIES WITH CONTROLLED CURVATURE
To create a family of trajectories with controllable curvature, we assumed that at each time-step the sensor location x ( t ) , y ( t ) is updated via polar increment δr, δφ .
Namely:
[Equation] [Equation]
With δφ ( t ) being i.i.d.
stochastic variables drawn from a von Mises distribution with controlled parameter κ .
Zero κ corresponds to uniform distribution of δφ ( t ) , positive values of κ correspond to straighter trajectories and negative κ corresponds to more curved ones (here we deﬁne that for κ < 0 , δφ = π + δφ (cid:48) with φ (cid:48) ∼ von Mises ( − κ ) ).
The second parameter, δr was drawn from a half-normal distribution, so that δr = r 0 + | r 1 | where r 1 ∼ N (0 , 1) and where we set r 0 = √ 2 to ensure that two consequent steps do not fall on the same point for any angle φ after rounding to integer pixel coordinates.
B. S PIRALS
Spirals were created by setting:
[Equation]
with δφ ( t ) ∼ N ( ± π 2 , π 8 ) , φ (0) drawn uniformly from circle, and the polarity of ± is ﬁxed along each individual trajectory.
The parameters r (0) = 3 , δr ∼ N ( − 0 .
16 2 ) were picked heuristically to optimize Small-net performance as well as prevent trajectories from coinciding.
Regarding coinciding and repetition of trajectories, we found that approximately 10.3K distinct trajectories were generated for a single pass over 45K large training set, making any ﬁtting to speciﬁc trajectory unlikely.
B. C URVATURE INDEX DISTRIBUTIONS
The same curvature index as in Gruber & Ahissar (2020) was used in order to compare our synthetic trajectories to biological drift motion.
The index is deﬁned as follows: CI = 1 − ds/l where l is the length of the trajectory and ds is the ﬁnal displacement (i.e. the distance between the starting and ending point of the trajectory).
Using this deﬁnition we get one index per trajectory where highly curved trajectory would result with number close to (and smaller than) 1 while low curvature trajectories would result with number close to (and greater than) 0.
Comparing the curvature index distributions obtained with κ = − 1 . 0 and κ = 0 .
(Fig.
S10) with the relevant distributions from Gruber & Ahissar (2020) (Figure 3, ’Natural-small’ curvatureindex distributions), where distribution of curvature index with κ = − 1 . 0 corresponds to ’border’ drift trajectories distribution and distribution of curvature index with κ = 0 .
corresponds to ’nonborder’ drift trajectory distribution, we see similar trends: The κ = 0 .
(’non-border’) distribution is shifted to the left and broader compared to the κ = − 1 . 0 (’border’) distribution.
The means of the distributions are also comparable and exhibit the same trend: 0 .
699 ± 0 .
002 and 0 .
597 ± 0 .
003 for κ = − 1 . 0 and κ = 0 . 0 respectively, and 0 .
65 ± 0 . 05 and 0 .
55 ± 0 .
02 for ’border’ and ’non-border’ conditions respectively.
[Table]
[Table]
B. I NTEGRATION OF POSITIONAL I NFORMATION
The positional information fed into the network contained the (x,y) normalized coordinates of the lower left corner of each frame relative to the center position of the original CiFAR image.
It was integrated into the network by ﬁrst broadcasting it to the 2d dimensions (height and width) of a single input frame (8,8,2) and then concatenating it with the frame.
resulting in a (8,8,5) input for each time-step (three dimensions for RGB and two dimensions for location).
[Table] [Table]
For ResNet+convGRU controls the positional information was fed similarly to the ﬁrst recurrent layer, with appropriate scaling (approximately square root of the layer depth)
[Table] [Table]
[Table] [Table]