R ECURRENT NEURAL CIRCUITS FOR CONTOUR DETECTION
A BSTRACT
We introduce a deep recurrent neural network architecture that approximates known visual cortical circuits (Mély et al., 2018).
We show that this architecture, which we refer to as the γ -Net, learns to solve contour detection tasks with better sample efﬁciency than state-of-the-art feedforward networks, while also exhibiting a classic perceptual illusion, known as the orientation-tilt illusion.
Correcting this illusion signiﬁcantly reduces γ -Net contour detection accuracy by driving it to prefer lowlevel edges over high-level object boundary contours.
Overall, our study suggests that the orientation-tilt illusion is a byproduct of neural circuits that help biological visual systems achieve robust and efﬁcient contour detection, and that incorporating such circuits in artiﬁcial neural networks can improve computer vision.
I NTRODUCTION
An open debate since the inception of vision science concerns why we experience visual illusions.
Consider the class of “contextual” illusions, where the perceived qualities of an image region, such as its orientation or color are biased by the qualities of surrounding image regions.
A well-studied contextual illusion is the orientation-tilt illusion depicted in Fig.
1a where perception of the central grating’s orientation is inﬂuenced by the orientation of the surrounding grating (O’Toole & Wenderoth, 1977).
When the two orientations are similar, the central grating appears tilted slightly away from the surround (Fig. 1a, top).
When the two orientations are dissimilar, the central grating appears tilted slightly towards the surround (Fig.
1a, bottom).
Is the contextual bias of the orientation-tilt illusion a bug of biology or a byproduct of optimized neural computations?
Over the past 50 years, there has been a number of neural circuit mechanisms proposed to explain individual contextual illusions (reviewed in Mély et al., 2018).
Recently, Mély et al.
(2018) proposed a cortical circuit, constrained by physiology of primate visual cortex (V1), that offers a uniﬁed explanation for contextual illusions across visual domains – from the orientation-tilt illusion to color induction.
These illusions arise in the circuit from recurrent interactions between neural populations with receptive ﬁelds that tile visual space, leading to contextual (center/surround) effects.
For the orientation-tilt illusion, neural populations encoding the surrounding grating can either suppress or facilitate the activity of neural populations encoding the central grating, leading to repulsion vs. attraction, respectively.
These surround neural populations compete to inﬂuence encodings of the central grating: suppression predominates when center/surround are similar, and facilitation predominates when center/surround are dissimilar.
The neural circuit of Mély et al.
(2018) explains how contextual illusions might emerge, but it does not explain why.
One possibility is that contextual illusions like the orientation-tilt illusion are “bugs”: vestiges of evolution or biological constraints on the neural hardware.
Another possibility is that contextual illusions are the by-product of efﬁcient neural routines for scene segmentation (Keemink & van Rossum, 2016; Mély et al., 2018).
Here, we provide computational evidence for the latter possibility and demonstrate that the orientation-tilt illusion reﬂects neural strategies optimized for object contour detection.
Contributions We introduce the γ -Net, a trainable and hierarchical extension of the neural circuit of Mély et al.
(2018), which explains contextual illusions.
(i)
The γ -Net is more sample efﬁcient than state-of-the-art convolutional architectures on two separate contour detection tasks.
(ii) Similar to humans but not state-of-the-art contour detection models, the γ -Net exhibits an orientation-tilt illusion after being optimized for contour detection.
This illusion emerges from its preference for high-level object-boundary contours over low-level edges, indicating that neural circuits involved in contextual illusions also support sample-efﬁcient solutions to contour detection tasks.
R ELATED WORK Modeling
the visual system Convolutional neural networks (CNNs) are often considered the de facto “standard model” of vision.
CNNs and their extensions represent the state of the art for most computer vision applications with performance approaching – and sometimes exceeding – human observers on certain visual recognition tasks (He et al., 2016; Lee et al., 2017; Phillips et al., 2018).
CNNs also provide the best ﬁt to rapid neural responses in the visual cortex (see Kriegeskorte 2015;
Yamins & DiCarlo 2016 for reviews).
Nevertheless, multiple lines of evidence suggest that biological vision is still far more robust and versatile than CNNs (see Serre, 2019, for a recent review).
CNNs suffer from occlusions and clutter (Fyall et al., 2017; Rosenfeld et al., 2018;
Tang et al., 2018).
They are also sample inefﬁcient at learning visual relations (Kim et al., 2018) and solving simple grouping tasks (Linsley et al., 2018c).
State-of-the-art CNNs require massive datasets to reach their impressive accuracy (Lake et al., 2015) and their ability to generalize beyond training data is limited (Geirhos et al., 2018; Recht et al., 2018).
Cortical feedback contributes to the robustness of biological vision (Hochstein & Ahissar, 2002; Wyatte et al., 2014; Kafaligonul et al., 2015).
Feedforward projections in the visual system are almost always matched by feedback projections (Felleman & Van Essen, 1991), and feedback has been implicated in visual “routines” that cannot be implemented through purely feedforward vision, such as incremental grouping or ﬁlling-in (O’Reilly et al., 2013; Roelfsema, 2006).
There is a also a growing body of work demonstrating the potential of recurrent neural networks (RNNs) to account for neural recordings (Fyall et al., 2017; Klink et al., 2017; Siegel et al., 2015; Tang et al., 2018;
Nayebi et al., 2018; Kar et al., 2019; Kietzmann et al., 2019).
Feedback for computer vision In contrast to CNNs, which build processing depth through a cascade of ﬁltering and pooling stages with unique weights, RNNs process stimuli with ﬁltering stages that reuse weights over “timesteps” of recurrence. On each discrete processing timestep, an
RNN updates its hidden state through a nonlinear combination of an input and its the hidden state from its previous timestep. RNNs have been extended from their roots in sequence processing (e.g., Mozer 1992) to computer vision by computing the activity of RNN units through convolutional kernels.
The common interpretation of these convolutional-RNNs, is that the input to each layer functions as a (ﬁxed) feedforward drive, which is combined with layer-speciﬁc feedback from an evolving hidden state to dynamically adjust layer activity (Linsley et al., 2018c; George et al., 2017; Lotter et al., 2016; Wen et al., 2018; Liao & Poggio, 2016; Spoerer et al., 2017;
Nayebi et al., 2018; Tang et al., 2018). In the current work, we are motivated by a similar convolutional-RNN, the horizontal gated recurrent unit (hGRU, Linsley et al. 2018a), which approximates the recurrent neural circuit model of (Mély et al., 2018) for explaining contextual illusions.
R ECURRENT NEURAL MODELS
We begin by reviewing the dynamical neural circuit of Mély et al. This model explains contextual illusions by simulating interactions between cortical hypercolumns tiling the visual ﬁeld (where hypercolumns describe a set of neurons encoding features for multiple visual domains at a single retinotopic position). In the model, hypercolumns are indexed by their 2D coordinate ( x, y ) and feature channels k . Units in hypercolumns encode idealized responses for a visual domain (e.g., neural responses from the orientation domain were used to simulate an orientation-tilt illusion;
[Figure]
(a) (a)
[Figure]
(a) (a)
[Figure]
illusion where a central grating’s perceived orientation is inﬂuenced by a surround grating’s orientation . (a) When a central grating has a similar orientation as its surround, it is judged as tilting away from the surround (repulsion). When the two gratings have dissimilar orientations, the central grating is judged as tilting towards the surround (attraction). (b) We extend the recurrent circuit proposed by Mély et al.
(2018) to explain this and other contextual illusions into a hierarchical model that learns horizontal (within a layer) and top-down (between layer) interactions between units. The circuit simulates dynamical suppressive ( H ( S ) xyk ) and facilitative ( H ( F ) xyk ) interactions between units in a layer (cid:96) , which receives feedforward drive from a center pathway encoding feature k (e.g., edges oriented at 0 ◦ or 22.5 ◦ ) at position (x, y) in an image. Blocks depict different layers, and arrowed connections denote top-down feedback.
(c) A deep network schematic of the circuit diagram in (b), which forms the basis of the γ -Net introduced here.
Horizontal and top-down connections are implemented with feedback gated recurrent units (fGRUs). Image encodings pass through these blocks on every timestep, from bottom-up (left path) to top-down (right path), and predictions are read out from the fGRU closest to image resolution on the ﬁnal timestep. This motif can be stacked to create a hierarchical model. Fig. 1b).
Dynamics of a single unit at xyk obey the following equations (we bold activity tensors to distinguish them from learned kernels and parameters):
[Equation]
facilitation of H ( S ) where [Equation] # Compute suppression interactions
[Equation]
# Compute facilitation interactions
Circuit activities consist of a feedforward drive, recurrent suppression, and recurrent facilitation, respectively denoted as Z , H ( S ) ,
H ( F ) ∈ R X
× Y × K ( X is width, Y is height of the tensor, and K is its feature channels) ∗ .
The circuit takes its “feedforward” input Z from hypercolumns (e.g., orientation encodings from hypercolumn units), and introduces recurrent suppressive and facilitatory interactions between units, C ( S )
, C ( F ) ∈ R X × Y × K (Fig. 1b).
These interactions are implemented with separate kernels for suppression and facilitation, W S , W F ∈ R E × E × K × K , where E is the spatial extent of connections on a single timestep (connectivity in this model is constrained by primate physiology).
These interactions are implemented through convolutions, allowing them to serially spread over timesteps of processing to connect units positioned at different spatial locations.
The circuit outputs
H ( F ) after reaching steady state.
The circuit model of Mély et al.
(2018) depends on competition between H ( S ) and H ( F ) to explain the orientation-tilt illusion.
Competition is implemented by (i) computing suppression vs. facilitation in separate stages, and (ii) having non-negative activities, which enforces these functionally distinct processing stages.
With these constraints in the circuit model, the strength of recurrent suppression – but not facilitation – multiplicatively increases with the net recurrent output.
For the orientation-tilt illusion, suppression predominates when center and surround gratings have similar orientations. This causes encodings of the surround grating to “repulse” encodings of the center grating. On the other hand, facilitation predominates (causing “attraction”) when center and surround gratings have dissimilar orientations because it is additive and not directly scaled by the circuit output.
Parameters controlling the circuit’s integration, suppression/facilitation, and patterns of horizontal connections between units are tuned by hand.
Linear and multiplicative suppression (i.e., shunting inhibition) are controlled by scalars µ and α , feedforward drive is modulated by the scalar ξ , and linear facilitation is controlled by the scalar ν .
Circuit time constants are scalars denoted by η , (cid:15) , τ and σ .
All activities are non-negative and both stages are linearly rectiﬁed (ReLU) [ · ] + = max( · , 0) .
Feedback gated recurrent units Linsley et al.
(2018a) developed a version of this circuit for computer vision applications, called the hGRU.
In their formulation they use gradient descent (rather than hand-tuning like in the original circuit) to ﬁt its connectivity and parameters to image datasets.
The hGRU was designed to learn a difﬁcult synthetic incremental grouping task, and a single layer of the hGRU learned long-range spatial dependencies that CNNs with orders-of-magnitude more weights could not.
The hGRU replaced the circuit’s time constants with dynamic gates, converted the recurrent state H ( S ) for suppression into an instantaneous activity, and introduced a term for quadratic facilitation.
The hGRU also relaxed biological constraints from the original circuit, including an assumption of non-negativity, which enforced competition between recurrent suppression vs. facilitation (e.g., guaranteeing that Stage 1 in the circuit model describes suppression of Z xyk ).
We extend the hGRU formulation in two important ways.
First, like Mély et al.
(2018), we introduce non-negativity.
This constraint was critical for Mély et al.
(2018) to explain contextual illusions, and as we describe below, was also important for our model.
Second, we extend the circuit into a hierarchical model which can learn complex contour detection tasks.
Recent neurophysiological work indicates that contextual effects emerge from both horizontal and top-down feedback (Chettih & Harvey, 2019).
Motivated by this, we develop versions of the circuit to simulate horizontal connections between units within a layer, and top-down connections between units in different layers.
We call our module the feedback gated recurrent unit (fGRU).
We describe the evolution of fGRU recurrent units in H ∈ R X
× Y × K , which are inﬂuenced by non-negative feedforward encodings
Z ∈ R X × Y × K (e.g., a convolutional layer’s response to a stimulus) over discrete timesteps · [ t ] :
[Equation]
# Compute channel-wise selection
[Equation]
# Compute suppression interactions
[Equation]
Stage 2:
[Equation]
# Compute channel-wise recurrent updates
[Equation]
# Compute facilitation interactions ˜ H
= (cid:104) ν
( C F + S ) + ω ( C F ∗ S ) (cid:105) + # Facilitation of S
H [ t ] =
(1 − G F ) (cid:12) H [ t − 1] + G F (cid:12)
˜ H .
# Update recurrent state
Like the original circuit, the fGRU has separate stages for suppression ( S ) and facilitation ( H ).
In the ﬁrst stage, the feedforward encodings Z are suppressed by non-negative interactions between units in
H [ t − 1] (an fGRU hidden state from the previous timestep).
Suppressive interactions are computed with the kernel W S ∈ R E × E × K × K , where E describes the spatial extent of horizontal connections on a single timestep.
This kernel is convolved with a gated version of the persistent hidden state H [ t − 1] .
The gate activity G S is computed by applying a sigmoid nonlinearity to a convolution of the kernel U S ∈ R 1 × 1 × K × K with H [ t − 1] , which transforms its activity into the range [0 , 1] .
Additive and multiplicative forms of suppression are controlled by the parameters µ, α ∈ R K , respectively.
In the second stage, additive and multiplicative facilitation is applied to the instantaneous activity S .
The kernels W F ∈ R E
×
E
×
K × K controls facilitation interactions.
Additive and multiplicative forms of facilitation are scaled by the parameters ν, ω ∈ R K , respectively.
A gate activity is also computed during this stage to update the persistent recurrent activity H .
The gate activity G F is computed by applying a sigmoid to a convolution of the kernel U F ∈ R 1 × 1 × K × K with S .
This gate updates H [ t ] by interpolating H [ t − 1] with the candidate activity ˜ H .
After every timestep of processing, H [ t ] is taken as the fGRU output activity.
As detailed in the following section, the fGRU output hidden state is either passed to the next convolutional layer (Fig. 1c, fGRU ( (cid:96) ) −→ conv ( (cid:96) +1) ), or used to compute top-down connections (Fig. 1c, fGRU ( (cid:96) +1) −→ fGRU
(cid:96) ) ).
The fGRU has different conﬁgurations for learning horizontal connections between units within a layer or top-down connections between layers (Fig. 1b).
These two conﬁgurations stem from changing the activities used for a fGRU’s feedforward encodings and recurrent hidden state.
“Horizontal connections” between units within a layer are learned by setting the feedforward encodings Z to the activity of a preceding convolutional layer, and setting the hidden state H to a persistent activity initialized as zeros (Fig. 1c, conv ( (cid:96) )
−→ fGRU ( (cid:96) ) ).
“Top-down connections” between layers are learned by setting fGRU feedforward encodings Z to the persistent hidden state H ( (cid:96) ) of a fGRU at layer (cid:96) in a hierarchical model, and the hidden state H to the persistent activity H ( (cid:96) +1) of an fGRU at a layer one level up in the model hierarchy (Fig. 1c, fGRU ( (cid:96) +1) −→ fGRU ( (cid:96) ) ).
The functional interpretation of the top-down fGRU is that it ﬁrst suppresses activity in the lower layer using the higher layer’s recurrent horizontal activities, and then applies a kernel to the residue for facilitation, which allows for computations like interpolation, sharpening, or “ﬁlling in”.
Note that an fGRU for top-down connections does not have a persistent state (it mixes high and low-level persistent states), but an fGRU for horizontal connections does.
γ -Net
Our main objective is to test how a model with the capacity for contextual illusions performs on natural image analysis.
We do this by incorporating fGRUs into leading feedforward architectures for contour detection tasks, augmenting their feedforward processing with modules for learning feedback from horizontal and top-down connections (Fig. 1c).
We refer to the resulting hierarchical models as γ -Nets, because information ﬂows in a loop that resembles a γ : image encodings make a full bottom-up to top-down cycle through the architecture on every timestep, until dense predictions are read-out from the lowest-level recurrent layer of the network (thus, information ﬂows in at the top of the hierarchy, loops through the network, and ﬂows out from the top of the hierarchy).
In our experiments we convert leading architectures for two contour detection problems into γ -Nets:
A
VGG16 for BSDS500 (He et al., 2019), and a U-Net for detection of cell membranes in serial electron microscopy images (Lee et al., 2017).
See Appendix A for an algorithmic description of γ -net.
C ONTOUR DETECTION EXPERIMENTS
Overview We evaluated γ -Net performance on two contour detection tasks: object contour detection in natural images (BSDS500 dataset; Arbeláez et al., 2011) and cell membrane detection in serial electron microscopy (SEM) images of mouse cortex (Kasthuri et al., 2015) and mouse retina (Ding et al., 2016).
Different γ -Net conﬁgurations were used on each task, with each building on the leading architecture for their respective datasets.
All γ -Nets use 8-timesteps of recurrence and instance normalization (normalization controls vanishing gradients in RNN training; Ulyanov et al., 2016;
Cooijmans et al., 2017, see Appendix A for details).
The γ -Nets were trained with Tensorﬂow and
NVIDIA Titan RTX GPUs using single-image batches and the Adam optimizer (Kingma & Ba, 2014, dataset-speciﬁc learning rates are detailed below).
Models were trained with early stopping, which terminated training if the validation loss did not drop for 50 straight epochs. The weights with the best validation-set performance were used for testing.
Model evaluation We evaluated models in two ways.
First, we validated them against state-of-theart models for each contour dataset using standard benchmarks.
As discussed below, we veriﬁed that our implementations of these state-of-the-art models matched published performance.
Second, we tested sample-efﬁciency after training on subsets of the contour datasets without augmentations.
Sample-efﬁciency compares the inductive biases of different architectures, and is critical for understanding how the capacity for exhibiting contextual illusions inﬂuences performance.
We report model “wall time” in Appendix A; however, the focus of our work is on sample efﬁciency rather than the hardware/software-level optimizations that inﬂuence wall time.
Model performance is evaluated as the F-measure at the Optimal Dataset Scale across images after non-maximum suppression post-processing (F1-ODS; Arbeláez et al., 2011), as is standard for contour detection tasks.
4. O BJECT CONTOUR DETECTION IN NATURAL IMAGES
Dataset We trained models for object contour detection on the BSDS500 dataset (Arbeláez et al., 2011).
The dataset contains object-contour annotations for 500 natural images, which are split into train (200), validation (100), and test (200) sets.
Architecture details The leading approach to BSDS500 is the Bi-Directional Cascade Network (BDCN,
He et al. 2019), which places multi-layer readout modules at every processing block in a
ILSVRC12-pretrained VGG16, and optimizes a loss that balances contributions from each of these readouts to achieve better scale tolerance in ﬁnal prediction.
All leading deep learning approaches to
BSDS500 begin with a VGG16 pretrained on ILSVRC12 object recognition (He et al., 2019).
Our γ -Net for BSDS500 begins with the same ILSVRC12-pretrained VGG16 as the BDCN.
fGRUs were introduced for learning horizontal (conv2_2, conv3_3, conv4_3, conv5_3) and top-down connections (conv5_3 −→ conv4_3, conv4_3 −→ conv3_3, and conv3_3 −→ conv2_2).
To pass topdown activities between layers, higher-level activities were resized to match lower-level ones, then passed through two layers of 1 × 1 convolutions with linear rectiﬁcation, which registered feature representations from higher-to-lower layers.
The γ -Net was trained with learning rates of 3 e − 4 on its randomly initialized fGRU weights and 1 e − 5 on its VGG-initialized weights. Training time and parameter counts for this γ -Net
and the BDCN is in Appendix Table S1.
In contrast to the BDCN (and other recent approaches to BSDS) with multiple read-outs and engineered loss functions, we take γ -Net predictions as a linear transformation of the lowest fGRU layer in its feature hierarchy, and optimize the model with binary cross entropy between per-pixel predictions and labels (following the method of Xie & Tu, 2017).
This approach works because the γ -Net uses feedback to merge high- and low-level image feature representations at the bottom of its feature hierarchy, resembling classic “V1 scratchpad” hypotheses for computation in visual cortex (Gilbert & Sigman, 2007; Lee & Mumford, 2003).
We compared γ -Nets with a BDCN implementation released by the authors, which was trained using the routine described in He et al.
Results We validated -Net BDCN on a and training set (Xie & Tu, 2017).
The γ -Net performed similarly in F1 ODS (0.802) as the BDCN (0.806) and humans (0.803), and outperformed all other approaches to BSDS (Fig.
2a; Deng et al.
; Xie & Tu 2017; Hallman & Fowlkes 2015; Kokkinos 2015; Wang et al. 2019; Liu et al. 2019).
Our hypothesis is that contextual illusions reﬂect routines for efﬁcient scene analysis, and that the capacity for exhibiting such illusions improves model sample efﬁciency.
Consistent with this hypothesis, the γ -Net was up to an order-of-magnitude more efﬁcient than the BDCN.
A γ -Net trained on 5% of BSDS performs on par with a BDCN trained on 10% of the BSDS, and a γ -Net trained on 10% of the BSDS performs on par with a BDCN trained on 100% of BSDS.
Unlike the BDCN, the γ -Net trained on 100% of BSDS outperformed the state of the art for non-deep learning based models (Hallman & Fowlkes, 2015), and nearly matched the performance of the popular HED trained with augmentations (Xie & Tu, 2017).
We also evaluated lesioned versions of γ -Net to measure the importance of its horizontal/top-down connections, recurrence, non-negativity, and different speciﬁcations of its fGRU recurrent modules for detecting contours in BSDS500 (Fig. S4).
We examined recurrent feedback strategies learned by γ -Net for object contour by visualizing its performance on every timestep of processing.
This was done by passing its activity at a timestep through the ﬁnal linear readout layer.
The γ -Net iteratively reﬁnes its initially coarse contour predictions.
For example, the top row of Fig.
2c shows that the γ -Net selectively enhances the boundaries around the runner’s bodies while suppressing the feature activities created by the crowd.
In the next row of predictions, salient zebra stripes are gradually suppressed in favor of body contours (see Fig.
S5 for γ -Net prediction dynamics and its tendency towards steady state solutions).
4. C ELL MEMBRANE DETECTION
Datasets “Connectomics” involves extracting the wiring diagrams of neurons from serial electron microscope (SEM) imaging data, and is an important step towards understanding the algorithms of brains (Briggman & Bock, 2012).
CNNs can automate this procedure by segmenting neuron membranes in SEM images.
Large-scale challenges like SNEMI3D (Kasthuri et al., 2015), which contains annotated images of mouse cortex, have helped drive progress towards automation.
Here, we test models on membrane detection in SNEMI3D and a separate SEM dataset of mouse retina (“Ding” from Ding et al. 2016).
We split both datasets into training (80 images for SNEMI3D and 307 images for Ding) and test sets (20 images for SNEMI3D and 77 images for Ding).
Next, we generated versions of each training dataset with 100%, 10%, or 5% of the images, as well as versions of the full datasets augmented with random left-right and up-down ﬂips (A+100%).
Architecture details The state-of-the-art on SNEMI3D is a U-Net variant (Ronneberger et al., 2015), which uses a different depth, different number of feature maps at every layer, and introduces new features like residual connections (Lee et al., 2017).
We developed a γ -Net for cell membrane segmentation, which resembled this U-Net variant Lee et al.
(2017) (see Appendix B for details).
These γ -Net were trained from a random initialization with a learning rate of 1 e − 2 to minimize class-balanced per-pixel binary cross-entropy, and compared to the U-Net from Lee et al.
Training time and parameter counts for these models is in Appendix Table S2.
Results
The γ -Net and U-Net of Lee et al.
(2017) performed similarly when trained on full augmented versions of both the SNEMI3D and Ding datasets (Fig. 3a A+100%).
However, γ -Nets were consistently more sample efﬁcient than U-Nets on every reduced dataset condition (Fig. 3b).
We visualized the recurrent membrane detection strategies of γ -Nets trained on 100% of both datasets.
Membrane predictions were obtained by passing neural activity at every timestep through the ﬁnal linear readout.
The γ -Net prediction timecourse indicates that it learns a complex visual strategy for membrane detection: it gathers a coarse “gist” of membranes in the ﬁrst timestep of processing, and iteratively reﬁnes these predictions by enhancing cell boundaries and clearing out spurious contours of elements like cell organelles (Fig. 3c).
B UGS OR BYPRODUCTS OF OPTIMIZED NEURAL COMPUTATIONS ?
Orientation-tilt illusion Like the neural circuit of Mély et al.
, fGRU modules are designed with an asymmetry in their ability to suppress and facilitate feedforward input (see Section 3).
This potentially gives γ -Nets (which contain fGRU modules) the capacity to exhibit similar contextual illusions as humans.
Here, we tested whether a γ -Net trained on contour detection in natural images exhibits an orientation-tilt illusion.
We tested for this illusion by training orientation decoders on the outputs of models trained on the full BSDS500 dataset.
These decoders were trained on 100K grating images, in which the center and surround orientations were the same (Fig. S2a).
These images were sampled from all orientations and spatial frequencies.
The decoders had two 1 × 1 convolution layers and an intervening linear rectiﬁcation to map model outputs into the sine and cosine of grating orientation.
Both the γ -Net and
BDCN achieved nearly perfect performance on a held-out validation set of gratings.
We tested these models on 1K grating stimuli generated with different center-surround grating orientations (following the method of O’Toole & Wenderoth 1977, Fig.
S2b) and recorded model predictions for the center pixel in these images (detailed in Appendix C).
Surprisingly, γ -Net encodings of these test images exhibited a similar tilt illusion as found in human perceptual data (Fig. 4b). There was repulsion when the central and surround gratings had similar orientations, and attraction when these gratings were dissimilar.
This illusory phenomenon cannot be explained by accidental factors such as aliasing between the center and the surround, which would predict the opposite pattern, suggesting that the illusion emerges from the model’s strategy for contour detection.
In contrast, the BDCN, which only relies on feedforward processing, did not exhibit the effect (Fig. 4b).
We also tested lesioned γ -Nets for the orientation-tilt illusion, but only the full γ -Net and a version lesioned to have only (spatially broad) horizontal connections replicated the entire illusion (thought this lesioned model performed worse on contour detection than the full model; Fig.
S4).
These ﬁndings are consistent with the work of Mély et al. (2018), who used spatially broad horizontal connections to model contextual illusions, as well as the more recent neurophysiological work of Chettih & Harvey (2019), who explained contextual effects through both horizontal and top-down interactions.
Correcting the orientation-tilt illusion What visual strategies does the orientation-tilt illusion reﬂect?
We tested this question by taking a γ -Net that was trained to decode central grating orientation in tilt-illusion, and then training it further to decode the central grating orientation of tilt-illusion stimuli (Fig.
5a, “illusion-corrected” in red).
Importantly, γ -Net weights were optimized during training, but the orientation decoder was not.
Thus, improving performance for decoding the orientation of these illusory stimuli comes at the expense of changing γ -Net weights that were
[Figure]
I ll u s i o n - c o rr ec t e d F O D S
I m a g e
L a b e l Δ [Figure] responsible for its orientation-tilt illusion. As a control, another γ -Net was trained with the same routine to decode the orientation of full-image gratings for which there is no illusion (Fig.
5a, blue; see Fig. S6 for performance of both models)
Correcting the orientation-tilt illusion of a γ -Net signiﬁcantly hurts its object contour detection performance (Fig.
5a; 1-sample T -test of the per-image ODS F1 difference between models,
T (199) = 13.570, p < 0.001).
The illusion reﬂects γ -Net strategies for selecting object-boundaries rather than low-level contours (Fig. 5b; Fig.
S7 for more examples).
C ONCLUSION
Why do we experience visual illusions?
Our experiments indicate that one representative contextual illusion, the orientation-tilt illusion, is a consequence of neural strategies for efﬁcient scene segmentation.
We directly tested whether this contextual illusion is a bug or a byproduct of optimized neural computations using the γ -net: a dense prediction model with recurrent dynamics inspired by neural circuits in visual cortex. On separate contour detection tasks, the γ -Net performed on par with state-of-the-art models when trained in typical regimes with full augmented datasets, but was far more efﬁcient than these models when trained on sample-limited versions of the same datasets.
At the same time, the γ -Net exhibited an orientation-tilt illusion which biased it towards high-level object-boundary contours over low-level edges, and its performance was reduced when it was trained to correct its illusion.
While γ -Nets are more sample efﬁcient than leading feedforward models for contour detection, they also take much more “wall-time” to train than these feedforward models.
Learning algorithms and GPU optimizations for RNNs are lagging behind their feedforward counterparts in computational efﬁciency, raising the need for more efﬁcient approaches for training hierarchical RNNs like γ -Nets to unlock their full potential.
More generally, our work demonstrates novel synergy between artiﬁcial vision and vision neuroscience: we demonstrated that circuit-level insights from biology can improve the sample efﬁciency of deep learning models.
The neural circuit that inspired the fGRU module explained biological illusions in color, motion, and depth processing (Mély et al., 2018), and we suspect that γ -Nets will have similar success in learning sample-efﬁcient strategies – and exhibiting contextual illusions – in these domains.
A γ -N ET
{+Require: Image batch I 1: Z (cid:96) ← I 2: H 
 [ t ] ←3: for t+} = T do (cid:46) Recurrent loops over static images 4: for (cid:96) = 0 to L do (cid:46) Bottom-up pass 5: Z (cid:96) =
ReLU(Conv( Z (cid:96) ))
(cid:46) Feedforward activity 6: H (cid:96) [ t ] = fGRU
H ( Z = Z (cid:96) , H = H (cid:96) [ t − 1])
(cid:46) Update horizontal feedback state 7: if (cid:96) <
L then 8: Z (cid:96) +
=
Maxpool( H (cid:96) [ t ]) 9: for (cid:96) =
L to 0 do (cid:46) Top-down feedback pass 10: if (cid:96) <
L then 11:
H (cid:96) [ t ] = fGRU TD ( Z = H (cid:96) [ t ] , H =
ReLU(Conv(Upsample( H (cid:96) +
[ t ]))))
12: return H
[ t ] (cid:46)
Hidden state of layer (cid:96)
Connectomics The current standard for computer vision applications in connectomics is to train and test on separate partitions of the same tissue volume (Linsley et al., 2018b; Januszewski & Jain, 2019).
This makes it difﬁcult to develop new model architectures without overﬁtting to any particular dataset.
For this reason, we ﬁrst tuned our connectomics γ -Net and its hyperparameters on a synthetic dataset of cell images (data not shown).
In our experiments on synthetic data, we noted monotonically improved performance with increasing timesteps, which motivated our choice of building these models with as many timesteps as could ﬁt into GPU memory without sacriﬁcing (we carried this concept over to the design of our γ -Nets for
BSDS).
Thus, we settled on 8 timesteps for the γ -Nets .
We also compared our use of fGRU modules to learn recurrent connection vs. the classic LSTM and GRU recurrent modules, and found that the γ -Netwas far more effective on small datasets, which we take as evidence that its recurrent application of suppression separately from facilitation is a better inductive bias for learning contour tasks (see Hahnloser et al.
for a theoretical discussion on how these operations can amount to a digital selection of task-relevant features through inhibition, followed by an analog ampliﬁcation of the residuals through excitation).
We found that γ -Net cell membrane detection was improved when every bottom-up unit (from a typical convolution) was given a hidden state.
Like with gated recurrent architectures, these gates enable gradients to effectively skip timesteps of processing where they pathologically decay.
We do this by converting every convolutional layer (except the ﬁrst and last) into a “minimal gated unit” (Heck & Salem 2017).
This conversion introduced two additional kernels to each convolutional layer,
U F , W H ∈ R 1 × 1 × K × K , where the former was responsible for selecting channels from a persistent activity H ∈
R X × Y × K for processing on a given timestep and updating the persistent activity.
The latter kernel transformed a modulated version of the hidden state H .
This transformed hidden state was combined with a vanilla convolutional feedforward encoding, Z ∈ R X × Y × K
(see Eq.
1 for the treatment).
Weights in these layers were initialized with orthogonal random initializations, which help training recurrent networks (Vorontsov et al., 2017).
[Equation]
fGRU
Here we describe additional details of the fGRU.
fGRU kernels for computing suppressive and facilitative interactions have symmetric weights between channels, similar to the original circuit of Mély et al.[-This means that the weight W x 0 +∆ x,y 
 +∆ y,k 1 ,k 2 is equal to the weight W x 0 +∆ x,y 
 +∆ y,k 2 ,k 1 , where x 0 and y 0 denote kernel center.-]
This means that {+the weight W x 0 +∆ x,y 
+}
{++∆ y,k 1 ,k 2 is equal to the weight W x 0 +∆ x,y 
+}
+∆ y,k 2 ,k 1 , where x 0 and y 0 denote kernel center.
This constraint means that there are nearly half as many learnable connections as a normal convolutional kernel.
In our experiments, this constraint improved performance.
[Table]
[Table]
While optimizing γ -Nets on synthetic cell image datasets, we found that a small modiﬁcation of the fGRU input gate offered a modest improvement in performance.
We realized that the input gate in the fGRU is conceptually similar to recently developed models for feedforward self-attention in deep neural networks.
Speciﬁcally, the global-and-local attention modules of (Linsley et al., 2019), in which a non-linear transformation of a layer’s activity is used to modulate the original activity.
Here, we took inspiration from global-and-local attention, and introduced an additional gate into the fGRU, resulting in the following modiﬁcation of the main equations.
Stage 1:
[Equation]
# Compute channel-wise selection
[Equation]
# Compute spatial selection
[Equation]
# Compute suppression gate
[Equation]
# Compute suppression interactions
[Equation]
+
[Equation]
# Additive and multiplicative suppression of Z
Stage 2:
[Equation]
# Compute channel-wise recurrent updates
[Equation]
# Compute facilitation interactions
[Equation]
# Additive and multiplicative facilitation of S
[Equation]
# Update recurrent state where IN ( r ; δ, ν )
=
ν + δ (cid:12) r − (cid:98) E
[ r ] (cid:113)(cid:100) Var
[ r ] + η .
This yields the global input gate activity A S ∈ R X
× Y × K and the local input gate activity M S
∗ ∈ R X
× Y ×, which are computed as ﬁlter responses between the previous hidden state H [ t − 1] and the global gate kernel U A ∈ R 1 × 1 × K × K and the local gate kernel U M ∈ R 3 × 3 × K × 1 .
Note that the latter ﬁlter is learning a mapping into 1 dimension and is therefore ﬁrst tiled into K dimensions, yielding
M S ∗ , before elementwise multiplication with A S .
All results in the main text use this implementation.
Following the lead of (Linsley et al., 2018a), we incorporated normalizations into the fGRU.
Let r ∈ R
d
denote the vector of layer activations that will be normalized.
We chose instance normalization (Ulyanov et al., 2016) since it is independent of batch size, which was 1 for γ -Nets in our experiments.
Instance normalization introduces two k -dimensional learned parameters, δ , ν ∈ R d , which control the scale and bias of normalized activities, and are are shared across timesteps of processing.
In contrast, means and variances are computed on every timestep, since fGRU activities are not i.i.d.
across timesteps.
Elementwise multiplication is denoted by (cid:12) and η is a regularization hyperparameter.
Learnable gates, such as those in the fGRU, are helpful for training RNNs.
But there are other heuristics that are also important for optimizing performance.
We use several of these with γ -Nets, such as Chronos initialization of fGRU gate biases (Tallec & Ollivier, 2018) and random orthogonal initialization of kernels (Vorontsov et al., 2017).
We initialized the learnable scale parameter δ of fGRU normalizations to 0.1, since values near 0 optimize the dynamic range of gradients passing through its sigmoidal gates (Cooijmans et al., 2017).
Similarly, fGRU parameters for learning additive suppression/facilitation ( µ, ν ) were initialized to 0, and parameters for learning multiplicative inhibition/excitation ( α, ω ) were initialized to 0.1.
Finally, when implementing top-down connections, we incorporated an extra skip connection.
The activity of layer (cid:96) was added to the fGRU-computed top-down interactions between layer (cid:96) and layer (cid:96) + 1 .
This additional skip connection improved the stability of training.
here derived by watershedding agglomeration with GALA (Nunez-Iglesias et al., 2013), resulting in “superhuman” ARAND (evaluated according to the SNEMI3D standard; lower is better) of 0.04, which is below the reported human-performance threshold of 0.06 and on par with the published result (see Table 1 in Lee et al mean afﬁnity agglomeration)
B M EMBRANE PREDICTION MODELS
Our reference model for membrane prediction is the 3D U-Net of (Lee et al., 2017).
This architecture consists of four encoder blocks (multiple convolutions and skip connections, pooling and subsampling), followed by four decoder blocks (transpose convolution and convolution). This U-Net uses spatial pooling between each of its encoder blocks to downsample the input, and transpose convolutions between each of its decoder blocks to upsample intermediate activities. We validated our implementation of this U-Net following the author’s training routine, and were able to replicate their reported “superhuman” performance in cell segmentation on SNEMI3D (Fig. S1). The γ -Net for connectomics resembles the U-Net architecture of Lee et al.
This γ -Net replaces the blocks of convolutions and skip connections of that model with a single layer of convolution followed by an fGRU (as in the high level diagram of Fig. 1c, Conv ( (cid:96) ) , −→ fGRU ( (cid:96) ) ). In the encoder pathway, fGRUs store horizontal interactions between spatially neighboring units of the preceding convolutional layer in their hidden states. In the decoder pathway, γ -Net introduces fGRUs that learn top-down connections between layers, and connect recurrent units from higher-feature processing layers to lower-feature processing ones.
[Table]
Key to the approach of Lee et al.
(2017) is their use of a large set of random data augmentations applied to SEM image volumes, which simulate common noise and errors in SEM imaging.
These are (i) misalignment between consecutive z -locations in each input image volume.
(ii) Partial- or fully-missing sections of the input image volumes.
(iii) Blurring of portions of the image volume.
Augmentations that simulated these types of noise, as well as random ﬂips over the xyz -plane, rotations by 90 ◦ , brightness and contrast perturbations, were applied to volumes following the settings of Lee et al.
The model was trained using Adam (Kingma & Ba, 2014) and the learning rate schedule of Lee et al.
(2017), in which the optimizer step-size was halved when validation loss stopped decreasing (up to four times).
Training involved single-SEM volume batches of 160 × 160 × 18 (X/Y/Z), normalized to [0 , 1] .
As in Lee et al.
(2017), models were trained to predict nearest-neighbor voxel afﬁnities, as well as 3 other mid- to long-range voxel distances.
Only nearest neighbor afﬁnities were used at test time.
C O RIENTATION - TILT ILLUSION IMAGE DATASET
Models were tested for a tilt illusion by ﬁrst training on grating images of a single orientation, then testing on images in which a center grating had the same/different orientation as a surround grating.
Each image in the training dataset consisted of a circular patch of oriented grating on a gray canvas of size 500 × 500 pixels.
To ensure that the decoder successfully decoded orientation information from model activities, the training dataset incorporated a wide variety of grating stimuli with 4 randomly sampled image parameters: r , λ , θ , and φ .
r denotes the radius of the circle in which oriented grating has been rendered, and was sampled from a uniform distribution with interval between 80 andpixels; λ speciﬁes the wavelength of the grating pattern and was sampled from a uniform distribution with interval between 30 and 90 pixels; θ speciﬁes the orientation of the gratings and is uniformly sampled from all possible orientations; φ denotes the phase offset of the oriented gratings and is also uniformly sampled from all possible values.
The models’ BSDS-trained weights were ﬁxed and readout layers were trained to decode orientation at the center of each image (procedure described in the main text).
This setup allowed us to tease apart the effects of the surround on the representation of orientation in the center by introducing separate surround regions in each test image ﬁlled with gratings with same/different orientations as the center (Fig.
S2b).
Each test image was generated with one additional parameter, ∆ θ which speciﬁed orientation difference of the surround gratings with respect to the center orientation, θ , and was sampled from a uniform distribution with interval between − 90 and + 90 degrees.
The radius of the surround grating is denoted by r and was sampled from the same uniform distribution we used in training dataset.
Center gratings are then rendered in a circle of radius that is one half of the surround gratings.
[Figure]
towards steady state by the end of its processing timecourse.
[Figure] [Figure]