Task-Driven Learning of Contour Integration Responses in a V1 Model
Abstract
Under difﬁcult viewing conditions, the brain’s visual system uses a variety of modulatory techniques to augment its core feed-forward signals.
Incorporating these into artiﬁcial neural networks can potentially improve their robustness.
However, before such mechanisms can be recommended, they need to be fully understood.
Here, we present a biologically plausible model of one such mechanism, contour integration, embedded in a task-driven artiﬁcial neural network.
The model is neuroanatomically realistic and all its connections can be mapped onto existing connections in the V1 cortex.
We ﬁnd that the model learns to integrate contours from high-level tasks including those involving natural images.
Trained models exhibited several observed neurophysiological and behavioral properties.
In contrast, a parameter matched feed-forward control achieved comparable task-level performance but was largely inconsistent with neurophysiological data.
Introduction
Contour integration [1, 2, 3, 4] is a phenomenon in the V1 cortex where stimuli from outside a neuron’s classical receptive ﬁeld (cRF) modulate its feed-forward responses.
In particular, responses are enhanced if a preferred stimulus within the cRF is part of a larger contour.
Under difﬁcult viewing conditions, the visual ventral stream uses contour integration to pop out smooth natural contours (see Figure 2).
Contour integration is thought to be mediated by intra-area lateral and higher-layer feedback connections.
Past computational models [5, 6] have tested potential mechanisms in isolation and replicated observed neurophysiological data.
However, apart from some recent work [7, 8], past studies have done little to explore the role of contour integration in the perception of naturalistic scenes.
Separately, recent advances in deep neural networks have surpassed human-level performance on several high-level vision tasks such as object classiﬁcation [9, 10, 11] and object segmentation [12, 13].
This enables the possibility of embedding a contour integration model within a task-oriented system to quantitatively analyze the relationships between low-level contour integration and highlevel behavioral tasks.
However, neural networks are notorious black box function approximators that learn complex mapping between inputs and outputs by whatever mechanism minimizes a global cost function.
To fairly compare decision-making strategies and draw robust conclusions, architectures needed to be carefully considered and models need to be analyzed at multiple levels [14].
I n p u t c o n v [ c h = 64 , s i z e =, s t r i d e = 2 ] p r e t r a i n e d
B N
M a x P oo li n g
Edge Detection c o n v [ 16 , 3 ,] c o n v [, 1 , 1 ]
B N
O u t p u t
Fragments Classi (cid:1) er c o n v [, 3 , 3 ] c o n v [, 1 , 1 ]
B N
Binary Classi (cid:1) er
g l o b a l a v g .
p oo l
O u t p u t In this work, we explore whether contour integration can be learnt from high-level visual tasks and whether learnt mechanisms are consistent with observed neurophysiological and behavioral data. Li et. al.
[2] concurrently monitored behavioral performance and V1 neural responses of macaque monkeys.
At the behavioral level, contours became more salient as lengths increased.
When contours extended in the direction of the preferred orientation of V1 neurons, their ﬁring rate monotonically increased.
Conversely, when spacing between fragments increased, contours became less salient and V1 ﬁring rates decreased monotonically.
In a similar manner, we analyzed trained models behaviorally at the prediction level and neurophysiologically at the output of the embedded contour integration model.
Related work Recently, recurrent neural network (RNN) models of intra-layer horizontal connections have been proposed [7, 8, 15].
Similar to our model, these models learn lateral connectivity patterns from high-level tasks.
However, different from our model, they use complex multi-parameter gates inspired from Gated Recurrent Unit (GRU) [16] and Long Short Term Memory (LSTM)
RNNs.
These complex multi-parameter gates are functions of inputs, outputs and the internal states of multiple neurons and establish multiple parallel paths between them.
Some of these connections do not exist in the brain and their complex connectivity patterns makes it difﬁcult to map those that do exist back to the neuroanatomy.
In contrast, our simpler model is more biologically aligned and all its connections can be directly mapped onto existing V1 connections.
As pointed out by Fenke et. al.
[14], conclusions generalized beyond tested architectures can be fragile.
Furthermore, we focus primarily on replicating the brain’s mechanism of contour integration rather than achieving the best performance on high-level tasks using recurrence.
Model
The central component of the model is the contour integration (CI) layer.
It models V1 orientation columns and the interactions between them.
Each orientation column represents population of neurons that respond to edges of similar orientations at a particular spatial location.
Orientation columns are modeled with an pair of excitatory (E) and inhibitory (I) nodes.
E and I nodes connect locally with each other and selectively with nodes in neighboring columns.
The interaction between nodes are derived from the differential equation model of Piech et.
al.
To incorporate it into neural networks, we use Euler’s method to express its components as difference equations and use
RNN methods to make them easier to train [7, 18].
The ﬁnal form of these interactions is represented by
[Equation]
Here, x and y are the membrane potential of E and I nodes, respectively, f .
( . ) is a non-linear activation function that transforms membrane potentials into ﬁring rates, 1 / τ x = σ ( a ) , 1 / τ y
= σ ( b ) are membrane time constants, J xy , J yx are local I
→ E, E
→ I connection strengths, W e are lateral excitatory connections from E nodes in nearby columns to E, W i are lateral inhibitory connections from nearby E nodes to I, F .
( . ) is the output of all modeled nodes, (cid:126) is the convolution operator, σ () is the Sigmoid function which keeps time constants positive, I 0 .
is a node’s background activity, and I is the external input.
E nodes also locally self connect, E → E. This is included in W e which connects neighboring columns at the same spatial location as well.
Importantly, while we borrow the network structure of [6], we do not constrain the signs of the weights.
The ReLU functions tend to give the E and I neurons greater roles in increasing and decreasing activity, respectively, but these roles are not enforced.
The direct inﬂuence of x t − 1 on x has a similar form to the indirect inﬂuence via y t − 1 , but we found that constraining the weights strongly impacts learning.
We leave exploration of this issue for future work.
A
B
C
D
A
B
C
D
B, Using its edge label two markers are randomly placed on edge pixels.
C, In training, images are punctured with occlusion bubbles to randomly fragment contours.
D, To analyze the impact of fragment spacing, bubbles are placed along contours to fragment them with different spacing.
Lateral connections of V1 orientation columns are sparse and preferentially connect with neighbors with similar orientations, [19].
Existing models, [6, 5], use a ﬁxed association-ﬁeld [1] connection structure for each orientation column.
In our implementation, we do not force a ﬁxed lateral structure and learn it through training.
All E-I pairs are connected over a deﬁned spatial area and a sparsity constraint retains only the most prominent connections. All local connections and time constants are similarly learnt. The CI layer sits on-top an edge extraction layer (shallow layer of a neural network). E-I pairs are deﬁned for each input channel and spatial location.
Parameters are shared across spatial location but not across channels.
Feed-forward input is received only by E nodes.
After iterating through the CI layer for N itr steps, E node outputs are passed to the next layer.
The spatial extent of
V1 lateral connections S is up to 8 times the cRF of V1 neurons [19].
Consistent with this, we deﬁne
S to be much larger than edge extracting kernels.
For edge extraction we use the ﬁrst convolutional layer of a ResNet50 [9] that was pre-trained on ImageNet.
We added batch normalization and max pooling layers before the CI layer.
Not only did this reduced computational complexity
but it always improved performance as well.
The CI layer’s output was passed to classiﬁcation blocks that mapped activations to desired outputs.
The capacities of these blocks were kept to a minimum to allow the
CI layer to do most of the work.
Separate classiﬁcation blocks were used for the contour fragment detection and contour tracing in natural images tasks.
Figure 1 shows the architectures of our models and their parameters.
Training details are described in Section A.1.
The model was compared with a feed-forward control with matching capacity.
In the control, the CI layer was replaced by a block of sequentially arranged convolutional layers and did not use recurrence.
The block contained an equivalent number of convolutional layers and each layer had the same conﬁguration and parameters.
Dropout layers with a dropout probability of 0.3 were added after every convolutional layer.
This was necessary to prevent the control from over-ﬁtting training data.
For ease of notation we use CI layer to refer to this block as well and use the model type to distinguish between the two (model vs control).
Experiments 3.1 Synthetic contour fragments
As a ﬁrst task, we use stimuli typically used to study contour integration [1, 2].
Li et.
al.
[21] found that macaque monkeys progressively improved at detecting contours and had higher contour enhanced
V1 responses with experience on these stimuli.
Hence, contour integration is learnable from these stimuli.
Each stimulus consisted of several identical Gabor fragments that differed only in orientation.
The orientations of a few adjacent (contour) fragments were aligned to form a smooth contour. The
[Table] [Table] orientations of the remaining (background) fragments varied randomly.
Models were tasked with identifying contour fragments.
Embedded contours differed in their location, orientation, length l c (number of fragments), inter-fragment curvature β , and the Gabor fragment used in their construction.
Example stimuli are shown in Figure 2A and B and Section A.2 describes how they were constructed.
The dataset contained 64,000 training and 6,400 validation images.
In its construction, 64 different
Gabors, l c of 1, 3, 5, 7, 9 fragments and β rotations of 0 ◦ , ± 15 ◦ were used.
Gabor parameters were randomly selected with the only restriction that the resultant Gabor fragment appear as a well deﬁned line segment.
For l c = 1, the label was set to all zeros.
Contour integration requires inputs from outside the cRF and the model had to learn when not to apply enhancement gains as well.
Equal number of images were generated for each condition.
Effect of contour length and fragment spacing Trained models were tested for consistency with behavioral and neurophysiological data with centrally located straight contours only (consistent with available neurophysiological data [2]).
Behavioral performance was measured as the average intersection-over-union (IoU) of model predictions and labels.
Neurophysiological responses were monitored at the output of the CI layer of centrally located neurons of each channel.
The ﬁrst contour fragment was centered at the image center and was fully contained within the cRF of monitored neurons.
For each channel, ﬁrst, the optimal stimulus was found by monitoring which of the[-
-]
Gabor fragments elicited the maximum response in the cRF.
Next, contours were extended in the preferred direction.
Neurophysiological responses were quantiﬁed by the contour integration gain,
G ( l c , RCD ) =
Output l c , RCD Output l c =1 , RCD =1 , where the relative co-linear distance (RCD) [2] is the ratio of inter-fragment spacing to fragment length in pixels.
The effects of contour length were analyzed using l c = 1, 3, 5, 7, 9 fragments and a ﬁxed spacing of RCD=1 (see Figure 2C).
The effects of inter-fragment spacing were analyzed using RCD =
[7, 8, 9, 10, 11, 12, 13, 14] / 7 and a ﬁxed l c =fragments (see Figure 2D).
For each condition, results were averaged across 50 different images.
3.2 Contour tracing in natural images
To investigate how contour integration is learnt in our natural viewing environment, we created a new task on natural images.
For input images, we used the Barcelona Images for Perceptual Edge
Detection (BIPED) dataset [20] as it focuses on all contours rather than only on object boundaries.
Contour integration is a low-level phenomenon that occurs in shallow layers whereas object awareness typically develops in deeper layers.
To create a stimulus, two smooth non-overlapping contours were randomly selected from an image’s edges label.
Next, two easily identiﬁable markers were placed on the contours.
In some images, markers were placed to the same contour, while in others they were placed onto different contours.
Markers were added to input images and the models never saw the selected contours.
Finally, contours were fragmented by randomly puncturing the image with occlusion bubbles.
Models had to report whether markers laid on the same smooth contour.
An example stimulus is shown in Figure 3C. Stimulus construction is described in Section A.3.
The train dataset contained 50,000 contours from BIPED train images while the validation dataset contained 5,000 contours from BIPED test images.
Since the test dataset contains only 50 images, multiple contours per image were extracted.
Equal probabilities were used for both classes.
Effect of fragment spacing After training, models were tested for behavioral and neurophysiological consistency using contours with connected end-points that were fragmented in an orderly manner.
Different from training, bubbles were added along contours at calculated positions to fragment contours with ﬁxed inter-fragment spacing.
An example test stimulus is shown in Figure 3D and the procedure used to construct it is described in Section A.4.
Contours were punctured with bubbles of sizes 7, 9, 11, 13, 15, 17 pixels, corresponding to fragment spacing of [7, 9, 11, 13, 15, 17]/7 RCD.
Binary classiﬁcation accuracy was used to measure behavioral performance.
Neurophysiological responses were quantiﬁed by the contour integration gain for natural images,
G NI ( rcd ) =
Output RCD Output RCD
=1 .
Each channel of the CI layer was analyzed individually and results were averaged over 50 stimuli.
Results 4.1 Synthetic contour fragments
Mean IoU scores, averaged across 3 runs, are shown in Table 1.
The model performed ≈ 7% better than the control.
Table 1 also shows mean IoU scores over all centrally located straight contours from our analysis of the effects of contour length (test).
Both the model and the control found it easier to detect these contours with the model being ≈ 4% better.
While testing, we had an additional constraint of only considering neurons for whom the optimal stimulus was found (non-zero CI layer output for any single Gabor fragment in the cRF).
Out of the 192 possible, 185 model and 46 control neurons met this criteria.
Average IoU scores as contour length increases are shown in Figure 4A.
Both the model and control excelled ( ≥ 97% ) at detecting the absence of contours.
There were dips in performance for length 3 contours as they were the hardest to detect.
For all other lengths, prediction accuracy increased with length.
Larger contrasts between the model and control was observed in neurophysiological gains.
Figure 4B shows population average gains as contour lengths changed.
For the model, gains increased monotonically with contour length.
Control gains were more variable; gains dropped for lengthcontours but increased for other lengths.
In general, model gains were higher than control gains for all lengths.
Figure 4B also shows measured neurophysiological gains from Li et.
al.
Here, population average gains from the two monkeys used in their study were extracted using WebPlotDigitizer [22] and their weighted averages are plotted.
The impact of contour length on measured and model gains was consistent.
Figure 4C shows population average gains as the spacing between fragments was increased.
Model gains decreased while control gains increased with spacing.
The impact of spacing on measured and model gains was similar.
For population average gains, we additionally removed neurons that had gains above 20 for any contour condition.
Typically, this happened for neurons that had small optimal stimulus responses and resulted in high gains for small changes in CI layer outputs.
For the contour length investigation, we removed an additional 12 model and 9 control neurons while for the fragment spacing we removed an additional 21 model and 28 control neurons.
To get a better picture of overall trends, we plotted histograms of the gradients of linear ﬁts to CI layer outputs vs. length and vs. spacing curves of all neurons for whom the optimal stimulus was found.
Results of the model are shown in ﬁgure 4D and E while those of the control are shown in Figures 4F and G.
Most model neurons were consistent with population average trends and showed positive slopes as contour lengths increased and negative slope when fragment spacing increased.
The results of the model are consistent with observed neurophysiological trends while the control behaved differently.
Remarkably, their behavioral predictions were comparable.
The model and control appear to be employing different strategies to solve the task and only the model aligns with neurophysiology.
4.2 Contour tracing in natural images
Classiﬁcation accuracies, averages across 3 runs, are shown in Table 2.
The model performed ≈ 11% better than the control.
When occlusion bubbles were added along contours (test data), the predictions of both the model and control dropped, even for the smallest bubbles.
Accuracies as the spacing between contour fragments increased are shown in Figure 5A.
For both the model and the control, accuracies dropped as spacing increased, consistent with observed behavioral properties.
From the least to the most spacing conﬁguration, model accuracy dropped by ≈ 3% which was slightly more than that of the control, ≈ 1% .
However, the drop in performance from the training task where bubbles were randomly located to when they we placed along the contours was signiﬁcantly lower for the model (7 % ) compared to the control (14 % ) showing that model generalizes better.
Figure 5B shows population averaged contour integration gains, G NI , as the spacing between fragments increases.
Both model and control gains dropped as spacing increased, consistent with observed neurophysiological trends.
In contrast, where the output activations of individual neurons were compared, there was a larger difference between the model and control.
For each channel, we did a linear ﬁt to the output activation vs. fragment spacing curves and plotted the histogram of their gradients.
Histogram for the model are shown in Figure 5C while those of the control are shown in
Figure 5D. Model output activations dropped sharply as spacing increased (consistent with observed neurophysiological trends) while control outputs only slightly dropped.
Discussion
We found that brain-like contour integration can be learnt in artiﬁcial neural networks from high-level tasks.
However, models need to be architecturally constrained.
Moreover, it is important to validate models at multiple comparison points.
From naturalistic scenes, contour integration can be learnt by tracing smooth natural contours.
We plan to explore other more complex high-level tasks where such a capability would be useful.
We also plan to quantitatively compare other potential mechanisms of contour integration.
In particular, we are interested in the neurophysiological properties observed by
Chen et.
al.
Here, it is proposed that contour integration is mediated by a recurrent cross-cortical loop involving V1 and V4 cortices.
A Appendix A.1 Training process
Models were trained using gradient descent with the ADAM [24] optimizer.
For the synthetic contour fragments detection task, the starting learning rate was 3e-5, while for the contour tracing in natural images task it was 1e-4.
Learning rates were dropped by a factor of 10 every 30 epochs.
For loss we used Binary Cross Entropy plus a lateral connections weight constraint.
To encourage sparse lateral connections we used L1 regularization loss with a inverse 2D Gaussian mask.
The mask penalized weights that were far away from the center and encouraged them to be small.
Regularization loss was weighted by 1e-5 before adding to BCE loss.
The width of the Gaussian mask ( σ ) was set topixels.
Models were trained for at least 50 epochs with a batch size of 32.
All input images were 256 × 256 pixels.
Each task was trained separately.
A.2 Construction of synthetic contour fragments stimuli
The stimulus construction process was derived from [1].
First, an input image was sectioned into a grid of squares (full tiles) whose length was set to the pixel length of a fragment plus the desired inter-fragment spacing, d full .
The grid was aligned to map the center of the middle full tile to the image center.
Each fragment was also a square the same size as feed-forward kernels.
Input images were initialized with the channel-wise mean pixel value of the chosen Gabor fragment to blend in its edges.
Second, a starting contour fragment was randomly placed in the image.
Third, the location of the next contour fragment was determined by projecting a vector of length d full ± d full / 8 and orientation equal to the previous fragment
’s orientation ± β .
The random direction of β and distance jitter were added to prevent them appearing as cues.
Fourth, a fragment was rotated by β and added at this position.
The third and fourth steps were repeated until (cid:98) l c / 2 (cid:99) contour fragments were added to both ends of the starting fragment.
Next, background fragments were added to all unoccupied full tiles.
Background fragments were randomly rotated and positioned inside the larger full tiles.
Lastly, a binary label of whether the center of a contour fragment was present was generated for each full tile.
Input image size was ﬁxed to 256 × 256 pixels, resulting in labels of size 19 × 19.
All training stimuli used a ﬁx inter-fragment spacing of RCD=1.
In test stimuli, variable inter-fragment spacing was modeled by changing d full while keeping the fragment length constant.
A.3 Construction of training stimuli for contour tracing in natural images
To construct a stimulus, a random smooth contour C 1 was extracted from an (image, edge map) pair in the BIPED dataset.
Contours were extracted by selecting a starting edge pixel and incrementally adding contiguous edge pixels if they met a smoothness constraint that limited contour curvature to less than π / 4 radians.
One endpoint of C 1 was chosen as the position of the ﬁrst marker, M 1 .
Next, a second edge pixel that did not lie on C 1 was randomly selected.
To ensure connected and not connected stimuli have similar separation distances, the selection process used a non-uniform probability distribution to favor edge pixels that were equidistant with the unselected endpoint of
C 1 .
A second contour, C 2 , was extended from the second edge pixel.
If C 2 overlapped with C 1 , the process was repeated until a non-overlapping pair of contours was found.
The location of the second marker M 2 was determined by the type of stimulus.
For connected stimuli, M 2 was the opposite end of C 1 , while for not connected stimuli, one of the endpoints of C 2 was chosen.
Once marker positions are determined, markers were added to the corresponding input image.
The marker was a bulls-eye of alternating red and blue concentric circles (see Figure 3B).
To fragment contours, occlusion bubbles were randomly added to input images.
Following [25], 2D
Gaussian bubbles were used to reduce the impact of bubble edges.
Occluded parts were replaced by channel means.
Each image was punctured by 300 bubbles of various sizes.
Bubble sizes were speciﬁed by the full-width-half-magnitude (FWHM) of the chosen 2D Gaussian.
Bubbles were blended into the image over a square area deﬁned by 2 × FWHM of a bubble.
The location and size of bubbles was speciﬁed in a bubble mask which was blended into the image using, img punc = mask
× img + (1 − mask )
× mean ch .
Bubbles were allowed to overlap and a different mask was used for each image (see Figure 3C).
test stimuli for contour tracing in natural images
In the synthetic dataset, inputs were altered to ﬁnd the optimal stimulus of monitored neurons.
However, for natural images, inputs cannot be similarly changed.
Therefore, a new procedure was devised to ﬁnd optimal stimuli.
To ﬁnd optimal stimuli in natural images, multiple unoccluded connected contours were presented to the model (Figure 3B).
New random contours were selected from the BIPED train dataset.
We used the train dataset as opposed to the test dataset as it contained more images and a larger variety of contours.
For each image, the position of the most active neuron of each channel in the CI layer was found.
If it was within 3 pixels (the same as the stride length of the subsequent convolutional layer) of the contour, the image as well as the position of most active neuron were stored.
The process was repeated over 20,000 contours and the top 50 (contour, most active neuron) pairs were retained for each channel.
Given the optimal stimuli for each channel, each input contour was fragmented by inserted occlusion bubbles at speciﬁc positions along the contour.
Multiple bubble sizes were used to fragment contours with different inter-fragment spacing.
A ﬁxed fragment length of 7 pixels, the same size as the cRF, was used.
To ensure the cRF of the most active neuron was not affected, we found the position of the closest point on the contour.
Bubbles were then inserted along the contour at offsets of ± ( l frag + l bubble ),
[Equation]
[Equation]
, ... until the ends of the contour.
Finally, we restricted the blending-in area of bubbles to FWHM pixels to ensure visible contour fragments were unaffected.
This gave bubbles a square proﬁle but as can be seen in the inset plots of Figure 5C and
D), it did not signiﬁcantly impact contour integration layer inputs.