Task-Driven Learning of Contour Integration Responses in a V1 Model
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Under difﬁcult viewing conditions, the brain’s visual system uses a variety of modulatory techniques to augment its core feed-forward signals.
Incorporating theseinto artiﬁcial neural networks can potentially improve their robustness.
However,before such mechanisms can be recommended, they need to be fully understood.
Here, we present a biologically plausible model of one such mechanism, contourintegration, embedded in a task-driven artiﬁcial neural network.
The model isneuroanatomically realistic and all its connections can be mapped onto existingconnections in the V1 cortex.
We ﬁnd that the model learns to integrate contoursfrom high-level tasks including those involving natural images.
Trained modelsexhibited several observed neurophysiological and behavioral properties.
In contrast, a parameter matched feed-forward control achieved comparable task-levelperformance but was largely inconsistent with neurophysiological data.
Introduction
Contour integration [1, 2, 3, 4] is a phenomenon in the V1 cortex where stimuli from outside aneuron’s classical receptive ﬁeld (cRF) modulate its feed-forward responses.
In particular, responsesare enhanced if a preferred stimulus within the cRF is part of a larger contour.
Under difﬁcult viewingconditions, the visual ventral stream uses contour integration to pop out smooth natural contours(see Figure 2).
Contour integration is thought to be mediated by intra-area lateral and higher-layerfeedback connections.
Past computational models [5, 6] have tested potential mechanisms in isolationand replicated observed neurophysiological data.
However, apart from some recent work [7, 8], paststudies have done little to explore the role of contour integration in the perception of naturalisticscenes.
Separately, recent advances in deep neural networks have surpassed human-level performance onseveral high-level vision tasks such as object classiﬁcation [9, 10, 11] and object segmentation[12, 13].
This enables the possibility of embedding a contour integration model within a task-orientedsystem to quantitatively analyze the relationships between low-level contour integration and highlevel behavioral tasks.
However, neural networks are notorious black box function approximators thatlearn complex mapping between inputs and outputs by whatever mechanism minimizes a global costfunction.
To fairly compare decision-making strategies and draw robust conclusions, architecturesneeded to be carefully considered and models need to be analyzed at multiple levels [14].
In this work, we explore whether contour integration can be learnt from high-level visual tasks andwhether learnt mechanisms are consistent with observed neurophysiological and behavioral data.
Li et.
al.
[2] concurrently monitored behavioral performance and V1 neural responses of macaquemonkeys.
At the behavioral level, contours became more salient as lengths increased.
When contoursextended in the direction of the preferred orientation of V1 neurons, their ﬁring rate monotonicallyincreased.
Conversely, when spacing between fragments increased, contours became less salientand V1 ﬁring rates decreased monotonically.
In a similar manner, we analyzed trained models
I n p u t c o n v [ c h = 64 , s i z e =, s t r i d e = 2 ] p r e t r a i n e d
B N
M a x P oo li n g
Edge Detection c o n v [ 16 , 3 ,] c o n v [, 1 , 1 ]
B N
O u t p u t
Fragments Classi (cid:1) er c o n v [, 3 , 3 ] c o n v [, 1 , 1 ]
B N
Binary Classi (cid:1) er
g l o b a l a v g .
p oo l
O u t p u t behaviorally at the prediction level and neurophysiologically at the output of the embedded contourintegration model.
Related work Recently, recurrent neural network (RNN) models of intra-layer horizontal connections have been proposed [7, 8, 15].
Similar to our model, these models learn lateral connectivitypatterns from high-level tasks.
However, different from our model, they use complex multi-parametergates inspired from Gated Recurrent Unit (GRU) [16] and Long Short Term Memory (LSTM)
RNNs.
These complex multi-parameter gates are functions of inputs, outputs and the internal statesof multiple neurons and establish multiple parallel paths between them.
Some of these connectionsdo not exist in the brain and their complex connectivity patterns makes it difﬁcult to map those thatdo exist back to the neuroanatomy.
In contrast, our simpler model is more biologically aligned andall its connections can be directly mapped onto existing V1 connections.
As pointed out by Fenke et.al.
[14], conclusions generalized beyond tested architectures can be fragile.
Furthermore, we focusprimarily on replicating the brain’s mechanism of contour integration rather than achieving the bestperformance on high-level tasks using recurrence.
Model
The central component of the model is the contour integration (CI) layer.
It models V1 orientationcolumns and the interactions between them.
Each orientation column represents population ofneurons that respond to edges of similar orientations at a particular spatial location.
Orientationcolumns are modeled with an pair of excitatory (E) and inhibitory (I) nodes.
E and I nodes connectlocally with each other and selectively with nodes in neighboring columns.
The interaction betweennodes are derived from the differential equation model of Piech et.
al.
To incorporate it intoneural networks, we use Euler’s method to express its components as difference equations and use
RNN methods to make them easier to train [7, 18].
The ﬁnal form of these interactions is representedby
[Equation]
Here, x and y are the membrane potential of E and I nodes, respectively, f .
( . ) is a non-linearactivation function that transforms membrane potentials into ﬁring rates, 1 / τ x = σ ( a ) , 1 / τ y
= σ ( b )are membrane time constants, J xy , J yx are local I
→ E, E
→ I connection strengths, W e are lateralexcitatory connections from E nodes in nearby columns to E, W i are lateral inhibitory connectionsfrom nearby E nodes to I, F .
( . ) is the output of all modeled nodes, (cid:126) is the convolution operator,σ () is the Sigmoid function which keeps time constants positive, I 0 .
is a node’s background activity,and I is the external input.
E nodes also locally self connect, E → E. This is included in W e whichconnects neighboring columns at the same spatial location as well.
Importantly, while we borrow thenetwork structure of [6], we do not constrain the signs of the weights.
The ReLU functions tend togive the E and I neurons greater roles in increasing and decreasing activity, respectively, but theseroles are not enforced.
The direct inﬂuence of x t − 1 on x has a similar form to the indirect inﬂuencevia y t − 1 , but we found that constraining the weights strongly impacts learning.
We leave explorationof this issue for future work.
Lateral connections of V1 orientation columns are sparse and preferentially connect with neighborswith similar orientations, [19].
Existing models, [6, 5], use a ﬁxed association-ﬁeld [1] connectionstructure for each orientation column.
In our implementation, we do not force a ﬁxed lateral structureand learn it through training.
All E-I pairs are connected over a deﬁned spatial area and a sparsityconstraint retains only the most prominent connections.
All local connections and time constants aresimilarly learnt.
The CI layer sits on-top an edge extraction layer (shallow layer of a neural network). A B C
D
A
B C D B, Using its edge label two markers are randomly placed on edge pixels.
C, In training, images are punctured with occlusion bubbles to randomly fragment contours.
D, To analyze the impact of fragment spacing, bubbles are placed along contours to fragment them with different spacing.
E-I pairs are deﬁned for each input channel and spatial location.
Parameters are shared across spatiallocation but not across channels.
Feed-forward input is received only by E nodes.
After iteratingthrough the CI layer for N itr steps, E node outputs are passed to the next layer.
The spatial extent of
V1 lateral connections S is up to 8 times the cRF of V1 neurons [19].
Consistent with this, we deﬁne
S to be much larger than edge extracting kernels.
For edge extraction we use the ﬁrst convolutionallayer of a ResNet50 [9] that was pre-trained on ImageNet.
We added batch normalization and maxpooling layers before the CI layer.
Not only did this reduced computational complexity
but it alwaysimproved performance as well.
The CI layer’s output was passed to classiﬁcation blocks that mappedactivations to desired outputs.
The capacities of these blocks were kept to a minimum to allow the
CI layer to do most of the work.
Separate classiﬁcation blocks were used for the contour fragmentdetection and contour tracing in natural images tasks.
Figure 1 shows the architectures of our modelsand their parameters.
Training details are described in Section A.1.
The model was compared with afeed-forward control with matching capacity.
In the control, the CI layer was replaced by a blockof sequentially arranged convolutional layers and did not use recurrence.
The block contained anequivalent number of convolutional layers and each layer had the same conﬁguration and parameters.
Dropout layers with a dropout probability of 0.3 were added after every convolutional layer.
Thiswas necessary to prevent the control from over-ﬁtting training data.
For ease of notation we use CIlayer to refer to this block as well and use the model type to distinguish between the two (model vscontrol).
Experiments
3.1 Synthetic contour fragments
[Figure] As a ﬁrst task, we use stimuli typically used to study contour integration [1, 2].
Li et.
al.
[21] foundthat macaque monkeys progressively improved at detecting contours and had higher contour enhanced
V1 responses with experience on these stimuli.
Hence, contour integration is learnable from thesestimuli.
Each stimulus consisted of several identical Gabor fragments that differed only in orientation.
The orientations of a few adjacent (contour) fragments were aligned to form a smooth contour.
Theorientations of the remaining (background) fragments varied randomly.
Models were tasked withidentifying contour fragments.
Embedded contours differed in their location, orientation, length l c(number of fragments), inter-fragment curvature β , and the Gabor fragment used in their construction.
Example stimuli are shown in Figure 2A and B and Section A.2 describes how they were constructed.
The dataset contained 64,000 training and 6,400 validation images.
In its construction, 64 different
Gabors, l c of 1, 3, 5, 7, 9 fragments and β rotations of 0 ◦ , ± 15 ◦ were used.
Gabor parameters were [Table] [Table] randomly selected with the only restriction that the resultant Gabor fragment appear as a well deﬁnedline segment.
For l c = 1, the label was set to all zeros.
Contour integration requires inputs fromoutside the cRF and the model had to learn when not to apply enhancement gains as well.
Equalnumber of images were generated for each condition.
Effect of contour length and fragment spacing Trained models were tested for consistency withbehavioral and neurophysiological data with centrally located straight contours only (consistentwith available neurophysiological data [2]).
Behavioral performance was measured as the averageintersection-over-union (IoU) of model predictions and labels.
Neurophysiological responses weremonitored at the output of the CI layer of centrally located neurons of each channel.
The ﬁrst contourfragment was centered at the image center and was fully contained within the cRF of monitoredneurons.
For each channel, ﬁrst, the optimal stimulus was found by monitoring which of the
Gabor fragments elicited the maximum response in the cRF.
Next, contours were extended in thepreferred direction.
Neurophysiological responses were quantiﬁed by the contour integration gain,
G ( l c , RCD ) =
Output l c , RCD Output l c =1 , RCD =1 , where the relative co-linear distance (RCD) [2] is the ratio ofinter-fragment spacing to fragment length in pixels.
The effects of contour length were analyzedusing l c = 1, 3, 5, 7, 9 fragments and a ﬁxed spacing of RCD=1 (see Figure 2C).
The effects ofinter-fragment spacing were analyzed using RCD =
[7, 8, 9, 10, 11, 12, 13, 14] / 7 and a ﬁxed l c = 7fragments (see Figure 2D).
For each condition, results were averaged across 50 different images.
3.2 Contour tracing in natural images
To investigate how contour integration is learnt in our natural viewing environment, we created anew task on natural images.
For input images, we used the Barcelona Images for Perceptual Edge
Detection (BIPED) dataset [20] as it focuses on all contours rather than only on object boundaries.
Contour integration is a low-level phenomenon that occurs in shallow layers whereas object awarenesstypically develops in deeper layers.
To create a stimulus, two smooth non-overlapping contours wererandomly selected from an image’s edges label.
Next, two easily identiﬁable markers were placedon the contours.
In some images, markers were placed to the same contour, while in others theywere placed onto different contours.
Markers were added to input images and the models never sawthe selected contours.
Finally, contours were fragmented by randomly puncturing the image withocclusion bubbles.
Models had to report whether markers laid on the same smooth contour.
Anexample stimulus is shown in Figure 3C. Stimulus construction is described in Section A.3.
The traindataset contained 50,000 contours from BIPED train images while the validation dataset contained5,000 contours from BIPED test images.
Since the test dataset contains only 50 images, multiplecontours per image were extracted.
Equal probabilities were used for both classes.
Effect of fragment spacing After training, models were tested for behavioral and neurophysiological consistency using contours with connected end-points that were fragmented in an orderlymanner.
Different from training, bubbles were added along contours at calculated positions tofragment contours with ﬁxed inter-fragment spacing.
An example test stimulus is shown in Figure3D and the procedure used to construct it is described in Section A.4.
Contours were puncturedwith bubbles of sizes 7, 9, 11, 13, 15, 17 pixels, corresponding to fragment spacing of [7, 9, 11,13, 15, 17]/7 RCD.
Binary classiﬁcation accuracy was used to measure behavioral performance.
Neurophysiological responses were quantiﬁed by the contour integration gain for natural images,
G NI ( rcd ) =
Output RCD Output RCD
=1 .
Each channel of the CI layer was analyzed individually and resultswere averaged over 50 stimuli.
Results
4.1 Synthetic contour fragments
Mean IoU scores, averaged across 3 runs, are shown in Table 1.
The model performed ≈ 7% betterthan the control.
Table 1 also shows mean IoU scores over all centrally located straight contoursfrom our analysis of the effects of contour length (test).
Both the model and the control found iteasier to detect these contours with the model being ≈ 4% better.
While testing, we had an additionalconstraint of only considering neurons for whom the optimal stimulus was found (non-zero CI layeroutput for any single Gabor fragment in the cRF).
Out of the 192 possible, 185 model and 46 controlneurons met this criteria.
Average IoU scores as contour length increases are shown in Figure 4A.
Both the model and control excelled ( ≥ 97% ) at detecting the absence of contours.
There weredips in performance for length 3 contours as they were the hardest to detect.
For all other lengths,prediction accuracy increased with length.
Larger contrasts between the model and control was observed in neurophysiological gains.
Figure4B shows population average gains as contour lengths changed.
For the model, gains increasedmonotonically with contour length.
Control gains were more variable; gains dropped for length 3contours but increased for other lengths.
In general, model gains were higher than control gains for alllengths.
Figure 4B also shows measured neurophysiological gains from Li et.
al.
Here, populationaverage gains from the two monkeys used in their study were extracted using WebPlotDigitizer [22]and their weighted averages are plotted.
The impact of contour length on measured and model gainswas consistent.
Figure 4C shows population average gains as the spacing between fragments wasincreased.
Model gains decreased while control gains increased with spacing.
The impact of spacingon measured and model gains was similar.
For population average gains, we additionally removedneurons that had gains above 20 for any contour condition.
Typically, this happened for neurons thathad small optimal stimulus responses and resulted in high gains for small changes in CI layer outputs.
For the contour length investigation, we removed an additional 12 model and 9 control neurons whilefor the fragment spacing we removed an additional 21 model and 28 control neurons.
To get a betterpicture of overall trends, we plotted histograms of the gradients of linear ﬁts to CI layer outputs vs.length and vs. spacing curves of all neurons for whom the optimal stimulus was found.
Results ofthe model are shown in ﬁgure 4D and E while those of the control are shown in Figures 4F and G.
Most model neurons were consistent with population average trends and showed positive slopes ascontour lengths increased and negative slope when fragment spacing increased.
The results of themodel are consistent with observed neurophysiological trends while the control behaved differently.
Remarkably, their behavioral predictions were comparable.
The model and control appear to beemploying different strategies to solve the task and only the model aligns with neurophysiology.
4.2 Contour tracing in natural images
Classiﬁcation accuracies, averages across 3 runs, are shown in Table 2.
The model performed ≈ 11%better than the control.
When occlusion bubbles were added along contours (test data), the predictionsof both the model and control dropped, even for the smallest bubbles.
Accuracies as the spacingbetween contour fragments increased are shown in Figure 5A.
For both the model and the control,accuracies dropped as spacing increased, consistent with observed behavioral properties.
From theleast to the most spacing conﬁguration, model accuracy dropped by ≈ 3% which was slightly morethan that of the control, ≈ 1% .
However, the drop in performance from the training task wherebubbles were randomly located to when they we placed along the contours was signiﬁcantly lowerfor the model (7 % ) compared to the control (14 % ) showing that model generalizes better.
Figure5B shows population averaged contour integration gains, G NI , as the spacing between fragmentsincreases.
Both model and control gains dropped as spacing increased, consistent with observedneurophysiological trends.
In contrast, where the output activations of individual neurons were compared, there was a larger difference between the model and control.
For each channel, we dida linear ﬁt to the output activation vs. fragment spacing curves and plotted the histogram of theirgradients.
Histogram for the model are shown in Figure 5C while those of the control are shown in
Figure 5D. Model output activations dropped sharply as spacing increased (consistent with observedneurophysiological trends) while control outputs only slightly dropped.
Discussion
We found that brain-like contour integration can be learnt in artiﬁcial neural networks from high-leveltasks.
However, models need to be architecturally constrained.
Moreover, it is important to validatemodels at multiple comparison points.
From naturalistic scenes, contour integration can be learnt bytracing smooth natural contours.
We plan to explore other more complex high-level tasks where sucha capability would be useful.
We also plan to quantitatively compare other potential mechanisms ofcontour integration.
In particular, we are interested in the neurophysiological properties observed by
Chen et.
al.
Here, it is proposed that contour integration is mediated by a recurrent cross-corticalloop involving V1 and V4 cortices.
A AppendixA.1 Training process
Models were trained using gradient descent with the ADAM [24] optimizer.
For the synthetic contourfragments detection task, the starting learning rate was 3e-5, while for the contour tracing in naturalimages task it was 1e-4.
Learning rates were dropped by a factor of 10 every 30 epochs.
For loss weused Binary Cross Entropy plus a lateral connections weight constraint.
To encourage sparse lateralconnections we used L1 regularization loss with a inverse 2D Gaussian mask.
The mask penalizedweights that were far away from the center and encouraged them to be small.
Regularization losswas weighted by 1e-5 before adding to BCE loss.
The width of the Gaussian mask ( σ ) was set to 10pixels.
Models were trained for at least 50 epochs with a batch size of 32.
All input images were256 × 256 pixels.
Each task was trained separately.
A.2 Construction of synthetic contour fragments stimuli
The stimulus construction process was derived from [1].
First, an input image was sectioned intoa grid of squares (full tiles) whose length was set to the pixel length of a fragment plus the desiredinter-fragment spacing, d full .
The grid was aligned to map the center of the middle full tile to theimage center.
Each fragment was also a square the same size as feed-forward kernels.
Input imageswere initialized with the channel-wise mean pixel value of the chosen Gabor fragment to blend in itsedges.
Second, a starting contour fragment was randomly placed in the image.
Third, the locationof the next contour fragment was determined by projecting a vector of length d full ± d full / 8 andorientation equal to the previous fragment’s orientation ±
β .
The random direction of β and distancejitter were added to prevent them appearing as cues.
Fourth, a fragment was rotated by β and addedat this position.
The third and fourth steps were repeated until (cid:98) l c / 2 (cid:99) contour fragments were addedto both ends of the starting fragment.
Next, background fragments were added to all unoccupied fulltiles.
Background fragments were randomly rotated and positioned inside the larger full tiles.
Lastly,a binary label of whether the center of a contour fragment was present was generated for each fulltile.
Input image size was ﬁxed to 256 × 256 pixels, resulting in labels of size 19 × 19.
All training stimuli used a ﬁx inter-fragment spacing of RCD=1.
In test stimuli, variable inter-fragmentspacing was modeled by changing d full while keeping the fragment length constant.
A.3 Construction of training stimuli for contour tracing in natural images
To construct a stimulus, a random smooth contour C 1 was extracted from an (image, edge map) pairin the BIPED dataset.
Contours were extracted by selecting a starting edge pixel and incrementallyadding contiguous edge pixels if they met a smoothness constraint that limited contour curvatureto less than π / 4 radians.
One endpoint of C 1 was chosen as the position of the ﬁrst marker, M 1 .
Next, a second edge pixel that did not lie on C 1 was randomly selected.
To ensure connected andnot connected stimuli have similar separation distances, the selection process used a non-uniformprobability distribution to favor edge pixels that were equidistant with the unselected endpoint of
C 1 .
A second contour, C 2 , was extended from the second edge pixel.
If C 2 overlapped with C 1 , theprocess was repeated until a non-overlapping pair of contours was found.
The location of the secondmarker M 2 was determined by the type of stimulus.
For connected stimuli, M 2 was the oppositeend of C 1 , while for not connected stimuli, one of the endpoints of C 2 was chosen.
Once markerpositions are determined, markers were added to the corresponding input image.
The marker was abulls-eye of alternating red and blue concentric circles (see Figure 3B).
To fragment contours, occlusion bubbles were randomly added to input images.
Following [25], 2D
Gaussian bubbles were used to reduce the impact of bubble edges.
Occluded parts were replacedby channel means.
Each image was punctured by 300 bubbles of various sizes.
Bubble sizeswere speciﬁed by the full-width-half-magnitude (FWHM) of the chosen 2D Gaussian.
Bubbleswere blended into the image over a square area deﬁned by 2 × FWHM of a bubble.
The locationand size of bubbles was speciﬁed in a bubble mask which was blended into the image using,img punc =
mask
× img + (1 − mask )
× mean ch .
Bubbles were allowed to overlap and a differentmask was used for each image (see Figure 3C).
A.
4 Construction of test stimuli for contour tracing in natural images
In the synthetic dataset, inputs were altered to ﬁnd the optimal stimulus of monitored neurons.
However, for natural images, inputs cannot be similarly changed.
Therefore, a new procedure wasdevised to ﬁnd optimal stimuli.
To ﬁnd optimal stimuli in natural images, multiple unoccludedconnected contours were presented to the model (Figure 3B).
New random contours were selectedfrom the BIPED train dataset.
We used the train dataset as opposed to the test dataset as it containedmore images and a larger variety of contours.
For each image, the position of the most active neuronof each channel in the CI layer was found.
If it was within 3 pixels (the same as the stride length ofthe subsequent convolutional layer) of the contour, the image as well as the position of most activeneuron were stored.
The process was repeated over 20,000 contours and the top 50 (contour, mostactive neuron) pairs were retained for each channel.
Given the optimal stimuli for each channel, each input contour was fragmented by inserted occlusionbubbles at speciﬁc positions along the contour.
Multiple bubble sizes were used to fragment contourswith different inter-fragment spacing.
A ﬁxed fragment length of 7 pixels, the same size as the cRF,was used.
To ensure the cRF of the most active neuron was not affected, we found the positionof the closest point on the contour.
Bubbles were then inserted along the contour at offsets of±
[Equation]
[Equation]
, ±
[Equation]
, ... until the ends of the contour.
Finally, werestricted the blending-in area of bubbles to FWHM pixels to ensure visible contour fragments wereunaffected.
This gave bubbles a square proﬁle but as can be seen in the inset plots of Figure 5C and
D), it did not signiﬁcantly impact contour integration layer inputs.