Original: Unsupervised learning of features and object boundaries from local prediction
Revised: Unsupervised learning of features and object boundaries from local prediction

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: A visual system has to learn both which features to extract from images and howto group locations into (proto-)objects.
Revised: A visual system has to learn both which features to extract from images and howto group locations into (proto-)objects.

Original: Those two aspects are usually dealt withseparately, although predictability is discussed as a cue for both.
Revised: Those two aspects are usually dealt withseparately, although predictability is discussed as a cue for both.

Original: To incorporatefeatures and boundaries into the same model, we model a layer of feature mapswith a pairwise Markov random ﬁeld model in which each factor is paired with anadditional binary variable, which switches the factor on or off.
Revised: To incorporatefeatures and boundaries into the same model, we model a layer of feature mapswith a pairwise Markov random ﬁeld model in which each factor is paired with anadditional binary variable, which switches the factor on or off.

Original: Using one of twocontrastive learning objectives, we can learn both the features and the parameters ofthe Markov random ﬁeld factors from images without further supervision signals.
Revised: Using one of twocontrastive learning objectives, we can learn both the features and the parameters ofthe Markov random ﬁeld factors from images without further supervision signals.

Original: The features learned by shallow neural networks based on this loss are local averages, opponent colors, and Gabor-like stripe patterns.
Revised: The features learned by shallow neural networks based on this loss are local averages, opponent colors, and Gabor-like stripe patterns.

Original: Furthermore, we can inferconnectivity between locations by inferring the switch variables.
Revised: Furthermore, we can inferconnectivity between locations by inferring the switch variables.

Original: Contours inferredfrom this connectivity perform quite well on the Berkeley segmentation database(BSDS500) without any training on contours.
Revised: Contours inferredfrom this connectivity perform quite well on the Berkeley segmentation database(BSDS500) without any training on contours.

Original: Thus, computing predictions acrossspace aids both segmentation and feature learning, and models trained to optimizethese predictions show similarities to the human visual system.
Revised: Thus, computing predictions acrossspace aids both segmentation and feature learning, and models trained to optimizethese predictions show similarities to the human visual system.

Original: We speculate thatretinotopic visual cortex might implement such predictions over space throughlateral connections.
Revised: We speculate thatretinotopic visual cortex might implement such predictions over space throughlateral connections.

Original:  Introduction
Revised:  Introduction

Original: A long-standing question about human vision is how representations initially be based on parallelprocessing of retinotopic feature maps can represent objects in a useful way.
Revised: A long-standing question about human vision is how representations initially be based on parallelprocessing of retinotopic feature maps can represent objects in a useful way.

Original: Most research onthis topic has focused on computing later object-centered representations from the feature maprepresentations.
Revised: Most research onthis topic has focused on computing later object-centered representations from the feature maprepresentations.

Original: Psychology and neuroscience  identiﬁed features that lead to objects being groupedtogether [37, 38], established feature integration into coherent objects as a sequential process [73] anddeveloped solutions to the binding problem, i e. ways how neurons could signal whether they representparts of the same object [17, 57, 67, 72].
Revised: Psychology and neuroscience have identiﬁed features that lead to objects beinggrouped together [31, 32], have established feature integration into coherent objects as a sequentialprocess [62] and have developed solutions to the binding problem, i e. ways how neurons could signalwhether they represent parts of the same object [12, 46, 56, 61].

Original: In computer vision, researchers also focused on how featuremap representations could be turned into segmentations and object masks.
Revised: In computer vision, researchers havealso focused on how feature map representations could be turned into segmentations and object masks.

Original: Classically, segmentationalgorithm were clustering algorithms operating on extracted feature spaces [2, 12, 13, 16, 66], andthis approach is still explored with more complex mixture models today [74].
Revised: Classically, segmentation algorithm have been clustering algorithms operating on extracted featurespaces [1, 7, 8, 11, 55].

Original: Since the advent ofdeep neural network models, the focus has shifted towards models that directly map to contour mapsor semantic segmentation maps [21, 27, 39, 50, 65, 83], as reviewed in [54].
Revised: Since the advent of deep neural network models, the focus has shifted towardsmodels that directly map to contour maps or semantic segmentation maps [16, 22, 33, 40, 54, 69].

Original: Diverse ﬁndings suggest that processing within the feature maps take object boundaries into account.
Revised: Diverse ﬁndings suggest that processing within the feature maps take object boundaries into account.

Original: For example, neurons appear to encode border ownership [34, 57, 63] and to ﬁll in information acrosssurfaces [40] and along illusory contours [23, 76].
Revised: For example, neurons appear to encode border ownership [28, 46, 52] and to ﬁll in information acrosssurfaces [34] and along illusory contours [18, 64].

Original: Also, attention spreading through the featuremaps seems to respect object boundaries [4, 59].
Revised: Also, attention spreading through the feature mapsseems to respect object boundaries [3, 48].

Original: And selecting neurons that correspond to an objecttakes time, which scales with the distance between the points to be compared [35, 41].
Revised: And selecting neurons that correspond to an object takestime, which scales with the distance between the points to be compared [29, 35].

Original: Finally, along history of psychophysical studies showed that changes in spatial frequency and orientation content can deﬁne (texture) boundaries [e.g. 5, 45, 81].

Original: In both human vision and computer vision,relatively little attention has been given to these effects of grouping or segmentation on the featuremaps themselves. 
Revised: In both humanvision and computer vision, relatively little attention has been given to these effects of grouping orsegmentation on the feature maps themselves. A B C +

Revised: [Equation]

Original:  Additionally, most theories for grouping and segmentation take the features in the original featuremaps as given.
Revised: = [Equation] D Additionally, most theories for grouping and segmentation take the features in the original featuremaps as given.

Original: In human vision, these features are traditionally chosen by the experimenter [37,73, 72] or are inferred based on other research [57, 63].
Revised: In human vision, these features are traditionally chosen by the experimenter [31,62, 61] or are inferred based on other research [46, 52].

Original: Similarly, computer vision algorithms usedoff-the-shelf feature banks originally [2, 12, 13, 16, 66], and have recently moved towards deep neuralnetwork representations trained for other tasks as a source for feature maps [21, 27, 39, 50, 65, 83].
Revised: Similarly, computer vision algorithms usedoff-the-shelf feature banks originally [1, 7, 8, 11, 55], and have recently moved towards deep neuralnetwork representations trained for other tasks as a source for feature maps [16, 22, 33, 40, 54, 69].

Original: Interestingly, predictability of visual inputs over space and time has been discussed as a solution forboth these limitations of earlier theories.
Revised: Interestingly, predictability of visual inputs over space and time has been discussed as a solution forboth these limitations of earlier theories.

Original: Predictability has been used as a cue for segmentation sincethe law of common fate of Gestalt psychology [37], and both lateral interactions in visual cortices andcontour integration respect the statistics of natural scenes [19, 20].
Revised: Predictability has been used as a cue for segmentation sincethe law of common fate of Gestalt psychology [31], and both lateral interactions in visual cortices andcontour integration respect the statistics of natural scenes [14, 15].

Original: Among other signals like sparsity[55] or reconstruction [36], predictability is also a well known signal for self-supervised learningof features [80], which has been exploited by many recent contrastive learning [e.g. 15, 24, 29, 75]and predictive coding schemes [e.g. 51, 52, 75] for self-supervised learning.
Revised: Among other signals like sparsity[44] or reconstruction [30], predictability is also a well known signal for self-supervised learningof features [68], which has been exploited by many recent contrastive learning [e.g. 10, 19, 24, 63]and predictive coding schemes [e.g. 41, 42, 63] for self-supervised learning.

Original: However, these uses ofpredictability for feature learning and for segmentation are usually studied separately.
Revised: However, these uses ofpredictability for feature learning and for segmentation are usually studied separately.

Original: Here, we propose a model that learns both features and segmentation without supervision.
Revised: Here, we propose a model that learns both features and segmentation without supervision.

Original: Predictionsbetween locations provide a self-supervised loss to learn the features, how to perform the predictionand how to infer which locations should be grouped.
Revised: Predictionsbetween locations provide a self-supervised loss to learn the features, how to perform the predictionand how to infer which locations should be grouped.

Original: Also, this view combines contrastive learning[24, 75], a Markov random ﬁeld model for the feature maps [46] and segmentation into a coherentframework.
Revised: Also, this view combines contrastive learning[19, 63], a Markov random ﬁeld model for the feature maps [37] and segmentation into a coherentframework.

Original: We implement our model using some shallow architectures.
Revised: We implement our model using some shallow architectures.

Original: The learned featuresresemble early cortical responses and the object boundaries we infer from predictability align wellwith human object contour reports from the Berkeley segmentation database (BSDS500 [2]).
Revised: The learned featuresresemble early cortical responses and the object boundaries we infer from predictability align wellwith human object contour reports from the Berkeley segmentation database (BSDS500 [1]).

Original: Thus,retinotopic visual cortex might implement similar computational principles as we propose here.
Revised: Thus,retinotopic visual cortex might implement similar computational principles as we propose here.

Original:  Model
Revised:  Model

Original: To explain our combined model of feature maps and their local segmentation information, we startwith a Gaussian Markov random ﬁeld model [46] with pairwise factors.
Revised: To explain our combined model of feature maps and their local segmentation information, we startwith a Gaussian Markov random ﬁeld model [37] with pairwise factors.

Original: We then add a variablew 2 { 0 , 1 } to each factor that governs whether the factor enters the product or not.
Revised: We then add a variablew 2 { 0 , 1 } to each factor that governs whether the factor enters the product or not.

Original: This yields a jointdistribution for the whole feature map and all w ’s. Marginalizing out the w ’s yields a Markov randomﬁeld with "robust" factors for the feature map, which we can useto predict feature vectors from thevectors at neighboring positions.
Revised: This yields ajoint distribution for the whole feature map and all w ’s. Marginalizing out the w ’s yields a Markov random ﬁeld with "robust" factors for the feature map, which we can use predict feature vectors fromthe vectors at neighboring positions.

Original: We ﬁnd two contrastive losses based on these predictions that canbe used to optimize the feature extraction and the factors in the Markov random ﬁeld model.
Revised: We ﬁnd two contrastive losses based on these predictions thatcan be used to optimize the feature extraction and the factors in the Markov random ﬁeld model.

Original: We model the distribution of k -dimensional feature maps f 2 R k,m 0 ,n 0 that are computed from inputimages I 2 R c,m,n with c = 3 color channels (see Fig.
Revised: We model the distribution of k -dimensional feature maps f 2 R k,m 0 ,n 0 that are computed from inputimages I 2 R c,m,n with c = 3 color channels (see Fig.

Original: 1 A & B).
Revised: 1 A & B).

Original: We use a Markov random ﬁeldmodel with pairwise factors, i.e. we deﬁne the probability of encountering a feature map f withentries f
Revised: We use a Markov random ﬁeldmodel with pairwise factors, i.e. we deﬁne the probability of encountering a feature map f withentries f

Original: i at locations i 2 [1 . . .
Revised: i at locations i 2 [1 . . .

Original: m 0 ] ⇥ [1 . . .
Revised: m 0 ] ⇥ [1 . . .

Original: n 0 ] as follows:
Revised: n 0 ] as follows:

Original: [Equation]
Revised: [Equation]

Original: where i is the local factor, N is the set of all neighboring pairs, and ij is the pairwise factorbetween positions i and j 1 .
Revised: where i is the local factor, N is the set of all neighboring pairs, and ij is the pairwise factorbetween positions i and j 1 .

Original: We will additionally assume shift invariance, i.e. each point has the sameset of nearby relative positions in the map as neighbors, i is the same factor for each position, andeach factor ij depends only on the relative position of i and j .
Revised: We will additionally assume shift invariance, i.e. each point has the sameset of nearby relative positions in the map as neighbors, i is the same factor for each position, andeach factor ij depends only on the relative position of i and j .

Original: We now add a binary variable w 2 { 0 , 1 } to each pairwise factor that encodes whether the factoris ’active’ ( w = 1 ) for that particular image (Fig. 1 C).
Revised: We now add a binary variable w 2 { 0 , 1 } to each pairwise factor that encodes whether the factoris ’active’ ( w = 1 ) for that particular image (Fig. 1 C).

Original: To scale the probability of w = 1 andw = 0 relative to each other, we add a factor that scales them with constants p ij 2 [0 , 1] and 1  p ijrespectively:[Equation] A B
Revised: To scale the probability of w = 1 andw = 0 relative to each other, we add a factor that scales them with constants p ij 2 [0 , 1] and 1  p ijrespectively:

Original: C +

Original: [Equation]
Revised: [Equation]

Original: =

Original: [Equation] D [Figure] Finally we assume that the factors are Gaussian and the feature vectors are originally normalized tohave mean 0 and variance 1 :
Revised: Finally, we assume that the factors are Gaussian and the feature vectors are originally normalized tohave mean 0 and variance 1 :

Original: [Equation]
Revised: [Equation]

Original: where Z 0 is the overall normalization constant, N ( f , 0 , I ) is the density of a standard normaldistribution with k ⇥ m 0 ⇥ n 0 dimensions, C ij governs the strength of the coupling in the form of aprecision matrix, which we will assume to be diagonal and Z ( w ij , C ij ) scales the distributions withw ij = 0 and w ij = 1 relative to each other.
Revised: where Z 0 is the overall normalization constant, N ( f , 0 , I ) is the density of a standard normaldistribution with k ⇥ m 0 ⇥ n 0 dimensions, C ij governs the strength of the coupling in the form of aprecision matrix, which we will assume to be diagonal, and Z ( w ij , C ij ) scales the distributions withw ij = 0 and w ij = 1 relative to each other.

Original: We set Z ( w ij , C ij ) to the normalization constant of the Gaussian with standard Gaussian factorsfor f i and f j respectively.
Revised: We set Z ( w ij , C ij ) to the normalization constant of the Gaussian with standard Gaussian factorsfor f i and f j respectively.

Original: For w = 0 this is just (2 ⇡ )  k , the normalization constant of a standard
Revised: For w = 0 this is just (2 ⇡ )  k , the normalization constant of a standard

Original: Gaussian in 2 k dimensions.
Revised: Gaussian in 2 k dimensions.

Original: For w = 1 we get:
Revised: For w = 1 we get:

Original: [Equation]
Revised: [Equation]

Original: which we get by computing the normalization constant of a Gaussian with the given precision andthen using the assumption that C ij is a diagonal matrix with diagonal entries c ll .
Revised: which we get by computing the normalization constant of a Gaussian with the given precision andthen using the assumption that C ij is a diagonal matrix with diagonal entries c ll .

Original: This normalization depends only on w and the coupling matrix C of the factor ij and thus induces avalid probability distribution on the feature maps.
Revised: This normalization depends only on w and the coupling matrix C of the factor ij and thus induces avalid probability distribution on the feature maps.

Original: Two points are notable about this normalizationthough:
Revised: Two points are notable about this normalizationthough:

Original: First, once other factors also constrain f i and/or f j , this normalization will not guaranteep ( w ij = 1) =
Revised: First, once other factors also constrain f i and/or f j , this normalization will not guaranteep ( w ij = 1) =

Original: p ij .
Revised: p ij .

Original: 2 Second, the w ij are not independent in the resulting distribution.
Revised: 2 Second, the w ij are not independent in the resulting distribution.

Original: For example, ifpairwise factors connect a to b , b to c and a to c the corresponding w are dependent, because w ab = 1and w bc = 1 already imply a smaller difference between f a and f c than if these factor were inactive,which increases the probability for w ac = 1 .
Revised: For example, if Instead, p ( w ij = 1) will be higher, because other factors increase the precision for the feature vectors, which makes the normalization constants more similar. I d e n t i t y 3 F ea t m x n C o n v 50 xx50 F ea t m x n

Revised: C o n v 3 xx3 F ea t m x n

Revised: S t r i d e 3 x

Revised: C o n v 64 xx

Original: {+S t r i d e 
+}
Revised: {+S t r i d e 
+}

Revised: x64R

Revised: e s - b l

Revised: o

Revised: c

Revised: k 128 xx

Revised: S t r i d e 2 x128R

Revised: e s - b l

Revised: o

Revised: c

Revised: k 256 xx64 F ea t m /x n /

Revised: F ea t m /x n /

Revised: F ea t m /x n /

Revised: Pixel A

Revised: B

Revised: Linear

Revised: C

Revised: Predseg

Revised: D

Revised: E

Original: 2.1 Learning
Revised: 2.1 Learning

Original: To learn our model from data, we use a contrastive learning objective on the marginal likelihood p ( f ) .
Revised: To learn our model from data, we use a contrastive learning objective on the marginal likelihood p ( f ) .

Original: To do so, we ﬁrst need to marginalize out the w ’s, which is fortunately simple, because each w affectsonly a single factor:
Revised: To do so, we ﬁrst need to marginalize out the w ’s, which is fortunately simple, because each w affectsonly a single factor:

Original: [Equation]
Revised: [Equation]

Original: Using this marginal likelihood directly for ﬁtting is infeasible though, because computing Z 0 , i.e.normalizing this distribution is not computationally tractable.
Revised: Using this marginal likelihood directly for ﬁtting is infeasible though, because computing Z 0 , i.e.normalizing this distribution is not computationally tractable.

Original: We resort to contrastive learning to ﬁt the unnormalized probability distribution [24], i.e. we optimizediscrimination from a noise distribution with the same support as the target distribution.
Revised: We resort to contrastive learning to ﬁt the unnormalized probability distribution [19], i.e. we optimizediscrimination from a noise distribution with the same support as the target distribution.

Original: Following[75] we do not optimize the Markov random ﬁeld directly, but optimize predictions based on themodel using features from other locations as the noise distribution.
Revised: Following[63] we do not optimize the Markov random ﬁeld directly, but optimize predictions based on themodel using features from other locations as the noise distribution.

Original: For this noise distribution, thefactors that depend only on a single location (the ﬁrst product in (1)) will cancel.
Revised: For this noise distribution, thefactors that depend only on a single location (the ﬁrst product in (1)) will cancel.

Original: We thus ignore the
Revised: We thus ignore the

Original: N ( f , 0 , I ) in our optimization and instead normalize the feature maps to mean 0 and unit varianceacross each image.
Revised: N ( f , 0 , I ) in our optimization and instead normalize the feature maps to mean 0 and unit varianceacross each image.

Original: We deﬁne two alternative losses that make predictions for positions based on alltheir neighbors or for a single factor respectively.
Revised: We deﬁne two alternative losses that make predictions for positions based on alltheir neighbors or for a single factor respectively.

Original: 2.1.1 Position loss
Revised: 2.1.1 Position loss

Original: The position loss optimizes the probability of the feature vector at each location relative to theprobability of randomly chosen other feature vectors from different locations and images:
Revised: The position loss optimizes the probability of the feature vector at each location relative to theprobability of randomly chosen other feature vectors from different locations and images (Fig. 2 D):

Original: [Equation]
Revised: [Equation]

Original: where N ( i ) is the set of neighbors of i .
Revised: where N ( i ) is the set of neighbors of i .

Original: This loss is consistent with the prediction made by the whole Markov random ﬁeld, but is relativelyinefﬁcient, because the predicted distribution p ( f i | f j 8 j 2 N ( i )) and the normalization constants forthese conditional distributions are different for every location i .
Revised: This loss is consistent with the prediction made by the whole Markov random ﬁeld, but is relativelyinefﬁcient, because the predicted distribution p ( f i | f j 8 j 2 N ( i )) is different for every location i .

Original: Thus, the second term in equation(9) cannot be reused across the locations i .
Revised: Thus, the second term in equation (9) cannot be reused across the locations i .

Original: Instead, we need to compute the second term for eachlocation separately, which requires a similar amount of memory as the whole feature representationfor each negative sample i 0 and each neighbor. To enable asufﬁciently large  set ofnegative points i 0 with the available memory, we compute this lossmultiple timeswith few negative samples and sum the gradients.
Revised: To enable a large enough set of points i 0 to contrast against, we accumulate the gradient from thisloss over repeated draws of random points, i.e. we compute this loss multiple times and propagatethe gradient of the loss back to the values of the feature maps.

Original: This trick saves memory, becausewe can free the memory for the loss computation after each repetition. As the initial computationof the feature maps is the same for all negative samples, we can save some computation for thisprocedure by computing the feature maps only once. To propagate the  gradients  throughthis singlecomputation, we add up the gradients of the loss repetitions with regard to  the feature maps and thenpropagate this summed gradient through the feature map computation.
Revised: At that level, we sum the gradientover multiple repetitions and then propagate the sum of these gradients back through the network. This trick saves memory and effectively allows us to use many more points in the normalizationset, because we can free the values and gradients for the different random sets of points after eachrepetition.

Original: This procedure does not savecomputation time compared to the loss with many negative samples, as we still need to calculate theevaluation for each position and each sample in the normalization [-set.
 Instead, p (-]
Revised: This procedure does not save computation time, as we still need to calculate the evaluationfor each position and each sample in the normalization [-set.
 Instead, p (-]set.

Original: 2.1.2 Factor loss
Revised: 2.1.2 Factor loss

Original:  The factor loss instead maximizes each individual factor for the correct feature vectors relative torandom pairs of feature vectors sampled from different locations and images:
Revised: [Figure] The factor loss instead maximizes each individual factor for the correct feature vectors relative torandom pairs of feature vectors sampled from different locations and images (Fig. 2 E):

Original: [Equation]
Revised: [Equation]

Original: where i, j index the correct locations and i 0 , j 0 index randomly drawn locations, in our implementationgenerated by shufﬂing the feature maps and taking all pairs that occur in these shufﬂed maps.
Revised: where i, j index the correct locations and i 0 , j 0 index randomly drawn locations, in our implementationgenerated by shufﬂing the feature maps and taking all pairs that occur in these shufﬂed maps.

Original: This loss does not lead to a consistent estimation of the MRF model, because the prediction p ( f i | f j )should not be based only on the factor ij , but should include indirect effects as f j also constrainsthe other neighbors of i .
Revised: This loss does not lead to a consistent estimation of the MRF model, because the prediction p ( f i | f j )should not be based only on the factor ij , but should include indirect effects as f j also constrainsthe other neighbors of i .

Original: Optimizing each factor separately will thus overaccount for informationthat could be implemented in two factors.
Revised: Optimizing each factor separately will thus overaccount for informationthat could be implemented in two factors.

Original: However, this loss has the distinct advantage that the samenoise evaluations can be used for all positions and images in a minibatch, which enables a muchlarger number of noise samples and thus much faster convergence.
Revised: However, this loss has the distinct advantage that the samenoise evaluations can be used for all positions and images in a minibatch, which enables a muchlarger number of noise samples and thus much faster convergence.

Original: 2.1.3 Optimization
Revised: 2.1.3 Optimization

Original: We optimize all weights of the neural network used for feature extraction and the parameters of therandom ﬁeld, i.e. the connectivity matrices C and the p ij for the different relative spatial locationssimultaneously.
Revised: We optimize all weights of the neural network used for feature extraction and the parameters of therandom ﬁeld, i.e. the connectivity matrices C and the p ij for the different relative spatial locationssimultaneously.

Original: As an optimization algorithm we use stochastic gradient descent with momentum.
Revised: As an optimization algorithm we use stochastic gradient descent with momentum.

Original: Further details of the optimization can be found in the supplementary materials.
Revised: Further details of the optimization can be found in the supplementary materials.

Original: 2.2 Segmentation inference
Revised: 2.2 Segmentation inference

Original: Computing the probability for any individual pair of locations ( i, j ) to be connected, i.e. computingp ( w ij = 1 | f ) , depends only on the two connected feature vectors
Revised: Computing the probability for any individual pair of locations ( i, j ) to be connected, i.e. computingp ( w ij = 1 | f ) , depends only on the two connected feature vectors

Original: f i and f j :
Revised: f i and f j :

Original: [Equation]
Revised: [Equation]

Original: This inference effectively yields a connectivity measure for each pair of neighboring locations, i.e. asparse connectivity matrix.
Revised: This inference effectively yields a connectivity measure for each pair of neighboring locations, i.e. asparse connectivity matrix.

Original: Given that we did not apply any prior information enforcing continuousobjects or contours, the inferred w ij do not necessarily correspond to a valid segmentation or set ofcontours.
Revised: Given that we did not apply any prior information enforcing continuousobjects or contours, the inferred w ij do not necessarily correspond to a valid segmentation or set ofcontours.

Original: Finding the best ﬁtting contours or segmentation for given probabilities for the w s is anadditional process, which in humans appears to be an attention-dependent serial process [35, 63].
Revised: Finding the best ﬁtting contours or segmentation for given probabilities for the w s is anadditional process, which in humans appears to be an attention-dependent serial process [29, 52].

Original: To evaluate the detected boundaries in computer vision benchmarks, we nonetheless need to convertthe connectivity matrix we extracted into a contour image.
Revised: To evaluate the detected boundaries in computer vision benchmarks, we nonetheless need to convertthe connectivity matrix we extracted into a contour image.

Original: To do so, we use the spectral-clusteringbased globalization method developed by [2].
Revised: To do so, we use the spectral-clusteringbased globalization method developed by [1].

Original: This method requires that all connection weightsbetween nodes are positive.
Revised: This method requires that all connection weights between nodes are positive.

Original: To achieve this, we transform the log-probability ratios for the w ij asfollows: For each image, we ﬁnd the 30% quantile of the values, subtract it from all log-probabilityratios, and set all values below 0 .
Revised: To achieve this, we transform the log-probability ratios for the w ij asfollows: For each image, we ﬁnd the 30% quantile of the values, subtract it from all log-probabilityratios, and set all values below 0 .

Original: 01 to 0 .
Revised: 01 to 0 .

Original: We then compute the smallest eigenvectors of the graph
Revised: We then compute the smallest eigenvectors of the graph

Original: Laplacian as in graph spectral clustering.
Revised: Laplacian as in graph spectral clustering.

Original: These eigenvectors are then transformed back into imagespace and are ﬁltered with simple edge detectors to ﬁnd the ﬁnal contours.Evaluation
Revised: These eigenvectors are then transformed back into imagespace and are ﬁltered with simple edge detectors to ﬁnd the ﬁnal contours.

Original: We implement 3 model types implementing feature extractions of increasing complexity in PyTorch[56]:
Revised: We implement 3 models of increasing complexity in PyTorch (Fig. 2 A-C, [45]):

Original: Pixel value model.
Revised: Pixel value model.

Original: For illustrative purposes, we ﬁrst apply our ideas to the rgb pixel values of animage as features.
Revised: For illustrative purposes, we ﬁrst apply our ideas to the rgb pixel values of animage as features.

Original: This provides us with an example, where we can easily show the feature valuesand connections.
Revised: This provides us with an example, where we can easily show the feature valuesand connections.

Original: Additionally, this model provides an easy benchmark for all evaluations.[Figure] [Figure]
Revised: Additionally, this model provides an easy benchmark for all evaluations.

Original: Linear model.
Revised: Linear model.

Original: As the simplest kind of model that allows learning features, we use a single convolutional deep neural network layer as our feature model.
Revised: As the simplest kind of model that allows learning features, we use a single convolutional deep neural network layer as our feature model.

Original: Here, we use 50 11 ⇥ 11 linear features.
Revised: Here, we use 50 11 ⇥ 11 linear features.

Original: Predseg1 :
Revised: Predseg1 :

Original: To show that our methods work for more complex architecture with non-linearities, weuse a relatively small deep neural network with 4 layers (2 convolutional layers and 2 residual blockswith subsampling layers between them, see supplement for details).
Revised: To show that our methods work for more complex architecture with non-linearities, weuse a relatively small deep neural network with 4 layers.

Original: For each of these architectures, we train 24 different networks with all combinations of the followingsettings: 4 different sizes of neighborhoods (4, 8, 12, or 20 neighbors, see Fig.
Revised: For each of these architectures, we train 24 different networks with all combinations of the followingsettings: 4 different sizes of neighborhoods (4, 8, 12, or 20 neighbors, see Fig.

Original: 1D); 3 different noiselevels (0, 0.1, 0.2) and the two learning objectives.
Revised: 1D); 3 different noiselevels (0, 0.1, 0.2) and the two learning objectives.

Original: As a training set, we used the unlabeled imageset from MS COCO [48], which contains 123,404 color images with varying resolution.
Revised: As a training set, we used the unlabeled imageset from MS COCO [38], which contains 123,404 color images with varying resolution.

Original: To enablebatch processing, we randomly crop these images to 256 ⇥ 256 pixel resolution, but use no otherdata augmentation (See supplementary information for further training details).  
Revised: To enablebatch processing, we randomly crop these images to 256 ⇥ 256 pixel resolution, but use no otherdata augmentation (See supplementary information for further training details).  Evaluation

Original: We want to evaluate whether our models learn meaningful features and segmentations.
Revised: We want to evaluate whether our models learn meaningful features and segmentations.

Original: To do so, weﬁrst analyze the features in the ﬁrst layers of our networks where we can judge whether featuresare representative of biological visual systems.
Revised: To do so, weﬁrst analyze the features in the ﬁrst layers of our networks where we can judge whether featuresare representative of biological visual systems.

Original: In particular, we extract segmentations from ouractivations and evaluate those on the Berkeley Segmentation Dataset [2, BSDS500]
Revised: In particular, we extract segmentations from ouractivations and evaluate those on the Berkeley Segmentation Dataset [1, BSDS500]

Original: 3.1 Learned features
Revised: 3.1 Learned features

Original: Linear Model We ﬁrst analyze the weights in our linear models (Fig 2 A-C).
Revised: Linear Model We ﬁrst analyze the weights in our linear models (Fig 3 A-C).

Original: All instances learnlocal averages and Gabor-like striped features, i.e. spatial frequency and orientation tuned features [Figure] Linear PS1-
Revised: All instances learn

Original: [Figure] PS1- [Figure] with limited spatial extend.
Revised: Gabor-like striped features and local averages.

Original: These features clearly resemble receptive ﬁelds of neurons in primaryvisual cortex.
Revised: These features clearly resemble receptive ﬁelds ofneurons in primary visual cortex.

Original: Additionally, there appears to be some preference for features that weight the red andgreen color channels much stronger than the blue channel, similar to the human luminance channel,which leads to the yellow-blue contrasts in the plots.
Revised: In particular there appears to be some preference for featuresthat weight the red and green color channels much stronger than the blue channel, similar to thehuman luminance channel.

Original: There is some difference between the twolearning objectives though.
Revised: There is some difference between the two learning objectives though.

Original: The position based loss generally leads to lower frequency and somewhatnoisier features.
Revised: The position based loss generally leads to lower frequency and somewhat noisier features.

Original: This could either be due to the higher learning efﬁciency of the factor based loss, i.e.the factor based loss is closer to convergence, or due to a genuinely different optimization goal.
Revised: This couldeither be due to the higher learning efﬁciency of the factor based loss, i.e. the factor based loss iscloser to convergence, or due to a genuinely different optimization goal.

Original: Predseg

Revised: Predseg

Original: In Predseg1, we ﬁrst analyze the layer 0 convolution (Fig. 2D), which has only 3 channelswith 3 ⇥ 3 receptive ﬁelds, which we originally introduced as a learnable downsampling.
Revised: In Predseg1, we ﬁrst analyze the layer 0 convolution (Fig. 3D), which has only 3 channelswith 3 ⇥ 3 receptive ﬁelds, which we originally introduced as a learnable downsampling.

Original: This layerconsistently converges to applying near constant weights over space.
Revised: This layerconsistently converges to applying near constant weights over space.

Original: Additionally, exactly one ofthe channels has a non-zero mean (the 3rd, 1st and 3rd in Fig.
Revised: Additionally, exactly one ofthe channels has a non-zero mean (the 3rd, 1st and 3rd in Fig.

Original: 2D) and the other two take balanceddifferences between two of the channels (red vs green and green vs. blue in the examples).
Revised: 3D) and the other two take balanceddifferences between two of the channels (red vs green and green vs. blue in the examples).

Original: Thisparallels the luminance and opponent color channels of human visual perception.
Revised: Thisparallels the luminance and opponent color channels of human visual perception.

Original: In the second convolution, we observe a similar pattern of oriented ﬁlters and local averages as inthe linear model albeit in false color as the input channels are rotated by the weighting of the layer 0convolution
Revised: In the second convolution, we observe a similar pattern of oriented ﬁlters and local averages as inthe linear model albeit in false color as the input channels are rotated by the weighting of the layer 0convolution

Original: (Fig.
Revised: (Fig.

Original: 2 E & F).
Revised: 3 E & F).

Original: 3.2 Contour detection
Revised: 3.2 Contour detection

Original: To evaluate whether the connectivity information extracted by our model corresponds to humanperceived segmentation, we extract contours from our models and compare them to contours reportedby humans for the Berkeley Segmentation database [2, 53].
Revised: To evaluate whether the connectivity information extracted by our model corresponds to humanperceived segmentation, we extract contours from our models and compare them to contours reportedby humans for the Berkeley Segmentation database [1, 43]. This database contains human drawn [Figure] C

Revised: [Figure]

Revised: [Figure]

Original: This database contains human drawnobject boundaries for 500 natural images and is accompanied by methods for evaluating segmentationmodels.
Revised: Canny’s classical edge detector [5], the mean shift algorithm [7], Felzenschwalbs algorithm [11] and segmentation based on normalized cuts [8]. For all comparison algorithms evaluations on BSDS were extracted from the ﬁgure by [1] [Table] object boundaries for 500 natural images and is accompanied by methods for evaluating segmentationmodels.

Original: Using the methods provided with the database, we compute precision-recall curves for eachmodel and use the best F-value (geometric mean of precision and recall) as the ﬁnal evaluation metric.
Revised: Using the methods provided with the database, we compute precision-recall curves for eachmodel and use the best F-value (geometric mean of precision and recall) as the ﬁnal evaluation metric.

Original: As we had multiple models to choose from, we choose the models from each class that performbest on the training data for our reports.
Revised: As we had multiple models to choose from, we choose the models from each class that performbest on the training data for our reports.

Original: For all models this was one of the models with the largestneighborhood, i.e. using 20 neighbors, and the factor loss.
Revised: For all models this was one of the models with the largestneighborhood, i.e. using 20 neighbors, and the factor loss.

Original: It seems the factor loss performedbetter simply due to its technical efﬁciency advantage as discussed above.

Original: Performance increasesmonotonically with neighborhood size and Markov random ﬁeld based approaches to semanticsegmentation also increased their performance with larger neighborhoods up to fully connected

Original: Markov random ﬁelds [43, 8, 9].

Original: We thus expect that larger neighborhoods could work even better.

Original: Qualitatively, we observe that all our models yield sensible contour maps (see Fig.
Revised: Qualitatively, we observe that all our models yield sensible contour maps (see Fig.

Original: 3 A).
Revised: 4 A).

Original: Even thecontours extracted from the pixel model yield sensible contours.
Revised: Even thecontours extracted from the pixel model yield sensible contours.

Original: Additionally, we note that the linearmodel and Layer 1 of the predseg model tend to produce double contours, i.e. they tend to produce [Table] two contours on either side of the contour reported by human subjects with some area between themconnected to neither side of the contour.
Revised: Additionally, we note that the linearmodel and Layer 1 of the predseg model tend to produce double contours, i.e. they tend to producetwo contours on either side of the contour reported by human subjects with some area between themconnected to neither side of the contour.

Original: Quantitatively, our models also perform well except for the deeper layers of Predseg 1 (Fig.
Revised: Quantitatively, our models also perform well except for the deeper layers of Predseg 1 (Fig.

Original: 3B and Table 1).
Revised: 4Band Table 1).

Original: The other models beat most hand-crafted contour detection algorithms that were testedon this benchmark [7, 12, 13, 16] and perform close to the gPb-owt-ucm contour detection andsegmentation algorithm [2] that was the state of the art at the time.
Revised: The other models beat most hand-crafted contour detection algorithms that weretested on this benchmark [5, 7, 8, 11] and perform close to the gPb-owt-ucm contour detection andsegmentation algorithm [1] that was the state of the art at the time.

Original: Layer-0 of Predseg 1 performsbest followed by the linear feature model and ﬁnally the pixel value model.
Revised: Layer-0 of Predseg 1 performsbest followed by the linear feature model and ﬁnally the pixel value model.

Original: Interestingly, the bestperforming models seem to be mostly the local averaging models (cf.
Revised: Interestingly, the bestperforming models seem to be mostly the local averaging models (cf.

Original: Fig.
Revised: Fig.

Original: 2 C).
Revised: 3 C).

Original: In particular, thehigh performance of the ﬁrst layer of Predseg 1 is surprising, because it uses only 3 ⇥ 3 pixel localcolor averages as features.
Revised: In particular, thehigh performance of the ﬁrst layer of Predseg 1 is surprising, because it uses only 3 ⇥ 3 pixel localcolor averages as features.

Original: Since the advent of deep neural network models, networks trained to optimize performance onimage segmentation have reached much higher performance on the BSDS500 benchmark, essentiallyreaching perfect performance up to human inconsistency [e.g. 27, 39, 49, 50, 65, 71, 83, see Table 1].
Revised: Since the advent of deep neural network models, networks trained to optimize performance onimage segmentation have reached much higher performance on the BSDS500 benchmark, essentiallyreaching perfect performance up to human inconsistency [e.g. 22, 33, 39, 40, 54, 60, 69, see Table 1].

Original: However, these models all require direct training on human reported contours and often use featureslearned for other tasks.
Revised: However, these models all require direct training on human reported contours and often use featureslearned for other tasks.

Original: There are also a few deep neural network models that attempt unsupervisedsegmentation [e.g. 10, 47, 82], but we were unable to ﬁnd any that were evaluated on the contourtask of BSD500.

Original: The closest is perhaps the W-net [82], which used an autoencoder structure withadditional constraints and was evaluated on the segmentation task on BSDS500 performing slighlybetter than gPb-owt-ucm.

Original:  Discussion
Revised:  Discussion

Original: We present a model that can learn features and local segmentation information from images withoutfurther supervision signals.
Revised: We present a model that can learn features and local segmentation information from images withoutfurther supervision signals.

Original: This model integrates the prediction task used for feature learning and thesegmentation task into the same coherent probabilistic framework.
Revised: This model integrates the prediction task used for feature learning and thesegmentation task into the same coherent probabilistic framework.

Original: This framework and the dual usefor the connectivity information make it seem sensible to represent this information.
Revised: This framework and the dual usefor the connectivity information make it seem sensible to represent this information.

Original: Furthermore,the featureslearned by our models resemble receptive ﬁelds in the retina andprimary visual cortexand the contours we extract from connectivity information match contours drawn by human subjectfairly well, both without any training towards making them more human-like.
Revised: Furthermore,the features and the contours we extract from connectivity information match contours drawn by human subjectfairly well, both without any training towards making them more human-like.

Original: To improve biological plausibility, all computations in our model are local and all units are connectedto the same small, local set of other units throughout learning and inference, which matches earlyvisual cortex, in which the lateral connections that follow natural image statistics are implementedanatomically [6, 31, 59, 70].
Revised: To improve biological plausibility, all computations in our model are local and all units are connectedto the same small, local set of other units throughout learning and inference, which matches earlyvisual cortex, in which the lateral connections that follow natural image statistics are implementedanatomically [4, 26, 48, 59].

Original: This in contrast to other ideas that require ﬂexible pointers to arbitrarylocations and features [as discussed by 64] or capsules that ﬂexibly encode different parts of the input[14, 42, 61, 62].
Revised: This in contrast to other ideas that require ﬂexible pointers to arbitrarylocations and features [as discussed by 53] or capsules that ﬂexibly encode different parts of the input[9, 36, 50, 51].

Original: Nonetheless, we employ contrastive learning objectives and backpropagation here, for which we do not provide a biologically plausible implementations.
Revised: Nonetheless, we employ contrastive learning objectives and backpropagation here,for which we do not provide a biologically plausible implementations.

Original: However, there is currentlyactive research towards biologically plausible alternatives to these algorithms [e.g. 32, 84].
Revised: However, there is currentlyactive research towards biologically plausible alternatives to these algorithms [e.g. 27, 70].

Original: Selecting the neurons that react to a speciﬁc object appears to rely on some central resource [72, 73]and to spread gradually through the feature maps [34, 35, 63].
Revised: Selecting the neurons that react to a speciﬁc object appears to rely on some central resource [62, 61]and to spread gradually through the feature maps [28, 29, 52].

Original: We used a computer vision algorithmfor this step, which centrally computes the eigenvectors of the connectivity graph Laplacian [2],which does not immediately look biologically plausible.
Revised: We used a computer vision algorithmfor this step, which centrally computes the eigenvectors of the connectivity graph Laplacian [1],which does not immediately look biologically plausible.

Original: However, a recent theory for hippocampalplace and grid cells suggests that these cells compute the same eigenvectors of a graph Laplacianof a prediction network, albeit of a successor representation, i.e. of predictions of the animals statetransitions [68, 69].
Revised: However, a recent theory for hippocampalplace and grid cells suggests that these cells compute the same eigenvectors of a graph Laplacianof a prediction network, albeit of a successor representation, i.e. of predictions of the animals statetransitions [57, 58].

Original: Thus, this might be an abstract description of an operation brains are capable of.
Revised: Thus, this might be an abstract description of an operation brains are capable of.

Original: In particular, earlier accounts that model the selection as a marker that spreads to related locations[e.g.
Revised: In particular, earlier accounts that model the selection as a marker that spreads to related locations[e.g.

Original: 17, 58, 67] have some similarities with iterative algorithms to compute eigenvectors.
Revised: 12, 56] have some similarities with iterative algorithms to compute eigenvectors.

Original: Originally,phase coherence between the neurons encoding the same object was proposed [17, 57, 67], but a gainincrease with object based attention [58] or a known random modulation is also sufﬁcient to select atask relevant set of neurons [25, 26].
Revised: Originally,phase coherence between the neurons encoding the same object was proposed [12, 46, 56], but a gainincrease with object based attention [47] or a known random modulation is also sufﬁcient to select atask relevant set of neurons [20, 21].

Original: Regardless of the mechanistic implementation of the marker,connectivity information of the type our model extracts would be extremely helpful to explain thegradual spread of object selection.
Revised: Regardless of the mechanistic implementation of the marker,connectivity information of the type our model extracts would be extremely helpful to explain thegradual spread of object selection [12, 46].

Original: Our implementation of the model is not fully optimized, as it is meant as a proof of concept.
Revised: Our implementation of the model is not fully optimized, as it is meant as a proof of concept.

Revised: If onewanted to apply our type of model to large scale tasks such as ImageNet [49] or MS Coco [38], onewould have to scale up the networks we use substantially and presumably use some of the additionscontrastive learning methods have used for better learning [6, 10, 17, 23, 24, 63] including: imageaugmentations to explicitly train networks to be invariant to some image changes, prediction headsthat allow more complex shapes for the predictions between locations, and memory banks or othermethods to decrease the reliance on many negative samples.

Original: Inparticular, we did not  optimize the architectures or training parametersof our networks for thetask, like initialization, optimization algorithm, learning rate, or regularization.
Revised: Also, we did not fully optimize the basictraining parameters for our networks, like initialization, optimization algorithm, learning rate, orregularization.

Original: Presumably, betterperformance in all benchmarks could be reached by adjusting any or all of these parameters.
Revised: Presumably, better performance in all benchmarks could be reached by adjusting anyor all of these parameters.

Original: One possible next step for our model would be to train deeper architectures, such that the featurescould be used for complex tasks like object detection and classiﬁcation.

Original: Contrastive losses like theone we use here are successfully applied for such pretraining purposes even for large scale tasks suchas ImageNet [60] or MS Coco [48].

Original: These large scale applications often require modiﬁcations forbetter learning though [11, 15, 22, 28, 29, 75].

Original: For example: Image augmentations to explicitly trainnetworks to be invariant to some image changes, prediction heads that allow more complex shapesfor the predictions, and memory banks or other methods to decrease the reliance on many negativesamples.

Original: Similar modiﬁcations might be necessary to apply our formulation to deeper architecturesfor pretraining purposes.

Original: For understanding human vision, this line of reasoning opens the excitingpossibility that higher visual cortex could be explained based on similar principles, as representationsfrom contrastive learning also yield high predictive power for these cortices [86].

Original: The model we propose here is a probabilistic model of the feature maps.
Revised: The model we propose here is a probabilistic model of the feature maps.

Original: One implication of thisis that we could also infer the feature values if they were not ﬁxed based on the input.
Revised: One implication of this isthat we could also infer the feature values if they were not ﬁxed based on the input.

Original: Thus, ourmodel implies a pattern how neurons should combine their bottom-up inputs with predictions fromnearby other neurons, once we include some uncertainty for the bottom-up inputs.
Revised: Thus, our modelimplies an interesting pattern how neurons should combine their bottom-up inputs with predictionsfrom nearby other neurons, once we include some uncertainty for the bottom-up inputs.

Original: In particular, thecombination ought to take into account which nearby neurons react to the same object and whichones do not.
Revised: In particular,the combination ought to take into account which nearby neurons react to the same object and whichones do not.

Original: Investigating this pooling could provide insights and predictions for phenomena thatare related to local averaging like crowding for example [3, 18, 30, 77–79], where summary statisticmodels currently capture perceptual limitations best [3, 18, 78], but deviations from these predictionssuggest that object boundaries change processing [30, 77, 79].
Revised: Investigating this pooling could provide insights and predictions for phenomena that arerelated to local averaging like crowding for example [2, 13, 25, 65, 66, 67], where summary statisticmodels currently capture perceptual limitations best [2, 13, 66], but deviations from these predictionssuggest that object boundaries change processing [25, 65, 67].

Original: Another promising extension of our model would be processing over time, because predictions overtime were found to be a potent signal for contrastive learning [15] and because coherent object motionis among the strongest grouping signals for human observers [38] and computer vision systems [85].
Revised: Another promising extension of our model would be processing over time, because predictions overtime were found to be a potent signal for contrastive learning [10] and because coherent object motionis among the strongest grouping signals for human observers [32] and computer vision systems[71].

Original: Beside the substantial increases in  processing capacity necessary to move to video processing insteadof image processing, this step would require some extension of our framework to include objectmotion into the prediction.
Revised: Beside the substantial increases in necessary processing capacity necessary to move to videoprocessing instead of image processing, this step would require some extension of our framework toinclude object motion into the prediction.

Original: Nonetheless, including processing over time seems to be an interestingavenue for future research, especially because segmentation annotations for video are extremelyexpensive to collect such that unsupervised learning is particularly advantageous and popular inrecent approaches [1, 33, 44].
Revised: Nonetheless, including processing over time seems to bean interesting avenue for future research.

Original: This work aims to move us closer to understanding how human visual perception can take objectstructure into account in retinotopic feature map processing and may help us to build systems withsimilar capabilities in the future.
Revised: This work aims to move us closer to understanding how human visual perception can take objectstructure into account in retinotopic feature map processing and may help us to build systems withsimilar capabilities in the future.

Original: We acknowledge that such technological progress can have unknownsocietal consequences, but we do not foresee  speciﬁc negative consequences of this work.
Revised: We acknowledge that scientiﬁc and technological progress can haveunknown societal consequences, but we do not foresee any speciﬁc negative consequences of thiswork.