R OTATION - INVARIANT CLUSTERING OF NEURONAL RESPONSES IN PRIMARY VISUAL CORTEX
A BSTRACT
Similar to a convolutional neural network (CNN), the mammalian retina encodes visual information into several dozen nonlinear feature maps, each formed by one ganglion cell type that tiles the visual space in an approximately shift-equivariant manner.
Whether such organization into distinct cell types is maintained at the level of cortical image processing is an open question.
Predictive models building upon convolutional features have been shown to provide state-of-the-art performance, and have recently been extended to include rotation equivariance in order to account for the orientation selectivity of V1 neurons.
However, generally no direct correspondence between CNN feature maps and groups of individual neurons emerges in these models, thus rendering it an open question whether V1 neurons form distinct functional clusters.
Here we build upon the rotation-equivariant representation of a CNN-based V1 model and propose a methodology for clustering the representations of neurons in this model to ﬁnd functional cell types independent of preferred orientations of the neurons.
We apply this method to a dataset of 6000 neurons and visualize the preferred stimuli of the resulting clusters.
Our results highlight the range of non-linear computations in mouse V1.
[Figure]
I NTRODUCTION
A compact description of the nonlinear computations in primary visual cortex (V1) is still elusive.
Like in the retina (Baden et al., 2016; Sanes & Masland, 2015), such understanding could come from a functional classiﬁcation of neurons.
However, it is currently unknown if excitatory neurons in V1 are organized into functionally distinct cell types.
It has recently been proposed that predictive models of neural responses based on convolutional neural networks could help answer this question (Klindt et al., 2017; Ecker et al., 2019).
These models are based on a simple principle (Fig.
1- 1 (cid:13) ): learn a core (e.g. a convolutional network) that is shared among all neurons and provides nonlinear features Φ( x ) , which are turned into predictions of neural responses by a linear readout for each neuron (Antolík et al., 2016).
Models based on this basic architecture exploit aspects of our current understanding of V1 processing.
First, convolutional weight sharing allows us to characterize neurons performing the same computation but with differently located receptive ﬁelds by the same feature map (Klindt et al., 2017; McIntosh et al., 2016;
Kindel et al., 2019; Cadena et al., 2019).
Second, V1 neurons can extract local oriented features such as edges at different orientations, and most low-level image features can appear at arbitrary orientations.
Therefore, Ecker et al.
(2019) proposed a rotation-equivariant convolutional neural network model of V1 that extends the convolutional weight sharing to the orientations domain.
The basic idea of previous work (Klindt et al., 2017; Ecker et al., 2019) is that each convolutional feature map could correspond to one cell type.
While this idea is conceptually appealing, it hinges on the assumption that V1 neurons are described well by individual units in the shared feature space.
However, existing models do not tend to converge to such solutions.
Instead, V1 neurons are better described by linearly combining units from the same spatial location in multiple different feature maps (Ecker et al., 2019).
Whether or not there are distinct functional cell types in V1 is therefore still an open question.
Here, we address this question by introducing a clustering method on rotation-equivariant spaces.
We treat the feature weights (Fig.
1- 1 (cid:13) ) that map convolutional features to neural responses as an approximate low-dimensional vector representation of this neuron’s input-output function.
We then split neurons into functional types using a two-stage procedure: ﬁrst, because these feature weights have a rotation-equivariant structure, we ﬁnd an alignment that rotates them into a canonical orientation (Fig.
1- 2 (cid:13) ); in a second step, we cluster them using standard approaches such as k-means or Gaussian mixture models (Fig.
1- 3 (cid:13) ).
We apply our method to the published model and data of
Ecker et al.
(2019) that contains recordings of around 6000 neurons in mouse V1 under stimulation with natural images.
Our results suggest that V1 neurons might indeed be organized into functional clusters.
The dataset is best described by a GMM with around 100 clusters, which are to some extent redundant but can be grouped into a smaller number of 10–20 groups.
We analyse the resulting clusters via their maximally exciting inputs (MEIs) (Walker et al., 2018) to show that many of these functional clusters do indeed correspond to distinct computations.
R ELATED WORK
Unsupervised functional clustering via system identiﬁcation As outlined in the introduction, our work builds directly upon the methods developed by Klindt et al.
(2017) and Ecker et al.
While these works view the feature weights as indicators assigning each neuron to its ‘cell type’ (feature map), we here take a different view on the same model: rather than focusing on the convolutional features and viewing them as cell types, we treat the feature weights as a low-dimensional representation of the input-output function of each neuron and perform clustering in this space.
This view on the problem has the advantage that there is no one-to-one correspondence between the number of feature maps and the number of cell types and we disentangle model ﬁtting from its interpretation.
On the other hand, our approach comes with an addition complexity: because the feature weights obey rotational equivariance and we would like our clustering to be invariant to rotations, we require a clustering algorithm that is invariant with respect to a class of (linear) transformations.
A number of authors have developed clustering methods that are invariant to linear (Tarpey, 2007), afﬁne (Brubaker & Vempala, 2008) or image transformations by rotations, scalings and translations (Frey & Jojic, 2002).
Ju et al.
(2019) cluster natural images by using a
CNN to represent the space of invariant transformations rather than specifying it explicitly.
Alignments Instead of using custom clustering algorithms that are invariant under certain transformations, we take a simpler approach: we ﬁrst transform our features such that they are maximally aligned using the class of transformations the clustering should be invariant to.
This approach has been used in other contexts before, usually by minimizing the distances between the transformed observations.
Examples include alignment of shapes in R d using rigid motions (Gower, 1975; Dryden & Mardia, 1998), alignment of temporal signals by ﬁnding monotonic input warps (Zhou & De la Torre, 2012), or alignment of manifolds with the distance between the observations being deﬁned according to the metric on the manifold (Wang & Mahadevan, 2008; Cui et al., 2014).
There is also work on alignment objectives beyond minimizing distances between transformed observations, examples of which include objectives based on generative models of observations (Kurtek et al., 2011; Duncker & Sahani, 2018) or probabilistic ones which are particularly suited for alignment with multiple groups of underlying observations (Kazlauskaite et al., 2019).
R OTATION - EQUIVARIANT CLUSTERING
Our goal is to cluster neurons in the dataset into groups performing similar computations.
To do so, we use their low-dimensional representations obtained from the published rotation-equivariant CNN of Ecker et al.
(2019), which predicts neural activity as a function of an external image stimulus.
We brieﬂy review this model before describing our approach to rotation-invariant clustering.
CNN model architecture The model consists of two parts (Fig.
1- 1 (cid:13) ):
A convolutional core that is shared by all neurons and computes feature representations Φ( x ) ∈ R W × H × K , where x is the input image, W × H is the spatial dimensionality and K is the number of feature maps.
A separate linear readout w n = s n ⊗ r n ∈ R W × H × K for each neuron n = 1 , . .
N , factorized into a spatial mask s n and a vector of feature weights r n .
The predicted activity of a neuron n for image x is
[Equation]
where f ( · ) is a non-linear activation function.
Such a CNN therefore provides K -dimensional feature weights r n characterising linear combinations of spatially weighted image features s n · Φ( x ) that are predictive of neural activity.
We treat these feature weights as ﬁnite-dimensional proxies of actual computations implemented by the neurons.
Because masks s n (another component of readouts w n deﬁned above) are irrelevant for our analysis, we will often refer to r n simply as readout vectors.
We will be referring to the matrix having r i as its rows as the readout matrix R ∈ R N × K .
Rotation-equivariant core Feature representations Φ( x ) are rotation-equivariant, meaning that weight sharing is not only applied across space but also across rotations: for each convolutional ﬁlter there exist O rotated copies, each rotated by 2 π/O .
Feature vectors φ ij ( x ) at position ( i, j ) therefore consist of F different features, each computed in O linearly-spaced orientations (such that F × O = K ).
We can think of φ ij ( x ) as being reshaped into an array of size F × O .
Having computed φ ij ( x ) , we can compute φ ij ( x (cid:48) ) with x (cid:48) being a stimulus x rotated around ( i, j ) by 2 π/O by cyclically shifting the last axis of φ ij ( x ) by one step (this mechanism is illustrated in Fig. 2).
Rotation-equivalent computations
The linear readout adheres to the same rotation-equivariant structure as the core.
As our goal is to cluster the neurons by the patterns of features they pool while being invariant to orientation, we need to account for the set of weight transformations that correspond to a rotation of the stimulus when clustering neurons.
We illustrate this issue with a small toy example consisting of six neurons that fall into two cell types (Fig. 2B).
Within each cell type (columns in Fig.
2B), the individual neurons differ only in their orientation.
More formally, we deﬁne the computations performed by two neurons n i and n j to be rotation-equivalent if there exists a rotation ψ ij such that for any input stimulus x we have a n i ( x )
= a n j ( ψ ij ( x )) .
We will refer to such neurons as rotation-equivalent as well.
[Table] [Figure] Readouts of rotation-equivalent neurons Directly clustering the readout matrix R does not respect the rotation equivalence, because readout vectors of neurons implementing rotation-equivalent computations are not identical (Fig. 2).
To address that, we ﬁrst modify R to obtain a matrix ˜ R with the rows corresponding to rotation-equivalent neurons aligned to a canonical form, and then cluster ˜ R to obtain functional cell types.
Rotating an input x by a multiple of 2 π/O corresponds to cyclically shifting Φ( x ) , so the readout vectors of two rotation-equivalent neurons whose orientation difference ψ ij is a multiple of 2 π/O are also cyclic shifts of each other (Fig. 2).
For arbitrary rotations that are not necessarily a multiple of 2 π/O , we assume the readout r n j of neuron n j to be a linear interpolation of cyclic shifts of r n i corresponding to the two nearest rotations which are multiples of 2 π/O .
Formally, we deﬁne a cyclic rotation matrix by an angle α ∈ [0 , 2 π ) as follows (matrix has shape O × O ; column indices are shown above the matrix):
[Equation]
Given a readout vector r n ∈ R K , we can think of it as a matrix r n ∈ R O × F with columns corresponding to readout coefﬁcients for different orientations of a single feature.
The cyclic rotation of a readout r n by an angle α ∈ [0 , 2 π ) can be expressed as a matrix multiplication r n ( α ) =
S α r n .
Note, by writing r n ( α ) as a function of α we refer to cyclically rotated (transformed) readouts, while r n are the ﬁxed ones coming from a pre-trained CNN and r n (0) =
r n .
For two rotation-equivalent neurons n i and n j , the readout vector r n j can be computed as a cyclic rotation of r n i by α , which is the rotation angle of ψ ij .
If α is a multiple of 2 π/O ; otherwise it is only an approximation which becomes increasingly accurate as O increases.
Aligning the readouts Assuming V1 neurons form discrete functional cell types, all neurons in the dataset (and hence the readouts characterising them) can be partitioned into non-overlapping classes w.r.t.
the rotation equivalence relation we introduced above.
Choosing one representative of each class and replacing the rows of R with their class representatives, we can obtain ˜ R from R .
Next, we discuss an algorithm for ﬁnding such representatives of each class.
We claim that by minimising the sum of pairwise distances between the cyclically rotated readouts
[Equation]
we can transform each readout into a representative of a corresponding equivalence class (same for all readouts of a class), i.e. r i ( α ∗ i ) =
r j ( α ∗ j ) if neurons i and j are rotation-equivalent.
This is indeed the case because the readouts of the same equivalence class lie on the orbit obtained by cyclically rotating any representative of that class.
Such angles { α ∗ i } that neurons of the same class end up on the same point on the orbit (i.e. aligned to the same class representative) clearly minimise
Eq.
(3), and since different orbits do not intersect (they are different classes of equivalence), readouts of different equivalence classes cannot end up at the same point.
Note that the resulting class representatives are not arbitrary, but those with the smallest sum of distances between each other in the Euclidean space.
This mechanism is illustrated in Fig.
1- 2 (cid:13) .
Clustering aligned readouts Having obtained ˜ R with rows r i ( α ∗ i ) , we can cluster the rows of this matrix using any standard clustering method (e.g. K-Means, GMM, etc.) to obtain groups of neurons (cell types) performing similar computations independent of their preferred orientations.
The rotationinvariant clustering described above is based on solving the optimisation problem in Eq.
To do so, we would typically use a gradient-based optimisation, which is prone to local minima because of the way we deﬁne cyclic rotations in Eq.
According to that deﬁnition, a rotated readout is a linear combination of two nearest base rotations, or rather a linear combination of all such rotations with only two coefﬁcients being non-zero.
That means that gradients of all but two coefﬁcients w.r.t.
the angle α are zero, and the optimisation would converge to a local minimum corresponding to the best linear combination of the two base rotations initialised with non-zero coefﬁcients (Fig. 3).
To address this issue, we propose to approximate Eq.
(2), such that r n i ( α ) is a linear combination of all base rotations with nonzero coefﬁcients, with the coefﬁcients for the two nearest base rotations being the largest.
Speciﬁcally we compute the coefﬁcients by sampling the von Mises density at ﬁxed orientations to ensure cyclic boundary conditions and deﬁne ˜
r n i ( α ) , a continuous relaxation of r n i ( α ) , as ˜ r n i ( α ) =
˜ S
α r n i where
[Equation]
The parameter T ≥ 0 controls the sparseness of coefﬁcients { γ i } .
For small T , many of the coefﬁcients are signiﬁcantly greater than zero, allowing the optimiser to propagate the gradients and reduce the effect of initialisation.
As T increases, ˜ r n i ( α ) becomes more similar to r n i ( α ) , and the rotations by multiples of 2 π/O are recovered.
In the limit, ˜ r n i (2 πk/O ) → r n i (2 πk/O ) as T → ∞ (Fig. 3).
Instead of ﬁxing T , we learn it by optimising the regularised alignment objective with additional reconstruction loss preventing trivial solutions (e.g. all coordinates of ˜
r i ( α i ) being the same for T = 0 ):
[Equation]
[Figure]
β =β = 0 .
B
E XPERIMENTS
Synthetic dataset We generate a small toy dataset consisting of 16 hypothetical neurons (readouts) of two cell types to illustrate the proposed alignment method.
Each readout consists of just one feature in eight base orientations (linearly spaced between 0 and 7 π/ 4 ) and is generated by one of the two underlying types of readouts cyclically shifted by a random angle φ ∈ [0 , 2 π ) .
To generate such a dataset , we draw two independent noiseless functions from a Gaussian process (GP) with a periodic kernel (with period 2 π ), then for each readout we randomly choose one of the two GP samples, shift it by an angle φ and evaluate the shifted function at the base orientations to obtain an 8-dimensional vector modelling the observed readout values.
This process is illustrated in Fig.
4A.
Neural data We use the same dataset as in Ecker et al.
(2019), consisting of simultaneous recordings of responses of 6005 excitatory neurons in mouse primary visual cortex (layers 2/3 and 4).
Model details We analyse a rotation-equivariant CNN consisting of a three-layer core withfeatures in 8 orientations in each layer (kernel sizes 13, 5, 5) and 128-dimensional readouts ( F
= 16 ,
O = 8 ).
We use the pre-trained model provided by Ecker et al.
We align the readout matrix
R by minimising Eq.
(5) w.r.t.
the rotation angles α i and temperature T .
We ﬁt models forlog-spaced values of β in [0 . 001 , 10] , and choose for analysis the one with the smallest alignment loss (Eq.
(3)) among the models with optimised temperature T > 5 .
We use Adam (Kingma & Ba, 2015) with early stopping and initial learning rate of 0.01 decreased three times.
Clustering aligned readouts We use the Gaussian mixture model implemented in scikit-learn (Pedregosa et al., 2011) for clustering the aligned readouts ˜ R .
We use spherical covariances to reduce the number of optimised parameters.
To obtain a quantitative estimate of the number of clusters in ˜ R , we randomly split the dataset of 6005 neurons into training (4000 neurons) and test (2005 neurons) sets, ﬁt GMMs with different numbers of clusters on the training set, and then evaluate the likelihood of the ﬁtted model on the test set.
R ESULTS 5. S YNTHETIC DATA SET ALIGNMENT
We start by demonstrating on a synthetic dataset (Sec. 4) that optimising Eq.
(5) can successfully align the readouts (Fig. 4A), assuming β has been chosen appropriately.
Note that readouts have been shifted by arbitrary angles (not multiples of π/ 4 as demonstrated for readouts in colored boxes in Fig. 4A), and they are aligned precisely via interpolation Eq.
We can also see the effect of the parameter β , controlling the relative weight of the reconstruction term (i.e. similarity of readouts rotated by 0 degrees to the observations).
Small values of β incur a small cost for poor reconstructions resulting in small optimised values of T and over-smoothed aligned readouts.
We next ask whether the alignment procedure still works in the presence of observational noise (Fig. 4B).
For small to moderate noise levels (Fig.
4B, left), alignment reduces the pairwise distances to the level expected from the observation noise (shown at the top), conﬁrming the visual impression (shown at the bottom) that alignment works well.
For high noise levels (Fig.
4B, right), alignment breaks down as expected, and we observe overﬁtting to the noise patterns (shown by the pairwise distances after alignment dropping below the level expected from observation noise alone).
[Figure]
Clustering We evaluate the GMM used to cluster ˜ R for different numbers of clusters.
The test likelihood starts to plateau at aroundclusters (Fig. 5), so we use 100 clusters in the following.
Visualization of clusters To visualize the clustering result, we compute a two-dimensional t-SNE embedding (van der Maaten & Hinton, 2008) of the matrix of aligned readouts ˜ R , which is coloured according to the GMM clustering of ˜ R with 100 clusters (Fig. 6).
Note that we use the embedding only for visualization, but cluster 128D aligned readouts in ˜ R directly.
In addition to the embeddings, we also visualize the computations performed by some of the clusters by showing maximally exciting inputs (MEIs).
We compute MEIs via activity maximisation (Erhan et al., 2009; Walker et al., 2018) and show the stimuli that maximally drive the 16 best-predicted neurons of each cluster.
We observe that MEIs corresponding to neurons of the same cluster are generally consistent up to rotation and receptive ﬁeld location, suggesting that the proposed clustering method captures the similarities in the neural computations while ignoring the nuisances as desired.
Network learned redundant features We noticed a number of clusters with similar MEIs (e.g.
Block 9 and Block 13 in Fig. 6).
There could be two reasons for this observation: (a) the neural computations corresponding to these clusters could be different in some other aspect, which we cannot tell by inspecting MEIs as they represent only the maximum of a complex function, or (b) the features learned by the CNN could be redundant, i.e. the hidden layers could learn to approximate the same function in multiple different ways.
To answer this question, we compute a cluster confusion matrix (Fig. 7, left), which quantiﬁes how similar the response predictions of different clusters are across images.
The element ( p, q ) corresponds to the correlation coefﬁcient between the predicted responses on the entire training set of hypothetical neurons with cluster means for clusters p and q used as readouts, accounting for potential differences in canonical orientation across clusters.
By greedily re-arranging clusters in the matrix into blocks based on their correlations, we show that the 100 clusters in the model can be grouped into a much smaller number of functionally distinct clusters.
Using a correlation threshold of 0.5 in this re-arrangement procedure, we obtain an example arrangement into 17 blocks (Fig. 7).
Thus, the network has learned an internal representation that allows constructing very similar functions in multiple ways, suggesting that further pruning the learned network before clustering could lead to a more compact feature space for V1.
Finally, to quantify how consistent the resulting 17 groups of clusters are, we compute an MEI confusion matrix (Fig. 7, right panel).
Its ( i, j ) element is the predicted activity of neuron j for the
MEI of neuron i , after accounting for orientation and receptive ﬁeld location (i.e. a j ( y i ) , where y i is the MEI of neuron i moved and rotated such that it optimally drives neuron j ).
We show this matrix using the same grouping as for the cluster confusion matrix above and restrict it to the(out of 6005) best predicted neurons (with test set correlation ≥ 0 . 7 ).
Note that some of the blocks from the cluster confusion matrix do not appear here, indicating that those clusters include poorly predicted neurons (e.g. block 17).
The MEI confusion matrix exhibits a block-diagonal structure, with most neurons within the blocks most strongly, with different degrees of within-block similarity.
C ONCLUSIONS AND FUTURE WORK
We have presented an approach to clustering neurons into putative functional cell types invariant to location and orientation of their receptive ﬁeld.
We ﬁnd around 10–20 functional clusters, the boundaries of some of which are not very clear-cut.
To systematically classify the V1 functional cell types, these proposals need to be subsequently examined based on a variety of biological criteria reﬂecting the different properties of the neurons and the prior knowledge about the experiment.
A R ANDOM PERMUTATIONS OF FEATURES
B S YNTHETIC DATASET : DEPENDENCE ON NOISE
[Figure]