L EARNING A NEURAL RESPONSE METRIC FOR RETINAL PROSTHESIS
Anonymous authors
Paper under double-blind review
A BSTRACT
Retinal prostheses for treating incurable blindness are designed to electrically stimulate surviving retinal neurons, causing them to send artiﬁcial visual signals to the brain.
However, electrical stimulation generally cannot precisely reproduce normal patterns of neural activity in the retina.
Therefore, an electrical stimulus must be selected that produces a neural response as close as possible to the desired response.
This requires a technique for computing a distance between the desired response and the achievable response that is meaningful in terms of the visual signal being conveyed.
Here we propose a method to learn such a metric on neural responses, directly from recorded light responses of a population of retinal ganglion cells (RGCs) in the primate retina.
The learned metric produces a measure of similarity of RGC population responses that accurately reﬂects the similarity of the visual input.
Using data from electrical stimulation experiments, we demonstrate that this metric may improve the performance of a prosthesis.
I NTRODUCTION
An important application of neuroscience research is the development of electronic devices to replace the function of diseased or damaged neural circuits (Wilson et al., 1991; Schwartz, 2004).
Artiﬁcial vision has been a particularly challenging modality due to the richness of visual information, its diverse uses in perception and behavior, and the complexity of fabricating a physical device that can interface effectively with neural circuitry (Stingl et al., 2013; Wilke et al., 2011; Jepson et al., 2014a).
The most advanced example is a retinal prosthesis: a device that replaces the function of neural circuitry in the retina lost to degenerative disease.
Most of the computational work related to this application has focused on building encoding models that use the visual image to accurately predict the spiking activity of populations of retinal ganglion cells (RGCs), the output neurons of the retina that convey visual information to the brain.
Leading models include linear models (Chichilnisky, 2001), probabilistic point-process models (Pillow et al., 2008) and recently proposed models employing rich nonlinearities (McIntosh et al.
; Batty et al.
; Shah et al., 2017).
However, an accurate encoding model, although valuable, is insufﬁcient.
Any retinal prosthesis – whether based on electrical stimulation (Sekirnjak et al., 2008) or optical stimulation (Boyden et al., 2005; Bernstein et al., 2008) – is limited in its ability to create arbitrary desired patterns of neural activity, due to inefﬁciencies or lack of speciﬁcity in the stimulation modality (Barrett et al., 2014;
Jepson et al., 2014a).
Thus, a given stimulation system can only achieve a limited vocabulary of elicited spike patterns.
Hence, although a powerful and accurate encoding model might indicate that a particular spike pattern would be the natural biological response to the incident visual stimulus, the desired spike pattern might not reside within the feasible set of the stimulation device.
Previous studies (Jepson et al., 2014b) have addressed this problem by selecting the electrical stimulation which minimizes the number of unmatched spikes across cells (Hamming distance).
However, the goal of a prosthesis should instead be to select electrical stimulation parameters that produce a response which is most similar to the desired pattern of activity in terms of the visual sensation it produces (Figure 1).
The similarity measure ideally would capture – perhaps implicitly –
the statistics of relevant visual stimuli and the signal and noise properties of individual neurons
and populations of neurons (Shlens et al., 2009; Pillow et al., 2008; Field & Chichilnisky, 2007).
This measure should expand upon previously developed spike train metrics that trade off spike count,
[Figure]
R ˆ r r
C A
OFF
[Figure]
temporal precision and neural identity, and are easy to compute (Victor & Purpura, 1996; 1997;
Victor, 2005).
Here we present an efﬁcient, learned method for selecting the electrical stimulation pattern that produces the optimal pattern of neural activity achievable within the constraints of the device (Figure 1B and C), where optimality is assessed in terms of the visual stimulus associated with the response.
We ﬁrst build a new spike train similarity measure that is expensive to learn but cheap to use in real time, and captures the statistics of visual images and neural responses.
We ﬁt this model to recordings of populations of RGCs in non-human primate retina, and demonstrate that it provides an intuitive, meaningful representation of the similarity between spike patterns in the RGC population.
Finally, we use data from electrical stimulation experiments to demonstrate how the approach can lead to more faithful reproduction of the visual image.
M ETRIC AND SIMILARITY LEARNING
In this section we describe the algorithmic framework for learning pseudometrics or similarity measures in neural response space.
We start by introducing notations and conventions that we use throughout the paper.
We use bold face letters to denote vectors and upper case letters to denote matrices.
The element wise product of two matrices A and B of the same size is denoted by A • B =
Tr ( A (cid:62) B ) .
We denote the symmetrizing operator of a square matrix M by sym ( M ) =
12 ( M + M (cid:62) ) .
A single frame of visual stimulus, s , is an image represented as an n × n matrix.
The space of possible stimuli is S ⊂ R n × n .
A sequence s l , . . .
, s m of m − l + 1 frames, where s j ∈ S , is denoted as s l : m .
In order to simplify our notation, we deﬁne the responses of the cells to be a p dimensional vector and the space of possible responses as r ⊆ R p .
Analogously, a sequence of cell activities r t for t = l, . . .
, m is denoted r l : m .
To simplify the presentation below, we conﬁne the visual stimulus to be a single image and the corresponding response of each cell to be a scalar.
2. E MPIRICAL LOSS MINIMIZATION
Given the population response space R , we learn a function h : R × R
→ R which captures invariances in the spiking response when the same stimulus is presented multiple times.
The scoring function h is viewed either as a similarity function or a pseudometric.
To distinguish between the two
[Figure]
i
[Figure]
l
T r i
[Figure]
l
[Figure]
B r r + r  cases, we use d ( · , · ) to denote a pseudometric.
A pseudometric d needs to satisfy
≥
=
During the experiments, repeats of the same sequence of visual stimuli are presented.
The responses collected during the i th presentation (repeat) of visual stimulus are denoted by ( s it , r it ) .
Here s it is the stimulus history which elicits population response r it at time t .
The goal of this approach to learn a metric such that pairs of responses generated during different repeats of the same stimulus are closer, or more similar, than pairs of responses generated by different stimuli.
We slice the data into triplets of the form ( r , r + , r − ) where r and r + are responses of cells to the same stimulus while r − is the response to a different visual stimuli (Figure 2A).
We refer to ( r , r + ) as a positive pair and ( r , r − ) as a negative pair (Figure 2B).
Metric and similarity learning using triplets (Shalev-Shwartz et al., 2004; Weinberger & Saul, 2009) has been used extensively in several domains, such as computer vision for state of the art performance on face recognition (Schroff et al., 2015).
A central theme of this work (Schroff et al., 2015; Oh Song et al., 2016) has been improving metric learning by mining hard-to-classify negatives.
However, it can be computationally demanding to mine such hard negatives.
We found that a much simpler strategy of randomly sampling a common set of negatives for all the positive examples in the batch is very effective.
Hence we ﬁrst sample positive pairs of responses corresponding to random stimulus times and a common set of negative responses generated by stimuli distinct from any stimulus for positive responses.
Hence a batch of triplets is denoted by T = {{ r i , r i + } , { r j − }} .
Given a training set of triplets T the goal is to ﬁnd a pseudometric such that for most ( r i , r i + , { r j − } ) ∈
T the distance between responses of two repeats of same stimulus is smaller than their distance to any of the irrelevant response vectors,
[Equation]
We cast the learning task as empirical risk minimization of the form,
[Equation]
where (cid:96) () is a differential, typically convex, relaxation of the ordering constraints from (1).
We use
[Equation]
In the case of similarity learning, we swap the role of the pairs and deﬁne,
[Equation]
We implemented two parametric forms for distance and similarity functions.
The ﬁrst is a quadratic form where A (cid:23) 0 and
[Equation]
We also implemented and experimented with non-linear representations using a feed-forward convolutional network to deﬁne a mapping φ from the response space R to R p .
Brieﬂy, we employ a hierarchical, convolutional network topology to mirror the translation invariance expected in the receptive ﬁeld organization of the retina.
The convolutional network consists of 595K parameters across 7 layers and we employ batch normalization to speed up training.
Please the Appendix for more details.
We defer the details of network architecture to the appendix.
[Equation]
We learn the parameters by minimizing the loss using Adagrad (Duchi et al., 2011) for quadratic metric and Adam (Kingma & Ba, 2014) for convolutional neural network metric.
For the quadratic metric, we further project A onto the space of positive semi-deﬁnite matrices space after every update.
The projection is performed using singular value decomposition.
Concretely, we rewrite A as, UDU (cid:62) where U is a unitary matrix and D is a diagonal matrix.
We then threshold the diagonal elements of D to be non-negative.
R ESULTS 3. E XPERIMENTAL S ETUP
Spiking responses from hundreds of retinal ganglion cells (RGCs) in primate retina were recorded using a 512 electrode array system (Litke et al., 2004; Frechette et al., 2005).
ON and OFF parasol
RGC types were identiﬁed using visual stimulation with binary white noise and reverse correlation (Chichilnisky, 2001).
Since each analysis requires different stimulus conditions and number of cells, we leave the details of each preparation to the subsequent sections.
For each analysis, spike trains were discretized at the 120 Hz frame rate of the display (bins of 8.33ms), and responses across all the cells for 1 time bin were used to generate each training example.
In the following sections, we quantitatively assess the quality of multiple learned metrics – each metric with increasing complexity – with respect to a baseline (Hamming distance).
First, we assess the quality of the learned metrics with respect to traditional error analysis.
Second, we assess the quality of the learned embeddings with respect to optimal decoding of the stimulus.
Finally, we demonstrate the utility of the learned metric by employing the metric in a real electrical stimulation experiment.
3. Q UANTITATIVE EVALUATION OF LEARNED METRIC SPACE .
The quality of a neural metric is measured by the the ability of the model to determine whether a pair of ﬁring patterns arose from the same visual stimulus or a distinct visual stimulus.
For understanding the properties of learned metric at the scale of large RGC populations, we focus our analysis on responses of a collection of 36 OFF parasol cells and 30 ON parasol cells torepeats of 10s long white noise stimulus clip.
The responses were paritioned into training (ﬁrst 8s) and testing (last 2s) of each trial.
[Figure]
c i
[Figure]
i
[Figure]
r e c a ll
A
B
C
[Figure]
t r ue po s i t i v e r a t
e
[Figure]
We assessed a range of learned embedding models and baselines by employing receiver-operating characteristic (ROC) analysis.
Speciﬁcally, we selected the population ﬁring pattern, r , at a particular offset in time in the experiment (corresponding to a visual stimulus history) and compared this ﬁring pattern to two other types of ﬁring patterns: (1) the ﬁring pattern from the same group of cells at the same time during a second repeated presentation of the stimulus, r + ; and (2) the ﬁring pattern at a distinct, randomly selected time point, r − .
For a given threshold, if the metric results in a correct classiﬁcation of r + as the same stimulus, we termed the result a true positive .
For the same threshold, if an embedding metric incorrectly classiﬁed r − as the same stimulus, we termed it a false positive .
Note that unlike training, we do not choose a common set of negatives for testing.
Figure 3A traces out the trade-off between the false positive rate and true positive rate across a range of thresholds in an assortment of embedding models for neural population activity.
Better models trace out curves that bend to the upper-left of the ﬁgure.
The line of equality indicates a model that is performing at chance.
A simple baseline model of a Hamming distance (red curve) performs least accurately.
Quadratic metric which permit variable weight for each neuron and interaction between pairs of neurons improves the performance further (green curve).
Finally, replacing a quadratic metric for a euclidean distance between embedding of responses using convolutional neural network improves the performance even further (blue curve).
The ROC analysis provides strong evidence that increasingly sophisticated embedding models learn global structure above and beyond a Hamming distance metric.
We also examined how the local structure of the space is captured by the embedding metric by calculating the learned embeddings on a test dataset consisting of 600 repeats each of the 10 different visual stimuli.
We randomly selected a ﬁring pattern r from one presentation of the stimulus, and identiﬁed k nearest neighbors according to our metric, for increasing k .
Among the k nearest neighbors, we assessed precision, i.e. what fraction of the nearest neighbors correspond to 599 other presentations of the same stimulus.
A perfect learned embedding model would achieve a precision of 100% for k ≤ 599 .
We also measured recall, i.e. what fraction of the remaining 599 presentations of the same stimulus are within the k nearest neighbors.
A perfect learned embedding model would achieve recall of k/ 5 .
99% for k ≤ 599 .
Figure 3B highlights the performance of various learned methods across increasing k .
The results indicate that the precision and recall are below an optimal embedding, but the convolutional metric performs better than quadratic and hamming metrics.
Finally, we embed the 600 responses to 10 distinct stimuli using t-SNE (Maaten & Hinton, 2008) with distances estimated using the convolutional metric to visualize the separation between the responses.
We see in Figure 3C, that responses corresponding to same visual stimulus (same color) cluster in the same region of embedding space reﬂecting the ability of the response space metric to discriminate distinct stimuli
L EARNED METRIC CAPTURES STIMULUS SIMILARITY .
[Figure]
de c oded !
s t
i m u l u s
O N !
pa r a s o l
O FF !
pa r a s o l
A learned metric distance to target      de c od
i ng e rr o r
Hamming learned metric distance percentile
B
We train the metric only based on whether a pair of responses are generated by same stimulus or not.
However, a useful embedding model would implicitly capture further visual stimulus dependencies.
We test this by decoding the stimulus linearly from the neural response.
To answer this question, we focus on a spatially localized and overlapping population of 13 RGCs consisting of 6 ON parasol cells and 7 OFF parasol cells (Figure 1B).
We focus on a smaller population of cells in this (and subsequent) section as we can explicitly list out all the possible 2 13 response patterns.
Training data was accrued from RGC responses to 5 repeated presentations of a 5 minute long white noise sequence.
The ﬁrst 4 minutes of each presentation was employed for training; the last minute was employed for testing.
We examined the similarity between the decoded stimulus and the target stimulus, for responses that, according to our learned quadratic metric, were increasingly distant from the target.
Figure 4A bottom left shows the spatial proﬁle of the linear decoding 20ms prior to the target response.
We next calculate the distance of this target ﬁring pattern to all 2 13 ﬁring patterns and rank order them based on the learned metric.
Figure 4A, top rows, shows ﬁring patterns at the 1% , 2 .
5% , 5% and 75% percentiles.
Below these ﬁring patterns are the associated linearly decoded stimuli, and the errors with respect to the target ﬁring pattern.
As we choose patterns farther from the target in terms of our metric, the distance between the decoded stimulus for the chosen ﬁring pattern and target ﬁring pattern systematically increases.
We quantify this observation in Figure 4B by randomly selecting pairs of responses from the test data and calculating the optimal linearly decoded stimulus associated with them (see Methods).
We then plot the RMS distance between the linearly decoded stimuli against the normalized metric distance between the responses.
The decoding error systematically increases as the metric distance between the corresponding responses increases, for both the learned metric (blue) as well the Hamming distance (green).
However, the distances generated by Hamming distance are discrete and therefore provide a less granular representation of the decoding errors associated with the stimuli.
[Figure]
O N pa r a s o l
O FF pa r a s o l de c oded !
s t
i m u l u s
A
C
D
B
3. L EARNED RESPONSE METRIC COULD IMPROVE PERFORMANCE OF RETINAL PROSTHESIS .
Using recorded experimental data, we now show how response metrics could improve retinal prostheses by choosing optimal electrical stimulation patterns.
For a given target response, we use the learned quadratic metric to select the best electrical stimulation pattern and evaluate the effectiveness of this approach by linearly decoding the stimulus from the elicited responses.
Calibration of RGC responses to electrical stimulation patterns was performed by repeating a given electrical stimulation pattern 25 times, at each of 40 current levels, in a random order.
Due to the limited duration of experiments, we focused on stimulation patterns in which only one electrode was active.
The data was spike sorted and spiking probability was computed for each cell by averaging across trials for each electrical stimulation pattern (Mena et al., 2016).
The probability of ﬁring for each cell in response to increasing current amplitude through each electrode was approximated with a sigmoid function.
Since the RGC response to each current stimulation is probabilistic, we evaluate each stimulation pattern by the expected distance between the elicited responses and the target ﬁring pattern.
For a quadratic response metric this can be easily computed in closed form.
Given a response metric, we rank different stimulation patterns based on the expected distance to the target ﬁring pattern.
In
Figure 5A and B (ﬁrst columns)
we show example target response patterns and the corresponding linearly decoded visual stimulus.
We then analyze the best stimulation pattern determined by the learned quadratic metric, and by the Hamming distance.
The responses sampled from the response distributions for the selected stimulation patterns are shown in Figures 5A and B (second and third columns each).
We ﬁnd that the linearly decoded stimuli are closer to the target when the stimulation was chosen via the learned response metric compared to the Hamming distance.
To quantify this behavior, we calculated the mean squared error between the decoded stimuli when the stimulation was chosen using learned metric and the Hamming distance (Figure 5C).
We ﬁnd that the learned metric and Hamming metric identify the same stimulation pattern and hence achieve the same error for 49% for the target responses observed.
However, on 33% of the target responses, the learned metric achieves lower mean squared error then the Hamming distance; conversely, the learned metric has larger MSE then Hamming distance on 18% of the target responses.
The above analysis demonstrates the beneﬁt of using the learned metric over Hamming distance to choose the best stimulation pattern.
However, the collection of available electrical stimulation patterns might change over time due to hardware or biophysical constraints.
To assess the improvement under such cases, we next ask how well the learned metric performs relative to Hamming distance if we choose the k th best current pattern using each metric.
(Figure 5D).
Increasing rank order ( k ) in the learned metric leads to higher MSE in terms of a decoded stimulus.
Importantly, the learned metric achieves systematically lower MSE than the Hamming distance across the nearest k ≤ 10 stimulation patterns.
These results indicate that the learned metric systematically selects better electrical stimulation patterns for eliciting reasonably close ﬁring patterns.
D ISCUSSION
The learned metric approach has two major potential implications for visual neuroscience.
It provides a novel method to ﬁnd “symbols” in the neural code of the retina that are similar in the sense that they indicate the presence of similar stimuli Ganmor et al.
We speculate that our method may be more robust to training with small data sets than methods based on mutual information or covariance, a potentially signiﬁcant advantage, particularly for high-dimensional responses (many simultaneously-recorded cells, time-resolved).
Second, as shown above, the metric has application to retinal prosthesis technology, in which imperfect hardware constraints demand that the set of neural responses that can be generated with a device be used to effectively transmit useful visual information.
For this application, a metric on responses that reﬂects visual stimulus similarity could be extremely useful.
Additional techniques may also be helpful in extending our method to data involving many cells, temporal responses, and additional response structure.
For example, using recurrent neural networks (Lipton et al., 2015) to embed responses may help compute distances between spiking patterns consisting of multiple time bins, perhaps of unequal length.
Boosting (Freund & Schapire, 1999) may help combine multiple efﬁciently learned metrics for a smaller, spatially localized groups of cells.
Other metrics may be developed to capture invariances learned by commonly used encoding models (Chichilnisky, 2001; Pillow et al., 2008).
Also, triplet mining techniques (i.e., choosing hard negatives), a commonly used trick in computer vision, may improve efﬁciency (Schroff et al., 2015;
Oh Song et al., 2016).
Novel metrics could also be learned with additional structure in population responses, such as the highly structured correlated activity in RGCs Mastronarde (1983); Greschner et al.
This noise correlation structure may be learnable using negative examples that destroy the noise correlations in data, but preserve light response properties, by taking responses of different cells from different repeats of the stimulus.
Note that the convolutional metric outperforms the quadratic metric at both global (ROC curves) and local (precision recall curves) scales.
However, using current retinal prosthesis technologies, we might be able to resolve information only up to a particular scale.
We think that for current retinal prostheses, capturing global structure is probably of greatest importance, because state-of-the-art technology has a relatively coarse vocabulary for stimulating RGCs (Humayun et al., 2012; Zrenner et al., 2011) (see also Figure 1).
Speciﬁcally, the “nearest” elicited ﬁring pattern is “far” in terms of the corresponding visual stimulus (Figure 5) .
In terms of the proposed learned metric, the nearest feasible ﬁring pattern achievable by electrical stimulation in our experiments is at the 10 th percentile of all possible ﬁring patterns.
In this context, the average closest stimulation pattern, expressed as a percentile of the learned metric distances, provides a valuable benchmark to measure the performance of a prosthesis and how that performance is affected by advances in the underlying hardware and software.
A.1. H YPERPARAMETERS
[Table]
Padding
[Table]
SAME
[Table]
SAME
[Table]
[Table]
SAME
ReLU
[Table]
[Table]
SAME
ReLU
Convolution
[Table]
SAME
[Table]
SAME
[Table]