Deep Neural Imputation: A Framework for Recovering Incomplete Brain Recordings
Abstract
We study the problem of time series imputation in multivariate neural recordings.
Compared to standard time series imputation settings, new challenges for imputing neural recordings include the lack of adjacent timestamps for electrodes missing over days, and generalization across days and participants with different electrode configurations.
Due to these challenges, the standard practice in neuroscience is to discard electrodes with missing data, even if only a part of the recording is corrupted, significantly reducing the already limited and difficult-to-obtain data.
In this paper, we establish Deep Neural Imputation (DNI), a framework to recover missing electrode recordings by learning across sessions, spatial locations, and participants.
We first instantiate DNI with natural linear baselines, then develop encoder-decoder approaches based on masked electrode modeling.
We evaluate
DNI on 12 multielectrode, human neural datasets with naturalistic behavior.
We demonstrate DNI’s data imputation ability across a broad range of metrics as well as integrate DNI into an existing neural data analysis pipeline.
Introduction
Multielectrode recordings measure the dynamic activation of large networks of neurons in the brain and are a key enabling tool in studying neural function at scale [1, 2, 3, 4].
In humans, implanting such electrode arrays is an invasive neurosurgery procedure and individual electrode implants can fail at any time.
Therefore, there are almost always missing electrode values in these difficult-to-acquire and highly precious datasets.
The current standard practice is to discard electrodes with missing data, excluding potentially useful information from analysis and limiting model generalizability across days and individuals [5, 2, 6].
To address this challenge, we study the problem of imputing missing values from multielectrode data using recordings collected across multiple days in 12 human datasets (Figure 1).
While related to the generic problem of time-series imputation (e.g., [7, 8], more discussion in Section D), previous works are not directly applicable due to additional challenges associated with multielectrode recordings including: (1) electrodes are often missing for entire sessions with no adjacent recorded timestamps, and (2) existing methods do not handle data across participants who have completely different sets of electrodes.
Our contributions are summarized as follows:
∗ Equal contribution. Workshop on Learning from Time Series for Health, 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
P a r t i c
i p a n t
Problem
Day
V o l t a g e
Time
Local Field Potential Signal
On-Brain Electrode Coordinates
P
P i
P N P
Joint-Participant Electrode Coordinate Space
Participant-Speci ﬁ c
Electrode Locations
P N P
D
D
Neural Time Series
P 1 : Tensor & On-Brain Locations
Data Tensors
Imputed Electrodes Present Electrodes
P
D
D
E l e c t r o d e
D a y I n s t a n c e
E l e c t r o d e
D a y I n s t a n c e
D a y E l e c t r o d e
I n s t a n c e
• We propose the Deep Neural Imputation (DNI) Framework, a method for recovering fully missing multivariate electrode time series recordings across sessions and human participants.
Additionally, we evaluate DNI’s performance in 12 distinct human datasets where each participant performs naturalistic behavior with large inter-participant and inter-day variability [11, 12].
Canonically in neuroscience, only a few hours of a single participant’s recordings would be treated as a distinct dataset and analyzed separately [9, 2, 10].
• We instantiate DNI using multiple linear and nonlinear imputation methods.
The linear methods can be viewed as natural baseline approaches.
The nonlinear DNI methods use a self-supervised task based on mask-filling during training, which we call masked electrode modeling.
Masked electrode modeling is inspired by masking approaches in other domains [13, 7].
We then extend our learning approach to a multi-task setting to jointly model all participants across all 12 datasets.
• We provide experimental evidence of DNI’s direct utility in downstream scientific analyses: (1) DNI reconstructs not only time series content but also frequency based power-spectral content (highly significant to the neuroscience community [14, 15, 16]), and (2) DNI’s spatiotemporal reconstructions directly improve a brain decoder’s classification accuracy when missing data is present.
The DNI Framework
In our problem formulation, participants are indexed by i , recording days are indexed by j , ¯ E i,j represents the set of observed electrodes on day j for participant i , and E i,j represents the full set of active electrodes on day j for participant
i (more discussion in Section A).
The goal of DNI is to learn an imputation function f : ¯ E i,j → E i,j , as depicted in Figure 2 (Left).
f may be either linear or nonlinear, and we study both instantiations.
In the dataset-specific (participant-specific) case, we learn a function f
( i ) for each dataset (participant)
i (Figure 2).
We test these models on a held-out recording day, making them day-generalizable.
In the across-dataset (joint-participant) case, we learn a single function f across all datasets (participants) (Figure 5).
We call this model DOPE-generalizable,
[Figure]
f :
[Figure]
I n f e r e n c e
Model
Model Framework
[Figure]
Series
Inference
Train
[Figure]
Baseline
D
[Figure]
D
[Figure]
Imputed Data
Loss
Incomplete Data
Model Inference CNNAE
Train Model on Masked Electrodes
RandomElectrodeMask
DuringTrainingOnly
Participant-Speci ﬁ c
Deep Neural Imputation
[Figure]
Trained
ElectrodeLegend
D ay-with j O are also day-generalizable.
High-Level Approach.
Since the neural imputation problem is not well-explored, a key design goal is to identify the simplest approach that works well for this domain.
Two salient desiderata for imputation methods are (1) the recovery of missing data without adjacent timestamps using observations from different days; and (2) robustness to variations in brain morphology and physical electrode placement across participants.
Day-generalizable participant-specific modeling addresses (1) but not (2), while DOPE-generalizable joint-participant modeling addresses both (1) and (2).
To create day-generalizable models that address (1), the key idea is to learn conserved relationships between electrodes across recording days.
Conceptually straightforward methods are either the linear all electrode method or the linear nearest neighbors method, which we develop leveraging neuroscience domain knowledge.
Either model is a natural baseline since no other baselines exist.
We also study a nonlinear deep autoencoder model, Convolutional Neural Network AutoEncoder (CNNAE), that we later extend to train jointly over all participants, Multihead CNNAE (M-CNNAE).
The
M-CNNAE model is DOPE-generalizable and additionally addresses (2) (futher detail in Section B).
Experiments
We study DNI with a linear all electrode baseline model, linear nearest neighbors baseline model,
CNNAE, and M-CNNAE on real-world multielectrode recordings from 12 human datasets where participants perform naturalistic behavior across multiple recording days.
We summarize model comparisons (Section 3.1) and demonstrate that DNI recovers significant neural decoding performance with our imputations (Section 3.2).
Additional details and results are presented in the Appendix, including dataset descriptions ( C.1), training and evaluation setups ( C.2), additional results ( C.3), and frequency content evaluations ( C.4), k-NN hyperparameter sweep ( G.5), MSE metrics ( G.6), and the linear all electrode performance( G.1).
3.1 Benchmark Results
Our models perform three categories of signal recovery: reconstruction (compared to observed signal), artificially induced imputation (we mask observed signal for evaluation), and natural imputation (the original signal was missing).
We can perform direct quantitative evaluation of DNI’s signal recovery capabilites with reconstruction and artificially induced imputation (i.e. imputation).
Since there is no ground truth for naturally missing data, we do not evaluate natural imputation.
Time series correlations comparisons between the linear NN, CNNAE, and M-CNNAE demonstrates that deep learning approaches generally have higher correlation (Figure 3).
Prior approaches for imputing missing data in the neuroscience community use zero-filling, resulting in a correlation value of zero.
Pt
Pt
Pt 2 Pt
Pt 4 Pt
Pt 6 Pt
Pt 8 Pt
Pt 10 Pt
Opacity Recon.
Imput.
Symbols
[Figure]
M - C NN
A E
T i m e
S e r i e s
C o rr e l a t i o n
T i m e
S e r i e s
C o rr e l a t i o n
C NN A
E 0.5 0.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
3.2 Downstream Task: Neural Imputation + Neural Decoding
Given neural decoding’s significance in the neuroengineering community[4, 2, 3, 17], we study
DNI’s ability to directly improve neural decoding performance (Figure 4).
We use the movement neural decoder from [18], which intakes neural time series data and predicts whether the neural data corresponds to either an arm movement event or rest.
After recreating the random forest decoding performance for the original data, we randomly zero-filled either 50%, 70%, or 90% of the data for five random sets of missing electrodes and then computed the resulting performance.
We purposefully chose these proportions to test DNI’s limits on downstream applications with highly corrupted or missing data.
Out of the 36 distinct experiments across 12 datasets, our CNNAE-filled data either increased or maintained decoding accuracy in 34/36 experiments (94.44% of the time).
These results suggest that there is value in adopting DNI in the neuroscience and neuroengineering communities.
Conclusions
We presented DNI, a framework for imputing missing values in multivariate time series data tailored to the challenges in neuroscience data.
By recovering previously unusable data, this line of work can amplify the strength of existing neuroscience analysis workflows.
Future work includes pushing towards more powerful unified representations that can further enable joint-participant analysis.
Appendix A Background & Problem Setup
Data Modality & Format.
Our data, further described in Section C.1, consists of participants (indexed by i ), recording days (indexed by j ), 2 electrodes corresponding to spatial locations (indexed by k ), and time series instances (indexed by t ).
In other words, an observation E i,j,k,t corresponds to time series instance t from electrode k on day j for participant i .
Indices that follow others demonstrate dependence; for example, time series instance t depends on the specific electrode k , the specific day j , and the specific participant i .
Here we define generalizable as being able to perform on previously unseen data.
When our models impute data across dimensions j, k, t they generalize across time series instances, spatial locations, and days (Figure 1) and we call them day-generalizable .
When our models impute data across dimension i as well as dimensions j, k, t they additionally share a joint participant embedding space; we call them D ay-with-j O int-
P articipant-
E mbeddings-generalizable, or DOPE-generalizable .
By necessity, DOPE-generalizable models are also day-generalizable.
Challenges in Modeling Electrode Recordings .
Due to significant inter-participant and inter-day variability, data analysis methods in neuroscience commonly treat a few hours from single participants as individual datasets [9, 2, 10] to be analyzed separately.
Figure 1’s right panel demonstrates the stark electrode configuration variability in our participants, illustrating a common phenomenon in this data type.
Further sources of this variability include movement artifacts, insurmountable experimental noise, and electrode recording failure.
For example, in a neuropixel single-unit electrophysiology mouse study, only 55.4% of the data could be used for downstream analyses, because of “recording failure[s],” “low yield,” and “noise/artifact[s]” [5].
In this ECoG LFP human speech decoding study, electrode recordings were discarded from a majority of the participants due to “bad signal quality” [2].
The pervasiveness of missing data is exemplified by the standard data storage file format in neuroscience, Neurodata Without Borders, which explicitly emphasizes its support for “dense ragged arrays” that permit “missing fields” [6].
These ragged data tensors result from the lack of consistency across the dimensions i, j, k, t (Top Left
Figure 1).
Since ragged data are difficult to jointly analyze, they are most commonly broken up into smaller, non-ragged tensors.
This procedure leads to more manageable data for neuroscience analysis pipelines, meaning that discarding data remains a common practice despite its inefficiency.
Because neuroscience data processing pipelines seldom handle variation in the expected data structure, these pipelines fundamentally limit the generalization capability of neuroscience models.
Formal Imputation Goals.
To address the above challenges, our goal is to impute missing electrode values by learning from multielectrode time series data across spatial locations, days, and participants.
Let E i,j = { E i,j,k | k ∈ K i } be the full set of active electrodes on day j for participant i .
K i represents the full set of electrodes for participant i , which remains consistent across days.
Let
M i,j ⊂ K i represent the set of missing electrodes on day j for participant i .
Let ¯ E i,j = { E i,j,k | k ∈
K i \ M i,j } represent the set of observed electrodes.
Formally, DNI seeks to recover E i,j given ¯ E i,j .
Previous work for multivariate time series imputation [7, 19, 8] studies missing data at the level of sequences { t 1 , t 2 , ..., t T } , where there may be missing observations for a set of timestamps (e.g. { t 5 , t 6 , t 7 } ).
However, these methods could not impute when a feature was missing for all timestamps (e.g. { t 1 , t 2 , ..., t T } ); in other words, each feature required at least one observed timestamp.
Therefore, these methods are not directly applicable to ours because an entire time series instance E i,j,k,t = { t 1 , t 2 , ..., t T } may be missing and have no adjacent timestamps.
B The Deep Neural Imputation Framework - Additional Details
Linear All Electrode Baseline .
We trained a linear all electrode model, with the details as follows: let n e denote the total number of electrodes of a participant.
The linear all electrode baseline learns a single set of weights W ∈ R n e × n e and biases b ∈ R n e for each participant.
Suppose that x ∈ R n e represents the measured values (missing values are zeroed out) of all of a participant’s electrodes at some time step.
Then, this baseline outputs ˆ x = Wx + b .
If x i = 0 , then ˆ x i represents the imputed electrode value.
We also set the diagonal values of W to be 0 since we cannot use an electrode’s own value as a feature for imputation during inference.
We train on MSE loss and models were trained until convergence by reconstructing present electrode values and were evaluated by imputing values we randomly zeroed out.
The values we zeroed are the same across all previous models (i.e.
NN-baseline, CNNAE, M-CNNAE) for comparison consistency.
Linear Nearest Neighbors Baseline .
In the absence of appropriate existing baselines, we looked to known properties of neural signals for inspiration.
Correlations exist across spatially close electrode neighbors; this well known observation motivated us to impute missing electrodes with linear combinations of observed, neighboring electrodes.
In particular, we compute time series correlations from the N nearest neighbors of an electrode on observed days (e.g. “training days”).
Then, we use these correlations as weights to linearly combine the time series from missing electrode neighbors on the held out “test day.”
These weighted, linearly combined time series compilations constitute our reconstructed neural signal and form our day-generalizable baseline model.
CNNAE .
In our Convolutional Neural Network AutoEncoder (CNNAE) learning approach (Figure[-
-]
Bottom Right), the encoder f ( i ) θ maps ¯ E i,j with zero-filled missing electrodes to an embedded representation z i,j .
The decoder f
( i ) ψ then maps z i,j to ˆ E i,j , which is either a reconstruction or an imputation (which we distinguish below) of the full set of electrodes E i,j .
For k ∈ K i \ M i,j = E i,j , ˆ E i,j,k corresponds to a reconstruction of observed electrode data.
For k ∈ M i,j , ˆ E i,j,k corresponds to an imputation of unobserved electrode data.
Imputation is a more challenging task than reconstruction, since in imputation, the decoding target is not an input to the model.
The CNNAE is trained using the self-supervised objective described in Section B.1.
Specific architecture design choices are discussed in Section C.2 – the key criterion is to effectively encode and decode multivariate electrode time series data.
The CNNAE is a day-generalizable model, but it forms the backbone for the
DOPE-generalizable model class described below.
[Figure]
We extend our framework to jointly model many human participants, with different electrode spatial configurations, using participant-specific encoding and decoding layers and a shared CNNAE backbone.
M-CNNAE .
For joint-participant imputation, the
Multihead CNNAE model (M-CNNAE) learns a single function f for all participants, instead of one f ( i ) for each participant.
When extending the CNNAE to jointly model different participants (Figure 5), we first used a participant-specific encoding layer to map input electrodes to a shared embedding space.
This shared space is necessary because each participant has a different number of electrodes arranged in highly varied configurations.
The encoder, f θ , and decoder, f ψ , are then trained using the CNNAE’s learning procedure.
To map the representation from the shared embedding space back to the participant-specific electrode configuration, we also train a participantspecific decoding layer.
Components are trained jointly with data from all participants via a selfsupervised objective (Section B.1).
Unlike the
CNNAE, the M-CNNAE trains a shared representation across all participants, making it DOPEgeneralizable.
B.1 Training Objective
We train our CNNAE and M-CNNAE models with self-supervision using masked electrode modeling, where we randomly mask l observed electrodes during training.
Let ˜ E
i,j represent the observed electrodes with random masking.
Then, we train the autoencoder to simultaneously impute the masked values as well as reconstruct the observed values.
This task is similar to masking used in previous time series imputation approaches [7], except we mask entire time series from an electrode, instead of a subset of timestamps.
The autoencoder is trained with the following objective: L NLL i,j
[Equation]
C Experiments Continued C.1 Datasets for Deep Neural Imputation
To study Deep Neural Imputation we utilize recently released data from all 12 AJILE12 [11] participants, each with ∼ 100 electrodes recorded across multiple days.
Each participant has a wide range of naturalistic behaviors during recording, making reconstruction/imputation more difficult than if explored with canonical task-based neural data [20].
In the past, deep learning models have been used, often on subsets of the AJILE12 participants.
However, these models have explored more classical neural decoding tasks, namely binary prediction and binary classification of arm movement [21, 18, 22].
Please see the table below for more details on each dataset, where each participant’s data is one dataset.
With each participant, days 1-2 are never used, as participants are still under heavy drug usage given their recent brain surgery.
Days 6, 7, 7, 5-6 are discarded in participants 1, 2, 3,respectively for other medical reasons.
Days 3-7 are fully present in participants 4, 6, 7, 8, 9, 10, 11, 12.
In the table, Pt is an abbreviation for participant.
[Table]
Data Processing Pipelines .
Using the same training split as [18] where the last day is held out as the test day, we define two different data processing pipelines:
• Procedure A: lower frequency content data used in Section C. • Procedure B: higher frequency content data used in Sections C.4, 3.
We use Procedure A to study our models’ ability to reconstruct time series data.
We perform standardization on a 50,000 sample segment, corresponding to a 100 sec length time series, on a per electrode, per day, per participant basis.
We divide the 100 sec into 20 sec and 80 sec sections, calculate the mean ( µ 20 ) and standard deviation ( σ 20 ) of the 20 sec segment, and for each time step in the 80 sec segment compute [Table]
As typically done in neuroscience studies without task labels, performing local standardization on each time segment allows us to account for neural data distribution changes such as spatial electrode shifts, neural drift, habituation to external stimuli, etc.
We then downsample this data by a factor of 100, resulting in 400 time steps of data at a frequency of 5 Hz .
Using Procedure B, we study the performance of our imputation model with two downstream tasks: frequency content preservation, and a scientifically-relevant neural decoding task.
We follow the data processing pipeline outlined in [18], consisting of band-pass filtering, downsampling, and trimming, resulting in ∼ 1000 time steps of data at a frequency of 250 Hz .
Our main goal with DNI is to reconstruct the neural time series; therefore, with Procedure A we aggressively downsampled to reduce the high temporal resolution of the original data.
When using
Procedure A, the frequency content of the reconstructions was recovered in the limited frequency bands available for analysis (i.e. lower frequency bands due to downsampling).
The purpose of
Procedure B was to (1) test our models in a regime that featured different time series lengths and frequency content in the training data, (2) verify the frequency content recovery property in higher frequency bands, (3) use our reconstructions in a downstream neural decoding task.
C.2 Training and Evaluation Setup
Our train and test split follows [18], where each participant’s last day is held out as the test day.
For the linear baseline model, we use 3 nearest neighbors to compute the correlation weights on the train days.
We then apply these weights to the time series from the 3 nearest neighbors on the test day and combined them to reconstruct/impute the missing electrode’s neural signal.
For the CNNAE and M-CNNAE, we built the encoder with strided temporal convolutions and we based the decoder on [23]; together these comprise the shared backbone architecture (see Appendix for more modeling details).
During the CNNAE and M-CNNAE training, we use our self-supervised training objective to perform masked electrode modeling.
Masked electrode modeling (Figure 2) consists of creating a random electrode mask where 5 - 10% of the electrodes in each batch update are zero-filled.
We then train the models to decode the full set of input electrodes; decoded electrode values are either reconstructed or imputed.
For evaluation, we use Pearson’s Correlation between the original time series and reconstructed/imputed time series.
We explore our model in 3 missing data regimes: 10% missing, 20% missing, and 50% missing.
Within each missing data regime, we average our correlation results over all time series instances, electrodes, and 3 distinct randomly generated sets of missing electrodes.
A crucial step in evaluating our models’ decoding capabilities is comparing our reconstructions/imputations against the ground truth data that is unseen by the model.
It is worth noting that the 12 participants’ original data does have naturally missing electrodes.
In these cases we cannot compare our imputations because ground truth does not exist.
This motivated us to create zero-filled masked electrode modeling because it allows us to evaluate not only our decoded reconstructions, but also our decoded imputations.
C.3 Benchmark Results Continued
We find that Deep Neural Imputation’s linear baseline performs moderately well, validating our intuition behind shared information between neighboring electrodes.
As the percentage of missing electrodes increases, the baseline performance understandably falls because there are fewer neighboring electrodes available.
At the same time, the performance of both the CNNAE and M-CNNAE holds across our missing data regimes and improves over the baseline.
For 0% missing data the
CNNAE outperforms the baseline for 9 participants, and in the most challenging evaluation regime ( 50% missing data) the CNNAE outperforms the baseline for all 12 participants in both reconstruction and imputation.
The M-CNNAE model shares the same trend as the CNNAE when compared to the baseline, and in the most difficult evaluation regime ( 50% missing data) also outperforms the baseline for all 12 participants in both reconstruction and imputation.
When studying the differences between the M-CNNAE and CNNAE, we found that participanthad the greatest performance improvement when using the M-CNNAE.
Despite the fact that there are 12 trained CNNAE models (one for each participant) and 1 jointly-trained M-CNNAE model, there are more cases where the M-CNNAE to baseline performance is better than CNNAE to baseline performance.
To decipher the M-CNNAE and CNNAE differences we decided to explore the MCNNAE joint-participant representation space and CNNAE’s participant-specific representation spaces.
For a similar comparison, we .
For a similar comparison, we stacked all 12 of the CNNAE’s participant-specific representation spaces and analyzed the concatenation.
The M-CNNAE’s jointparticipant embedding had more samples mapped to a shared space, when compared to the CNNAE’s concatenated embedding space which had far more participant-specific clusters.
For a table of the results please see G.
C.4 Frequency Correlation Analysis
In Figure 6, we explore the relationship between frequency correlation and time series correlation across two proportions of missing data ( 0% & 50% ) because of frequency content’s significance to the neuroscience community [15, 16].
We observe a positive relationship between these correlations, despite the fact that our models were not trained to perform reconstruction or imputation in the frequency domain.
In particular, we note that the points for both reconstruction ( 0% & 50% ) and imputation ( 50% ) form a curve (Figure 6 Left), suggesting that time series correlation is predictive of frequency correlation.
We additionally visualize the original time series and corresponding spectrogram as well as the
CNNAE decoded time series and corresponding spectrogram for both a typical and performant example.
A visual inspection shows that our CNNAE produces frequency reconstructions resembling the ground truth spectrograms even for low frequency correlation values.
It is possible that alternative metrics for spectrogram evaluation, such as mutual information in frequency space [24] [25], could lead to stronger quantitative correlation metrics compared to the Pearson’s Correlation metric that we used for evaluation.
For a table of the results please see G.
[Figure]
i m e
S e r i e s
C o rr e l a t i o n [Figure] Spectrogram Examples D Related Work
Multielectrode Neuroscience Experiments .
Elucidating scientific questions in neuroengineering, neural mechanism discovery, and systems neuroscience has been critically enabled by advances in multielectrode array technology.
Different electrode recording modalities (EEG, ECoG, Utah arrays, Neuropixels [26, 1]) have specific trade-offs, such as varying signal attenuation or difficulty of implantation, to name a few.
However, they all share the commonality that when individual electrodes are corrupted or fail from faulty manufacturing, electrical noise, scar tissue formation, etc., the signal is lost and cannot be experimentally recovered.
Previous approaches address neural drift through alignment [27, 28], but not neural imputation.
In this paper, we recover high-value incomplete brain recordings via our Deep Neural Imputation Framework.
Time Series Imputation .
Our work relates to time series imputation, which aims to recover missing timestamps from time series data.
In particular, multivariate time series modeling, where more than one feature is observed at each timestamp, has been studied using statistical methods [29, 30, 31] as well as deep generative machine learning models [7, 32, 19, 33].
Our linear model baseline falls under statistical approaches using nearest-neighbors, while our CNNAE and M-CNNAE approaches are based on generative modeling.
A few common model setups include directly regressing missing values [19, 33], adversarial training (such as GANs) [32, 34, 8, 35], as well as autoencoders [7, 35].
Our CNNAE architecture is based on [36], a model recently developed for time series representation learning, which we adapted for time series imputation via masked electrode modeling.
Methods for generic time series imputation typically cannot impute when a feature is missing for all timestamps.
These methods are orthogonal to our main contribution, which is a framework for multivariate electrode imputation in both day-generalizable and DOPE-generalizable regimes.
E Discussion
In multielectrode recordings, signals from corrupted electrodes are commonly discarded and treated as missing.
To fill these gaps, we propose Deep Neural Imputation: a framework to recover missing values using neural data across days and individuals.
DNI is compatible with linear and nonlinear models, such as deep generative autoencoders, and can be easily be incorporated into existing neuroscience analysis pipelines and downstream tasks.
Key Observations.
We find that our nearest neighbor linear baseline reconstructs and imputes missing electrode data well when there is a small percentage of data missing (e.g. 10% ).
Further, both the
CNNAE and M-CNNAE models recover neural data in day-generalizable and DOPE-generalizable manners respectively when there are large percentages of data missing (e.g. 50% ).
These results may inspire neuroscientists to adjust neural electrode placement or enable novel experimental design.
Moreover, there is a positive relationship between frequency correlation and time-series correlation, indicating that without training explicitly for frequency content, we can reconstruct and impute it.
Finally, DNI is practical in downstream neuroscience tasks, which we validate based on significant improvement on a neural decoding task in the presence of missing data.
Limitations and Future Directions.
There are many directions for future work stemming from the neural imputation problem.
For joint-participant experiments, our method currently requires a participant-specific layer to be trained for each spatial configuration.
Further explorations into transfer learning and meta-learning could eliminate this requirement.
The joint-participant model offers an intriguing possibility of studying the joint embedding of neural representations to, for instance, understand individual variability in neural correlates of similar behaviors [20].
Additionally, our models cannot currently impute data gathered from unobserved spatial locations.
Graph neural networks are one possible architecture that could develop this capability.
Furthermore, these models may be improved by exploring multimodal fusion of neural, kinematic, and other measurement modalities [37, 22].
Since no prior work has conducted a rigorous study of how to recover missing electrode recordings across days and participants (see an early attempt [38]), establishing multiparticipant, multi-day benchmark of DNI methods would be valuable.
Significance and Broader impacts .
Participants’ neural activity is deeply personal; therefore, we need to be thoughtful in our use of technology to analyze this information.
The ethical considerations include issues of data management to guard the subjects’ privacy and security.
At the same time, we need to consider this technology’s assistive potential in improving participants’ lives, because recovering missing data from incomplete brain recordings has the potential to transform the design and analysis of a wide range of neuroscience and neuroengineering studies.
Further discussion and information on the data collection process can be found in the original dataset release paper [11].
F Implementation Details
We provide additional details on the data processing and the model implementation.
Model hyperparameters are in Table 1.
F.1 Data Processing
We will start by providing more intuition behind the type of neural time series that is collected from the participants.
Each electrode in the ECoG array that participants are implanted with generates a local field potential (LFP) voltage trace.
Each LFP trace comes from the bulk voltage activity of a few tens-of-thousands to hundreds-of-thousands of neurons in the brain.
These LFP voltage traces the comprise the AJILE12 dataset are stored in the common neuroscience data file format, NWB, which is specifically designed to handle ragged data, missing values, and non-task-dependent (i.e. naturalistic) neural data [6].
In the AJILE12 dataset there are 12 human participants; each participant has roughly 100 ECoG electrodes implanted in their brain based on clinically determined locations.
Before pre-processing, the data is collected at 500Hz and in most cases continuously recorded for several days.
Given the multitude of goals in our paper we created two data processing pipelines to explore our models:
Procedure A, and Procedure B.
In Procedure A, we standardize 50,000 sample segments of the
AJILE12 dataset and downsample the data to 5Hz to study our ability to perform time series reconstruction.
In Procedure B, we following the data processing pipeline in [18], producing 250Hz data that allows us to study our frequency reconstruction ability, and our model’s capability at restoring move/rest neural decoding performance.
F.2 Model Details
Linear Baseline Model .
In our linear baseline, we first break the patients’ recording days into "train" days and a "test" day.
This data split follows the dataset split in [18], where the last day is held out as the test day.
We then use the full set of electrodes for each patient to create a distance matrix which allows us to calculate each electrode’s nearest neighbors by euclidean distance.
The main idea is to linearly combine the time series correlation of the nearest neighbors to impute missing electrodes.
For every electrode on the test day, we compute the time series correlations between the target electrode and its three nearest neighbors averaged across observed training days.
Then on the test day, the nearest neighbors are combined linearly using the train day correlations.
Step-by-step, this corresponds to: (1) select a test day target electrode, (2) find one of its three nearest neighbors, (3) get the previously calculated time series correlation value that was averaged across all training days and all time series for that target electrode and the specific nearest neighbor, (4) take the time series from this nearest neighbor on the test day and weight the time series (multiply it) by the training time series correlation, (5) repeat 2-4 for all 3 nearest neighbors [if a neighbor is not observed on the test day, we use only the correlation from observed neighbors], (6) take the sum of the three weighted time series as the baseline reconstruction.
For evaluation, we (1) calculate the correlation between this baseline reconstruction and the time series of the actual target electrode recordings, (2) after calculating the correlations for all the time series off of an electrode we can average across all our time series to get the per electrode correlation.
CNNAE .
The CNNAE model is trained for each participant.
The model architecture is based on the encoder-decoder setup in [36], with strided temporal convolutions for encoding layers, an upsampling network, and a WaveNet decoder [23].
Our model hyperparameters are not tuned to AJILE12 and doing a hyperparameter search on the validation set can likely further improve performance.
The model is trained to convergence of the train loss at epoch 100.
We input the time series data for the full set of electrodes for each participant (with missing electrodes zero-filled) to the model.
At the input layer, we input both the electrode time series data as well as the time derivative of the data (change in time series value across 1 timestamp).
The encoder downsamples the time series by a factor of 8 with the current kernel and stride configurations 1.
We then upsample the encoding by a factor of 8 using the upsampling network before input to the decoder.
The WaveNet decoder preserves the sequence length, and the output of WaveNet is fed into the final shallow temporal convolution layers to output the electrode reconstruction and imputation.
There are four sets of shallow networks, outputting the mean and variance of the time series data and the time derivative.
During evaluation, we use the output mean of the time series data.
Our CNNAE models are trained on either Amazon EC2 using p2 instances with a single Tesla [-K
-]K
GPU, or a single NVIDIA GeForce RTX 2080Ti GPU.
In addition to maximizing the log-likelihood of reconstruction and imputation during training, we also use the margin loss and slowness penalty proposed in [36] to regularize the embeddings.
M-CNNAE .
The M-CNNAE model is trained jointly for all participants, with a similar architecture to the CNNAE.
Our M-CNNAE model hyperparameters are not tuned to AJILE12 and doing a hyperparameter search on the validation set can likely further improve performance.
The model is trained to epoch 45.
The main difference for the M-CNNAE compared to the CNNAE is participantspecific encoding and decoding layers, with shared layers across participants similar to the CNNAE in between.
Given input time series data, the data is mapped by a participant-specific encoding layer before feeding into the CNNAE encoder, which is a set of strided convolutions.
Then similar to the CNNAE, the encodings are upsampled, then fed into the WaveNet decoder.
The decoder outputs are used as inputs to the final participant-specific decoding layers, which are shallow networks that map decoder outputs to the mean and variance of the time series data and the time derivative.
During evaluation, we use the output mean of the time series data.
Our M-CNNAE models are trained on Amazon EC2 using p2 instances with a single Tesla K80 GPU.
In addition to maximizing the log-likelihood of reconstruction and imputation during training, we also use the margin loss and slowness penalty proposed in [36] to regularize the embeddings.
G Additional Results G.1 Linear All Electrode Results
See the table 2 for a comparison of the linear all electrode baseline, the nearest neighbors baseline,
CNNAE, and M-CNNAE.
Low MSE values are better.
We find that the linear all electrode model performs worse than the 3-NN baseline across all conditions (10%, 20%, 50% missing) for participants [Table]
For participants 4, 7, 11, 12 the learned linear baseline performance is most often similar to the 3-NN baseline.
G.2 Benchmark Results with Standard Deviation
The mean and standard deviation from the correlation comparison in Figure 3 of the main paper is shown in Tables 3, 4.
At 0% missing electrodes, both the CNNAE and M-CNNAE generally outperform the baseline.
In addition, the M-CNNAE performs better than the CNNAE for participant 1, and similarly to the CNNAE for the other participants.
The gap between the machine learning models and the baseline increases as the amount of missing electrode increases - in particular, there is a significant improvement using the CNNAE and M-CNNAE across all participants at 50% of missing electrodes.
Similar to Figure 3, the CNNAE and M-CNNAE generally has higher correlation compared to the baseline at imputation as well as reconstruction.
G.3 Frequency Analysis for All Participants
Here we expand on the participant 3 results presented in the paper, Figure 6, by showing the results for all participants.
For participants 1, 2, 4 we show not only the time series and frequency correlation plots but also some example time series and spectrograms (Figures 7, 8, 9 respectively).
In addition, for the other participants 5, 6, 7, 8, 9, 10, 11, 12 we show the time series and frequency correlation plots, Figure 10.
Results for all patients are shown across two proportions of missing data, 10% and 50%.
We observed that there was a positive relationship between time series correlation and frequency correlation, despite not having trained our model on frequency reconstruction.
This trend between time series correlation and frequency correlation can been seen strongest in participants 1, 2, 3, 6, 7, 8, 10, 11, and 12.
G.4 Neural Decoding Results
Table 5 expands on the results presented in Figure 4.
In Table 5, we include the performance of the random forest neural decoder across 50%, 70%, and 90% of electrodes missing.
We report the mean performance across 5 random seeds for the zero-filled data, and CNNAE-filled data.
In addition, we include the mean and standard deviation for the pairwise relative accuracy between the CNNAE-filled data and zero-filled data.
Positive mean relative accuracy values indicate that the CNNAE-filled data outperforms the zero-filled data on the move/rest neural decoding task.
G.5 k -Nearest Neighbors Hyperparameter Sweep
In the figures ( 11, 12) and tables ( 6, 7, 8, 9) below we see that the baseline performance plateaus as we add more neighbors across both conditions (0%, 10%, 20%, and 50% missing) as well as participants (1-12).
This is in line with the brain’s spatial pattern of correlation, a well-studied neuroscience phenomenon.
The 3-Nearest Neighbors performance is very similar to the plateaued 7-Nearest Neighbors performance, as evidenced by Figure 12 (which is the Figure 3 recreation with 7NN baseline).
The trend we observed in the main paper (3NN Linear Baseline - Figure 3) that our CNNAE or M-CNNAE model outperforms the linear baseline as the percent of missing data [Figure] T i m e S e r i e s C o rr e l a t i o n
[Figure]
increases stays true with the 7NN Linear Baseline model (Figure 12).
We want to emphasize that the state of the art in neuroscience is a correlation of 0 for imputation, because current practice discards fully-missing electrode time series.
Therefore any of our linear baseline models (let alone our CNNAE and MCNNAE models) is a dramatic improvement above current practice.
G.6 Mean Squared Error (MSE) Metric
We compute the MSE as an additional metric to time series correlation.
In general, we found that the models’ rankings using MSE are similar to those using the correlation metric (Figure 3).
In the table we present, lower MSE values are better.
We also additionally report a Relative Imput.
MSE value (which is the imputation minus the reconstruction value) so that one can directly compare reconstruction with imputation.
Positive values mean that reconstructions outperform imputations.
See tables 10 and 11 for details.
[Table] [Table]
[Table] [Table]
relative accuracy means & standard deviations.
The results for each percent missing (50%, 70%, and 90%) were calculated acrossseeds.
There are only two instances where the zero-filled accuracy outperformed the CNNAE-filled accuracy, corresponding to the yellow shading in Figure 6.
All other cases correspond to where the
CNNAE-filled accuracy was equal to or outperformed the zero-filled accuracy.
Correlation Performance (Pt2)
[Figure]
i m e S e r i e s C
o rr e l a t i o n
[Figure] [Figure]
i
[Figure]
e S e r i e s C
o rr e l a t i o n
[Figure]
Exemplary Spectrograms
Performance
T i m e S e r i e s C
o rr e l a t i o n
[Figure]
i
[Figure]
S e r i e s C
[Figure]
rr
[Figure]
l a t i o n
[Figure]
conditions (model use in Figure 3) is the results for the 7NN Both of which are much better than the Neuroscience SOTA (State
Of The Art) for imputation, which gives a 0 correlation we report our time series correlation metric between our and reconstructed time as a neighbor We
[Table]
[Table] [Table]
as increases. [Table] [Figure] [Table]
[Figure]
t
i
[Figure] [Table] [Table]