Original: B RAIN BERT: S ELF - SUPERVISED REPRESENTATION LEARNING FOR INTRACRANIAL RECORDINGS
Revised: B RAIN BERT: S ELF - SUPERVISED REPRESENTATION LEARNING FOR INTRACRANIAL RECORDINGS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We create a reusable Transformer, BrainBERT, for intracranial  recordings bringing modern representation learning approaches to neuroscience.
Revised: We create a reusable Transformer, BrainBERT, for intracranial field potential recordings bringing modern representation learning approaches to neuroscience.

Original: Much like in NLP and speech recognition, this Transformer enables classifying complex concepts, i.e., decoding neural data, with higher accuracy and with much less data by being pretrained in an unsupervised manner on a large corpus of unannotated neural recordings.
Revised: Much like in NLP and speech recognition, this Transformer enables classifying complex concepts, i.e., decoding neural data, with higher accuracy and with much less data by being pretrained in an unsupervised manner on a large corpus of unannotated neural recordings.

Original: Our approach generalizes to new subjects with electrodes in new positions and to unrelated tasks showing that the representations robustly disentangle the neural signal.
Revised: Our approach generalizes to new subjects with electrodes in new positions and to unrelated tasks showing that the representations robustly disentangle the neural signal.

Original: Just like in NLP where one can study language by investigating what a language model learns, this approach opens the door to investigating the brain by studying what a model of the brain learns.
Revised: Just like in NLP where one can study language by investigating what a language model learns, this approach enables investigating the brain by studying what a model of the brain learns.

Original: As a first step along this path, we demonstrate a new analysis of the intrinsic dimensionality of the computations in different areas of the brain.
Revised: As a first step along this path, we demonstrate a new analysis of the intrinsic dimensionality of the computations in different areas of the brain.

Original: To construct BrainBERT, we combine super-resolution spectrograms of neural data with an approach designed for generating contextual representations of audio by masking.
Revised: To construct BrainBERT, we combine super-resolution spectrograms of neural data with an approach designed for generating contextual representations of audio by masking.

Original: In the future, far more concepts will be decodable from neural recordings by using representation learning, potentially unlocking the brain like language models unlocked language.
Revised: In the future, far more concepts will be decodable from neural recordings by using representation learning, potentially unlocking the brain like language models unlocked language.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Methods that analyze neural recordings have an inherent tradeoff between power and explainability.
Revised: Methods that analyze neural recordings have an inherent tradeoff between power and explainability.

Original: Linear decoders, by far the most popular, provide explainability; if something is decodable, it is computed and available in that area of the brain.
Revised: Linear decoders, by far the most popular, provide explainability; if something is decodable, it is computed and available in that area of the brain.

Original: The decoder itself is unlikely to be performing the task we want to decode, instead relying on the brain to do so.
Revised: The decoder itself is unlikely to be performing the task we want to decode, instead relying on the brain to do so.

Original: Unfortunately, many interesting tasks and features may not be linearly decodable from the brain for many reasons including a paucity of annotated training data, noise from nearby neural processes, and the inherent spatial and temporal resolution of the instrument.
Revised: Unfortunately, many interesting tasks and features may not be linearly decodable from the brain for many reasons including a paucity of annotated training data, noise from nearby neural processes, and the inherent spatial and temporal resolution of the instrument.

Original: More powerful methods that perform non-linear transformations have lower explainability: there is a danger that the task isn’t being performed by the brain, but by the decoder itself.
Revised: More powerful methods that perform non-linear transformations have lower explainability: there is a danger that the task is not being performed by the brain, but by the decoder itself.

Original: In the limit, one could conclude that object class is computed by the retina using a CNN-based decoder.
Revised: In the limit, one could conclude that object class is computed by the retina using a CNN-based decoder but it is well established that the retina does not contain explicit information about objects.

Original: Self-supervised representation learning provides a balance between these two extremes.
Revised: Self-supervised representation learning provides a balance between these two extremes.

Original: We learn representations that are generally useful for representing neural recordings, without any knowledge of a task being performed, and then employ a linear decoder.
Revised: We learn representations that are generally useful for representing neural recordings, without any knowledge of a task being performed, and then employ a linear decoder.

Original: The model we present here, BrainBERT, does exactly this. It learns a complex non-linear transformation of neural data using a Transformer.
Revised: The model we present here, BrainBERT 1 , learns a complex non-linear transformation of neural data using a Transformer.

Original: Using BrainBERT, one canthen linearly decode neural recordings with much higher accuracy and with far fewer examples.
Revised: Using BrainBERT, one can linearly decode neural recordings with much higher accuracy and with far fewer examples than from raw features.

Original: BrainBERT is pretrained once across a pool of subjects, and then provides off-the-shelf capabilities for analyzing new subjects with new electrode locations even when data is scarce.
Revised: BrainBERT is pretrained once across a pool of subjects, and then provides off-the-shelf capabilities for analyzing new subjects with new electrode locations even when data is scarce.

Original: Neuroscientific experiments tend to have little data in comparison to other machine learning settings, making additional sample efficiency critical.
Revised: Neuroscientific experiments tend to have little data in comparison to other machine learning settings, making additional sample efficiency critical.

Original: Other applications, such as brain-computer interfaces can also benefit from shorter training regimes, as well as from BrainBERT’s significant performance improvements.
Revised: Other applications, such as brain-computer interfaces can also benefit from shorter training regimes, as well as from BrainBERT’s significant performance improvements.

Original: In addition, the embeddings of the neural data provide a new means by which to investigate the brain.
Revised: In addition, the embeddings of the neural data provide a new means by which to investigate the brain.

Original: BrainBERT provides contextualized neural embeddings, in the same way that masked language modeling provides contextual word embeddings.
Revised: BrainBERT provides contextualized neural embeddings, in the same way that masked language modeling provides contextual word embeddings.

Original: Such methods have proven themselves in areas like speech recognition where a modest amount of speech, 200 to 400 hours, leads to models from which one can linearly decode the word being spoken.
Revised: Such methods have proven themselves in areas like speech recognition where a modest amount of speech, 200 to 400 hours, leads to models from which one can linearly decode the word being spoken.

Original: We use a comparable amount of recordings, 43.7 hours across all subjects (4,551 electrode-hours), of unannotated neural recordings to build similarly reusable and robust representations.
Revised: We use a comparable amount of recordings, 43.7 hours across all subjects (4,551 electrode-hours), of unannotated neural recordings to build similarly reusable and robust representations.

Original: To build contextualized embeddings, BrainBERT borrows from masked language modeling (Devlin et al., 2019) and masked audio modeling (Baevski et al., 2020; Liu et al., 2021).
Revised: To build contextualized embeddings, BrainBERT borrows from masked language modeling (Devlin et al., 2019) and masked audio modeling (Baevski et al., 2020; Liu et al., 2021).

Original: Given neural activity, as recorded by a stereo-electroencephalographic (SEEG) probe, we compute a spectrogram per electrode.
Revised: Given neural activity, as recorded by a stereo-electroencephalographic (SEEG) probe, we compute a spectrogram per electrode.

Original: We mask random parts of that spectrogram and train BrainBERT to produce embeddings from which the original can be reconstructed.
Revised: We mask random parts of that spectrogram and train BrainBERT to produce embeddings from which the original can be reconstructed.

Original: But unlike speech audio, neural activity has fractal and scale-free characteristics (Lutzenberger et al., 1995; Freeman, 2005), meaning that similar patterns appear at different time scales and different frequencies, and identifying these patterns is often a challenge.
Revised: But unlike speech audio, neural activity has fractal and scale-free characteristics (Lutzenberger et al., 1995; Freeman, 2005), meaning that similar patterns appear at different time scales and different frequencies, and identifying these patterns is often a challenge.

Original: To that end, we adapt modern neural signal processing techniques for producing superresolution time-frequency representations of neurophysiological signals (Moca et al., 2021).
Revised: To that end, we adapt modern neural signal processing techniques for producing superresolution time-frequency representations of neurophysiological signals (Moca et al., 2021).

Original: Such techniques come with a variable trade-off in time-frequency resolution, which we account for in our adaptive masking strategy.
Revised: Such techniques come with a variable trade-off in time-frequency resolution, which we account for in our adaptive masking strategy.

Original: Finally, the activity captured by intracranial electrodes is often characterized by a spiking morphology, which means that large regions of the neural spectrogram are zero or close to zero.
Revised: Finally, the activity captured by intracranial electrodes is often sparse.

Original: To incentivize the model to better represent short-bursts of packet activity, we use a content-aware loss that places more weight on non-zero spectrogram elements.
Revised: To incentivize the model to better represent short-bursts of packet activity, we use a content-aware loss that places more weight on non-zero spectrogram elements.

Original: Our contributions are:
Revised: Our contributions are:

Original: the BrainBERT model — a reusable, off-the-shelf, subject-agnostic, and electrode-agnostic model that provides embeddings for intracranial recordings, 2.
Revised: the BrainBERT model — a reusable, off-the-shelf, subject-agnostic, and electrode-agnostic model that provides embeddings for intracranial recordings, 2.

Original: a demonstration that BrainBERT systematically improves the performance of linear decoders, 3.
Revised: a demonstration that BrainBERT systematically improves the performance of linear decoders, 3.

Original: a demonstration that BrainBERT generalizes to previously unseen subjects with new electrode locations, and 4.
Revised: a demonstration that BrainBERT generalizes to previously unseen subjects with new electrode locations, and 4.

Original: a novel analysis of the intrinsic dimensionality of the computations performed by different parts of the brain made possible by BrainBERT embeddings.
Revised: a novel analysis of the intrinsic dimensionality of the computations performed by different parts of the brain made possible by BrainBERT embeddings.

Original:  M ETHOD
Revised:  M ETHOD

Original: The core of BrainBERT is a stack of Transformer encoder layers (Vaswani et al., 2017).
Revised: The core of BrainBERT is a stack of Transformer encoder layers (Vaswani et al., 2017).

Original: In pretraining, BrainBERT receives an unannotated time-frequency representation of the neural signal as input.
Revised: In pretraining, BrainBERT receives an unannotated time-frequency representation of the neural signal as input.

Original: This input is randomly masked, and the model learns to reconstruct the missing portions.
Revised: This input is randomly masked, and the model learns to reconstruct the missing portions.

Original: The pretrained BrainBERT weights can then be combined with a classification head and trained on decoding tasks using supervised data.
Revised: The pretrained BrainBERT weights can then be combined with a classification head and trained on decoding tasks using supervised data.

Original: Architecture
Revised: Architecture

Original: Given the voltage measurements x ∈ R r · t of a single electrode sampled at rate r for t seconds, we first find the time-frequency representation Φ( x ) =
Revised: Given the voltage measurements x ∈ R r · t of a single electrode sampled at rate r for t seconds, we first find the time-frequency representation Φ( x ) =

Original: Y ∈ R n × m , which has n frequency channels and m time frames.
Revised: Y ∈ R n × m , which has n frequency channels and m time frames.

Original: BrainBERT is built around a Transformer encoder stack core (Vaswani et al., 2017) with N layers, each with H attention heads and intermediate hidden dimension d h .
Revised: BrainBERT is built around a Transformer encoder stack core (Vaswani et al., 2017) with N layers, each with H attention heads and intermediate hidden dimension d h .

Original: The inputs to the first layer are non-contextual embeddings for each time frame,
Revised: The inputs to the first layer are non-contextual embeddings for each time frame,

Original: E 0 in =
Revised: E 0 in =

Original: ( W in Y + P ) , which are produced using a weight matrix W in ∈ R d
Revised: ( W in Y + P ) , which are produced using a weight matrix W in ∈ R d

Original: h × n and combined with a static positional embedding P (Devlin et al., 2019).
Revised: h × n and combined with a static positional embedding P (Devlin et al., 2019).

Original: Each layer applies self-attention and a feed forward layer to the input, with layer normalization (Ba et al., 2016) and dropout (Srivastava et al., 2014) being applied after each.
Revised: Each layer applies self-attention and a feed forward layer to the input, with layer normalization (Ba et al., 2016) and dropout (Srivastava et al., 2014) being applied after each.

Original: The outputs E j out of the j -th layer, become the inputs to the ( j +1) -th layer.
Revised: The outputs E j out of the j -th layer, become the inputs to the ( j +1) -th layer.

Original: The outputs of BrainBERT are E N out , the outputs at the N -th layer.
Revised: The outputs of BrainBERT are E N out , the outputs at the N -th layer.

Original: During pretraining, the hidden-layer outputs from the top of the stack are passed as input to a spectrogram prediction head, which is a stacked linear network with a single hidden layer, GeLU activation (Hendrycks & Gimpel, 2016), and layer normalization.
Revised: During pretraining, the hidden-layer outputs from the top of the stack are passed as input to a spectrogram prediction head, which is a stacked linear network with a single hidden layer, GeLU activation (Hendrycks & Gimpel, 2016), and layer normalization.

Original: Time-frequency representations
Revised: Time-frequency representations

Original: BrainBERT can take two different types of time-frequency representations as input: the Short-Time Fourier Transform (STFT) and the superlet transform (Moca et al., 2021), which is a composite of Morlet wavelet transforms; see appendix A.
Revised: BrainBERT can take two different types of time-frequency representations as input: the Short-Time Fourier Transform (STFT) and the superlet transform (Moca et al., 2021), which is a composite of Morlet wavelet transforms; see appendix A.

Original: Due to the Heisenberg-Gabor uncertainty principle, there exists an inherent trade-off between the time and frequency resolution that any representation can provide.
Revised: Due to the Heisenberg-Gabor uncertainty principle, there exists an inherent trade-off between the time and frequency resolution that any representation can provide.

Original: The most salient difference between the STFT and superlet transform is the way they handle this trade-off.
Revised: The most salient difference between the STFT and superlet transform is the way they handle this trade-off.

Original: For the STFT, resolution is fixed for all frequencies.
Revised: For the STFT, resolution is fixed for all frequencies.

Original: For the superlet transform, temporal resolution increases with frequency.
Revised: For the superlet transform, temporal resolution increases with frequency.

Original: This is a well-motivated choice for neural signal, where high frequency oscillations are tightly localized in time.
Revised: This is a well-motivated choice for neural signal, where high frequency oscillations are tightly localized in time.

Original: For both types of representations, the spectrograms are z-scored per frequency bin.
Revised: For both types of representations, the spectrograms are z-scored per frequency bin.

Original: This is done in order to better reveal oscillations at higher frequencies, which are usually hidden by the lower frequencies that typically dominate in power.
Revised: This is done in order to better reveal oscillations at higher frequencies, which are usually hidden by the lower frequencies that typically dominate in power.

Original: Additionally, the z-score normalization makes BrainBERT agnostic to the role that each frequency band might play for different tasks.
Revised: Additionally, the z-score normalization makes BrainBERT agnostic to the role that each frequency band might play for different tasks.

Original: Specific frequency bands have previously been implicated in different cognitive processes such as language (Babajani-Feremi et al., 2016), emotion (Drane et al., 2021), and vision (Jia et al., 2013).
Revised: Specific frequency bands have previously been implicated in different cognitive processes such as language (Babajani-Feremi et al., 2016), emotion (Drane et al., 2021), and vision (Jia et al., 2013).

Original: By intentionally putting all frequency bands on equal footing, we ensure that BrainBERT embeddings will be generic and useful for a wide variety of tasks.
Revised: By intentionally putting all frequency bands on equal footing, we ensure that BrainBERT embeddings will be generic and useful for a wide variety of tasks.

Original: Pretraining
Revised: Pretraining

Original: During pretraining, a masking strategy is applied to the time-frequency representation Y ∈ R n × m , and an augmented view of the spectrogram, ˜Y , is produced.
Revised: During pretraining, a masking strategy is applied to the time-frequency representation Y ∈ R n × m , and an augmented view of the spectrogram, ˜Y , is produced.

Original: Given ˜Y , BrainBERT creates representations for a spectrogram prediction network, which produces a reconstruction ˆ Y of the original signal; see fig.
Revised: Given ˜Y , BrainBERT creates representations for a spectrogram prediction network, which produces a reconstruction ˆ Y of the original signal; see fig.

Original: 1.d.
Revised: 1.d.

Original: For the STFT, we adapt the masking strategy of Liu et al.
Revised: For the STFT, we adapt the masking strategy of Liu et al.

Original: (2021), in which the spectrogram is corrupted at randomly chosen time and frequency intervals.
Revised: (2021), in which the spectrogram is corrupted at randomly chosen time and frequency intervals.

Original: The width of each time-mask is a randomly chosen integer from the range [ step timemin , step timemax ] .
Revised: The width of each time-mask is a randomly chosen integer from the range [ step timemin , step timemax ] .

Original: Following Devlin et al.
Revised: Following Devlin et al.

Original: (2019), intervals selected for masking are probabilistically either left untouched (probability p ID ) , replaced with a random slice of the same spectrogram (probability p replace ), or filled in with zeros otherwise; see appendix D for the pseudocode summary and parameter settings.
Revised: (2019), intervals selected for masking are probabilistically either left untouched (probability p ID ) , replaced with a random slice of the same spectrogram (probability p replace ), or filled in with zeros otherwise; see appendix D for the pseudocode summary and parameter settings.

Original: The procedure for masking frequency intervals is similar, but the width is chosen from the range [ step freqmin , step freqmax ] .
Revised: The procedure for masking frequency intervals is similar, but the width is chosen from the range [ step freqmin , step freqmax ] .

Original: For the superlet model, we use an adaptive masking scheme that reflects the variable trade-off in time-frequency resolution made by the continuous wavelet transform.
Revised: For the superlet model, we use an adaptive masking scheme that reflects the variable trade-off in time-frequency resolution made by the continuous wavelet transform.

Original: The temporal width of the time mask increases with the inverse of frequency.
Revised: The temporal width of the time mask increases with the inverse of frequency.

Original: Similarly, when masking frequencies, more channels are masked at higher frequencies; see appendix C for parameters.
Revised: Similarly, when masking frequencies, more channels are masked at higher frequencies; see appendix C for parameters.

Original: The model is optimized according to a novel content aware loss.
Revised: The model is optimized according to a novel content aware loss.

Original: For speech audio modeling, it is typical to use an L1 reconstruction loss (Liu et al., 2020; 2021):
Revised: For speech audio modeling, it is typical to use an L1 reconstruction loss (Liu et al., 2020; 2021):

Original: [Equation]
Revised: [Equation]

Original: where M is the set of masked spectrogram positions.
Revised: where M is the set of masked spectrogram positions.

Original: But intracranial neural signal is characterized by spiking activity and short oscillation bursts.
Revised: But intracranial neural signal is characterized by spiking activity and short oscillation bursts.

Original: Furthermore, since the spectrogram is z-scored along the time-axis, approximately 68% of the z-scored spectrogram is 0 or < 1 .
Revised: Furthermore, since the spectrogram is z-scored along the time-axis, approximately 68% of the z-scored spectrogram is 0 or < 1 .

Original: So, with just a bare reconstruction loss, the model tends to predict 0 for most of the masked portions, especially in the early stages of pretraining.
Revised: So, with just a bare reconstruction loss, the model tends to predict 0 for most of the masked portions, especially in the early stages of pretraining.

Original: To discourage this and to speed convergence, we add a term to the loss which incentivizes the faithful reconstruction of the spectrogram elements that are γ far away from 0, for some threshold γ .
Revised: To discourage this and to speed convergence, we add a term to the loss which incentivizes the faithful reconstruction of the spectrogram elements that are γ far away from 0, for some threshold γ .

Original: [Equation]
Revised: [Equation]

Original: { ( i, j ) | Y i,j >
Revised: ( i, j ) | Y i,j >

Original: [Equation]
Revised: [Equation]

Original: | (cid:88) ( i,j ) | ( [Equation] ,j ) ∈ M, Y i,j >γ (cid:12)(cid:12)
Revised: | (cid:88) ( i,j ) | ( i,j ) ∈ M, Y i,j >γ (cid:12)(cid:12)

Original: Y i,j − ˆY i,j (cid:12)(cid:12) (2)
Revised: Y i,j − ˆY i,j (cid:12)(cid:12) (2)

Original: This incentivizes the model to faithfully represent those portions of the signal where neural processes are most likely occurring.
Revised: This incentivizes the model to faithfully represent those portions of the signal where neural processes are most likely occurring.

Original: Then, our loss function is:
Revised: Then, our loss function is:

Original: [Equation]
Revised: [Equation]

Original: Fine-tuning
Revised: Fine-tuning

Original: After pretraining, BrainBERT can be used as a feature extractor for a linear classifier.
Revised: After pretraining, BrainBERT can be used as a feature extractor for a linear classifier.

Original: Then, given an input spectrogram Y ∈ R n × 2 l , the features are E = BrainBERT ( Y ) .
Revised: Then, given an input spectrogram Y ∈ R n × 2 l , the features are E = BrainBERT ( Y ) .

Original: For a window size k , the center 2 k features are W = E : ,l − k : l + k , and the input to the classification network is the vector resulting from taking the mean of W along the time (first) axis.
Revised: For a window size k , the center 2 k features are W = E : ,l − k : l + k , and the input to the classification network is the vector resulting from taking the mean of W along the time (first) axis.

Original: We use k = 5 , which corresponds with a time duration of ≈ 244 ms.
Revised: We use k = 5 , which corresponds with a time duration of ≈ 244 ms.

Original: During training, BrainBERT’s weights can either be frozen (no fine-tuning) or they can be updated along with the classification head (fine-tuning).
Revised: During training, BrainBERT’s weights can either be frozen (no fine-tuning) or they can be updated along with the classification head (fine-tuning).

Original: Fine-tuning uses more compute resources, but often results in better performance.
Revised: Fine-tuning uses more compute resources, but often results in better performance.

Original: We explore both use cases in this work.
Revised: We explore both use cases in this work.

Original: Data
Revised: Data

Original: Invasive intracranial field potential recordings were collected during 26 sessions fromsubjects (5 male, 5 female; aged 4-19, µ 11.9, σ 4.6) with pharmacologically intractable epilepsy.
Revised: Invasive intracranial field potential recordings were collected during 26 sessions fromsubjects (5 male, 5 female; aged 4-19, µ 11.9, σ 4.6) with pharmacologically intractable epilepsy.

Original: Approximately, 4.37 hours of data were collected from each subject; see appendix J. During each session, subjects watched a feature length movie in a quiet room while their neural data was recorded at a rate of 2kHz.
Revised: Approximately, 4.37 hours of data were collected from each subject; see appendix J. During each session, subjects watched a feature length movie in a quiet room while their neural data was recorded at a rate of 2kHz.

Original: Brain activity was measured from electrodes on stereo-electroencephalographic (SEEG) probes, following the methods of Liu et al.
Revised: Brain activity was measured from electrodes on stereo-electroencephalographic (SEEG) probes, following the methods of Liu et al.

Original: Across all subjects, data was recorded from a total of 1,688 electrodes, with a mean of 167 electrodes per subject.
Revised: Across all subjects, data was recorded from a total of 1,688 electrodes, with a mean of 167 electrodes per subject.

Original: Line noise was removed and the signal was Laplacian re-referenced (Li et al., 2018).
Revised: Line noise was removed and the signal was Laplacian re-referenced (Li et al., 2018).

Original: During pretraining, data from all subjects and electrodes is segmented into 5s intervals, and all segments are combined into a single training pool.
Revised: During pretraining, data from all subjects and electrodes is segmented into 5s intervals, and all segments are combined into a single training pool.

Original: The complete preprocessing pipeline is described in appendix F.
Revised: The complete preprocessing pipeline is described in appendix F.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: For pretraining purposes, neural recordings from 19 of the sessions ( µ duration = 2 . 3 hrs) was selected, and the remaining 7 sessions ( µ duration = 2 .
Revised: For pretraining purposes, neural recordings from 19 of the sessions ( µ duration = 2 . 3 hrs) was selected, and the remaining 7 sessions ( µ duration = 2 .

Original: 55 hrs) were held out to evaluate performance on decoding tasks; see section 3 Tasks.
Revised: 55 hrs) were held out to evaluate performance on decoding tasks; see section 3 Tasks.

Original: All ten subjects are represented in the pretraining data.
Revised: All ten subjects are represented in the pretraining data.

Original: In total,
Revised: In total,

Original: Sentenceonset speech
Revised: Sentenceonset speech

Original: [Table]
Revised: [Table]

Original:  there are 21 unique movies represented across all sessions — the set of movies used in pretraining does not overlap with the held-out sessions.
Revised: like determining some other there are 21 unique movies represented across all sessions — the set of movies used in pretraining does not overlap with the held-out sessions.

Original: From this held-out set, for each task, session and electrode, we extract annotated examples.
Revised: From this held-out set, for each task, session, and electrode, we extract annotated examples.

Original: Each example corresponds with a time-segment of the session recordings with time stamp t .
Revised: Each example corresponds with a time-segment of the session recordings with time stamp t .

Original: Depending on the classifier architecture, the input for each example is either 0.25s or 5s of surrounding context, centered on t .
Revised: Depending on the classifier architecture, the input for each example is either 0.25s or 5s of surrounding context, centered on t .

Original: The time-segments in each split are consistent across electrodes.
Revised: The time-segments in each split are consistent across electrodes.

Original: We run two sets of experiments, pretraining BrainBERT separately on both STFT and superlet representations.
Revised: We run two sets of experiments, pretraining BrainBERT separately on both STFT and superlet representations.

Original: Pretraining examples are obtained by segmenting the neural recordings into 5s intervals.
Revised: Pretraining examples are obtained by segmenting the neural recordings into 5s intervals.

Original: Tasks
Revised: Tasks

Original: To demonstrate BrainBERT’s abilities to assist with a wide range of classification tasks, we demonstrate it on four such tasks.
Revised: To demonstrate BrainBERT’s abilities to assist with a wide range of classification tasks, we demonstrate it on four such tasks.

Original: Tasks range from low-level perception such as determining the volume of the audio the subject is listening to, to mid-level tasks such as determining if the subject is hearing speech or non-speech, as well as the pitch of the overheard words, and higher-level tasks such as determining if the subject just heard the onset of a sentence as opposed to non-speech sounds.
Revised: Tasks range from low-level perception such as determining the volume of the audio the subject is listening to, to mid-level tasks such as determining if the subject is hearing speech or non-speech, as well as the pitch of the overheard words, and higher-level tasks such as determining if the subject just heard the onset of a sentence as opposed to non-speech sounds.

Original: BrainBERT is pretrained without any knowledge of these tasks.
Revised: BrainBERT is pretrained without any knowledge of these tasks.

Original: Volume of the audio was computed automatically from the audio track of the movie.
Revised: Volume of the audio was computed automatically from the audio track of the movie.

Original: Segments of neural recordings that corresponded to where audio track that was one standard deviation above or below the mean in terms of volume were selected.
Revised: Segments of neural recordings that corresponded to where audio track that was one standard deviation above or below the mean in terms of volume were selected.

Original: Speech vs. non-speech examples were manually annotated.
Revised: Speech vs. non-speech examples were manually annotated.

Original: An automatic speech recognizer first produced a rough transcript which was then corrected by annotators in the lab.
Revised: An automatic speech recognizer first produced a rough transcript which was then corrected by annotators in the lab.

Original: Pitch was automatically computed for each word with Librosa (McFee et al., 2015) and the same one standard deviation cutoff was used to select the data.
Revised: Pitch was automatically computed for each word with Librosa (McFee et al., 2015) and the same one standard deviation cutoff was used to select the data.

Original: Sentence onsets were automatically derived from the transcript.
Revised: Sentence onsets were automatically derived from the transcript.

Original: In the case of the sentence onset task and the speech vs. non-speech task, the two classes were explicitly balanced.
Revised: In the case of the sentence onset task and the speech vs. non-speech task, the two classes were explicitly balanced.

Original: See appendix J for the exact number of examples used per class.
Revised: See appendix J for the exact number of examples used per class.

Original: All results are reported in terms of the ROC (receiver operating characteristic)
Revised: All results are reported in terms of the ROC (receiver operating characteristic)

Original: AUC (area under the curve), where chance performance is 0.50 regardless of whether the dataset were balanced or not.
Revised: AUC (area under the curve), where chance performance is 0.50 regardless of whether the dataset were balanced or not.

Original: The data was randomly split into a 80/10/10 training/validation set/test set.
Revised: The data was randomly split into a 80/10/10 training/validation set/test set.

Original: Baselines
Revised: Baselines

Original: To establish baselines on supervised performance, we train decoding networks both on the raw neural signal x and on the time-frequency representations, Φ( x ) .
Revised: To establish baselines on supervised performance, we train decoding networks both on the raw neural signal x and on the time-frequency representations, Φ( x ) .

Original: We train two linear classifiers, at two different time scales, which take the raw neural signal as input.
Revised: We train two linear classifiers, at two different time scales, which take the raw neural signal as input.

Original: One network receives 250 ms of input, which is approximately the same size ( 244 ms) as the window of BrainBERT
Revised: One network receives 250 ms of input, which is approximately the same size ( 244 ms) as the window of BrainBERT

Original: Sentenceonset
Revised: Sentenceonset

Original: Speech
Revised: Speech

Original: Non

Original: [Table]
Revised: [Table]

Revised: Task

Original: [Table]
Revised: [Table]

Original: Because BrainBERT receives a time-frequency representation, Φ( x ) , as input
Revised: Because BrainBERT receives a time-frequency representation, Φ( x ) , as input

Original: [Table]
Revised: [Table]

Original: we also train two linear networks that take time-frequency representations (either STFT or superlet) as input.
Revised: we also train two linear networks that take time-frequency representations (either STFT or superlet) as input.

Original: Complete descriptions of each network architecture is given in appendix E.
Revised: Complete descriptions of each network architecture is given in appendix E.

Original: Evaluation is performed per task and per electrode.
Revised: Evaluation is performed per task and per electrode.

Original: In order to keep computational costs reasonable, we only evaluate on a subset of electrodes.
Revised: In order to keep computational costs reasonable, we only evaluate on a subset of electrodes.

Original: Our comparison to the baselines thus proceeds in two stages.
Revised: Our comparison to the baselines thus proceeds in two stages.

Original: First, the Linear (5s time domain input) network is trained once per electrode, for all data recorded in the held-out sessions; see section 3.
Revised: First, the Linear (5s time domain input) network is trained once per electrode, for all data recorded in the held-out sessions; see section 3.

Original: Then, we select, per task, the top 10 electrodes,
Revised: Then, we select, per task, the top 10 electrodes,

Original: S task , for which the linear decoder achieves the best ROC-AUC.
Revised: S task , for which the linear decoder achieves the best ROC-AUC.

Original: For each task, this set S task is held fixed for all baselines, ablations, and comparisons to our model.
Revised: For each task, this set S task is held fixed for all baselines, ablations, and comparisons to our model.

Original:  R ESULTS
Revised:  R ESULTS

Original: BrainBERT is pretrained as described above in an unsupervised manner without any manual annotations.
Revised: BrainBERT is pretrained as described above in an unsupervised manner without any manual annotations.

Original: Our goal is for BrainBERT to be usable off-the-shelf for new experiments with new data.
Revised: Our goal is for BrainBERT to be usable off-the-shelf for new experiments with new data.

Original: To that end, we demonstrate that it improves decoding performance on held-out data and held-out electrodes.
Revised: To that end, we demonstrate that it improves decoding performance on held-out data and held-out electrodes.

Original: Next, we show that its performance is conserved for never-before-seen subjects with novel electrode locations.
Revised: Next, we show that its performance is conserved for never-before-seen subjects with novel electrode locations.

Original: After that, we demonstrate that a linear decoder using BrainBERT embeddings outperforms the alternatives with 1/5th as much data.
Revised: After that, we demonstrate that a linear decoder using BrainBERT embeddings outperforms the alternatives with 1/5th as much data.

Original: Finally, we show that not only are
Revised: Finally, we show that not only are

Original: BrainBERT embeddings extremely useful practically, they open the doors to new kinds of analyses like investigating properties of the computations being carried out by different brain regions.
Revised: BrainBERT embeddings extremely useful practically, they open the doors to new kinds of analyses like investigating properties of the computations being carried out by different brain regions.

Original: Improving decoding accuracy We compared a pretrained BrainBERT against the baseline models described above (variants of linear decoders and a 5 feed-forward layer network); see table 1.
Revised: Improving decoding accuracy We compared a pretrained BrainBERT against the baseline models described above (variants of linear decoders and a 5 feed-forward layer network); see table 1.

Original: Using the linear decoder (5s time domain input) on the original data, without BrainBERT, we chose the 10 best electrodes across all of the subjects.
Revised: Using the linear decoder (5s time domain input) on the original data, without BrainBERT, we chose thebest electrodes across all of the subjects.

Original: This puts BrainBERT at a disadvantage since its highest performing electrodes might not be included in the top 10 from the linear decoder.
Revised: This puts BrainBERT at a disadvantage since its highest performing electrodes might not be included in the top 10 from the linear decoder.

Original: Then we investigated if BrainBERT could offer a performance improvement when performing the four tasks with data from these 10 electrodes.
Revised: Then we investigated if BrainBERT could offer a performance improvement when performing the four tasks with data from these 10 electrodes.

Original: Decoding BrainBERT embeddings resulted in far higher performance on every task.
Revised: Decoding BrainBERT embeddings resulted in far higher performance on every task.

Original: On average, a linear decoder using BrainBERT had an AUC of 0.83.
Revised: On average, a linear decoder using BrainBERT had an AUC of 0.83.

Original: The baseline deep network had an AUC of 0.63.
Revised: The baseline deep network had an AUC of 0.63.

Original: A linear decoder on the unprocessed data had an AUC of 0.59.
Revised: A linear decoder on the unprocessed data had an AUC of 0.59.

Original: This difference is very meaningful, it takes a result which is marginal with a linear decoder and makes it an unqualified success using BrainBERT.
Revised: This difference is very meaningful: it takes a result which is marginal with a linear decoder and makes it an unqualified success using BrainBERT.

Original: Decoding accuracy without fine-tuning
Revised: Decoding accuracy without fine-tuning

Original: In the previous experiment, we fine-tuned BrainBERT in conjunction with the linear decoder.
Revised: In the previous experiment, we fine-tuned BrainBERT in conjunction with the linear decoder.

Original: In table 2, we report results without fine-tuning, only updating the parameters of the linear decoder.
Revised: In table 2, we report results without fine-tuning, only updating the parameters of the linear decoder.

Original: BrainBERT without fine-tuning has an average AUC of 0.63, on par with the baseline deep network.
Revised: BrainBERT without fine-tuning has an average AUC of 0.63, on par with the baseline deep network.

Original: This is what we mean by BrainBERT being the best of both worlds: it provides the performance of a complex network with many non-linear transformations while only tuning a task-specific linear decoder.
Revised: This is what we mean by BrainBERT being the best of both worlds: it provides the performance of a complex network with many non-linear transformations while only tuning a task-specific linear decoder.

Original: Note that STFT provides the best performance when fine-tuning BrainBERT while superlets appear to be more robust when not fine-tuning.
Revised: Note that STFT provides the best performance when fine-tuning BrainBERT while superlets appear to be more robust when not fine-tuning.

Original: Ablations
Revised: Ablations

Original: BrainBERT might work only because of its structure; even a random network can disentangle part of its input.
Revised: BrainBERT might work only because of its structure; even a random network can disentangle part of its input.

Original: To verify that the pretraining is useful, in both table 1 and tablewe show that BrainBERT with randomly initialized weights is considerably worse at increasing decoding accuracy than the pretrained BrainBERT.
Revised: To verify that the pretraining is useful, in both table 1 and tablewe show that BrainBERT with randomly initialized weights is considerably worse at increasing decoding accuracy than the pretrained BrainBERT.

Original: For example, BrainBERT with STFT and finetuning gains on average 0.23 AUC over its randomized ablation on average (table 1).
Revised: For example, BrainBERT with STFT and finetuning gains on average 0.23 AUC over its randomized ablation on average (table 1).

Original: In BrainBERT, the weights matter, not just the structural prior.
Revised: In BrainBERT, the weights matter, not just the structural prior.

Original: BrainBERT has two other novelties in it: the content aware loss and the superlet adaptive mask.
Revised: BrainBERT has two other novelties in it: the content aware loss and the superlet adaptive mask.

Original: We evaluate the impact on both in table 1 and table 2.
Revised: We evaluate the impact on both in table 1 and table 2.

Original: The content aware loss has a mild impact on performance, e.g., BrainBERT with superlets and fine-tuning gains 0.01 AUC, increasing performance in 3 out of 4 tasks by about 5%.
Revised: The content aware loss has a mild impact on performance, e.g., BrainBERT with superlets and fine-tuning gains 0.01 AUC, increasing performance in 3 out of 4 tasks by about 5%.

Original: The adaptive mask has a similar impact, e.g., on the same model it also gains 0.01 AUC increasing performance on 2 out of 4 tasks by about 5%.
Revised: The adaptive mask has a similar impact, e.g., on the same model it also gains 0.01 AUC increasing performance on 2 out of 4 tasks by about 5%.

Original: Cumulatively these changes have a significant impact on performance, although a second order one compared to
Revised: Cumulatively these changes have a significant impact on performance, although a second order one compared to

Original: BrainBERT itself.
Revised: BrainBERT itself.

Original: Generalizing to new subjects
Revised: Generalizing to new subjects

Original: Since BrainBERT is meant to be used off-the-shelf to provide a performance boost for neuroscientific experiments, it must generalize to never-before-seen subjects with novel electrode locations.
Revised: Since BrainBERT is meant to be used off-the-shelf to provide a performance boost for neuroscientific experiments, it must generalize to never-before-seen subjects with novel electrode locations.

Original: To test this, we perform a hold-one-out analysis.
Revised: To test this, we perform a hold-one-out analysis.

Original: For a given subject, we train two variants of BrainBERT with superlet input, one that includes all subjects and one where the subject is held out.
Revised: For a given subject, we train two variants of BrainBERT with superlet input, one that includes all subjects and one where the subject is held out.

Original: We evaluate the fine-tuning performance on all of the four tasks described above with both variants.
Revised: We evaluate the fine-tuning performance on all of the four tasks described above with both variants.

Original: In this analysis, we include only the held-out subject’s bestelectrodes (while previously we had included the best 10 electrodes across all subjects).
Revised: In this analysis, we include only the held-out subject’s bestelectrodes (while previously we had included the best 10 electrodes across all subjects).

Original: We find that the performance difference while generalizing to a new subject with new electrode locations is negligible; see fig.
Revised: We find that the performance difference while generalizing to a new subject with new electrode locations is negligible; see fig.

Original: For all cases, BrainBERT still gains a large boost over a linear decoder on the original data.
Revised: For all cases, BrainBERT still gains a large boost over a linear decoder on the original data.

Original: BrainBERT generalizes extremely well to new subjects.
Revised: BrainBERT generalizes extremely well to new subjects.

Original: Improved data efficiency
Revised: Improved data efficiency

Original: Not only does BrainBERT increase performance, it does so with far fewer training examples than a linear decoder or the baseline deep network require.
Revised: Not only does BrainBERT increase performance, it does so with far fewer training examples than a linear decoder or the baseline deep network require.

Original: In fig.
Revised: In fig.

Original: 5 we compare the decoding AUC of baseline models and BrainBERT variants as a function of training examples.
Revised: 5, we compare the decoding AUC of baseline models and BrainBERT variants as a function of training examples.

Original: All models are trained on the data of a single electrode in the left superior temporal gyrus, which was selected from a full-brain analysis (see fig. 3) to find the electrode with the best combined ranking of linear decoding (5s time duration) and BrainBERT performance.
Revised: All models are trained on the data of a single electrode in the left superior temporal gyrus, which was selected from a full-brain analysis (see fig. 3) to find the electrode with the best combined ranking of linear decoding (5s time duration) and BrainBERT performance.

Original: BrainBERT has a much steeper learning curve, it achieves the performance of the best baseline model with 1000 examples with only 150 examples.
Revised: BrainBERT has a much steeper learning curve: it achieves the performance of the best baseline model with 1,000 examples with only 150 examples.

Original: In addition, if more examples are available, BrainBERT is able to use them and further increase its performance.
Revised: In addition, if more examples are available, BrainBERT is able to use them and further increase its performance.

Original: Intrinsic dimension
Revised: Intrinsic dimension

Original: BrainBERT’s embeddings provide value beyond just their immediate uses in increasing the performance of decoders.
Revised: BrainBERT’s embeddings provide value beyond just their immediate uses in increasing the performance of decoders.

Original: We demonstrate a first use of these embeddings to investigate a property of the computations in different areas of the brain without the restrictive lens of a task.
Revised: We demonstrate a first use of these embeddings to investigate a property of the computations in different areas of the brain without the restrictive lens of a task.

Original: We ask, what is the representation dimensionality (ID) of the neural activity?
Revised: We ask: what is the representation dimensionality (ID) of the neural activity?

Original: In other words, what is the minimal representation of that activity?
Revised: In other words, what is the minimal representation of that activity?

Original: Work with artificial networks has shown that this quantity can separate input and output regions from regions in a network that perform computations, such that high ID regions reflect increased intermediate preprocessing of the data (Ansuini et al., 2019).
Revised: Work with artificial networks has shown that this quantity can separate input and output regions from regions in a network that perform computations, such that high ID regions reflect increased intermediate preprocessing of the data (Ansuini et al., 2019).

Original: Given a pretrained BrainBERT model, we estimate the intrinsic dimension at each electrode by way of principal component analysis (PCA); see appendix I. For each electrode, we find the number of PCs required to account for 95% of the variance in the signal (see fig. 6).
Revised: Given a pretrained BrainBERT model, we estimate the intrinsic dimension at each electrode by way of principal component analysis (PCA); see appendix I. For each electrode, we find the number of PCs required to account for 95% of the variance in the signal (see fig. 6).

Original: We find that results are locally consistent, and that intrinsic dimension seems to vary fairly smoothly.
Revised: We find that results are locally consistent, and that intrinsic dimension seems to vary fairly smoothly.

Original: To determine where high ID processing is located, we take the electrodes which fall in the top 10-th percentile.
Revised: To determine where high ID processing is located, we take the electrodes which fall in the top 10-th percentile.

Original: These electrodes mainly fall in the frontal and temporal lobes.
Revised: These electrodes mainly fall in the frontal and temporal lobes.

Original: Among these electrodes, the regions with the highest mean ID are the supramarginal gyrus, which is involved with phonological processing (Deschamps et al., 2014), the lateral obitofrontal cortex, involved with sensory integration (Rolls, 2004), and the amygdala, involved with emotion (Gallagher & Chiba, 1996).
Revised: Among these electrodes, the regions with the highest mean ID are the supramarginal gyrus, which is involved with phonological processing (Deschamps et al., 2014), the lateral obitofrontal cortex, involved with sensory integration (Rolls, 2004), and the amygdala, involved with emotion (Gallagher & Chiba, 1996).

Original: The intrinsic dimension is computed without respect to any decoding task, e.g. speech vs. non-speech or volume classification, and provides a novel view of functional regions in a task agnostic way.
Revised: The intrinsic dimension is computed without respect to any decoding task, e.g. speech vs. non-speech or volume classification, and provides a novel view of functional regions in a task agnostic way.

Original: Investigations into the embedding space of models of neural data like BrainBERT are in their infancy.
Revised: Investigations into the embedding space of models of neural data like BrainBERT are in their infancy.

Original: They could in the future lead to an understanding of the relationship between brain regions, the general flow of computation in the brain, and brain state changes over time such as  sleep.
Revised: They could in the future lead to an understanding of the relationship between brain regions, the general flow of computation in the brain, and brain state changes over time such as during sleep.

Original:  R ELATED WORK Self-supervised representations
Revised:  R ELATED WORK Self-supervised representations

Original: Our work builds on masked-spectrogram modeling approaches from the field of speech processing (Liu et al., 2021; 2020; Ling & Liu, 2020), which use a reconstructive loss to learn representations of Mel-scale audio spectrograms.
Revised: Our work builds on masked-spectrogram modeling approaches from the field of speech processing (Liu et al., 2021; 2020; Ling & Liu, 2020), which use a reconstructive loss to learn representations of Mel-scale audio spectrograms.

Original: In other fields, selfsupervised models have been used to produce representations for a variety of downstream applica- tions (Oord et al., 2018), including language (Devlin et al., 2019), audio (Baevski et al., 2020; Hsu et al., 2021), and images (Chen et al., 2020).
Revised: In other fields, selfsupervised models have been used to produce representations for a variety of downstream applica- tions (Oord et al., 2018), including language (Devlin et al., 2019), audio (Baevski et al., 2020; Hsu et al., 2021), and images (Chen et al., 2020).

Original: The representations learned by these models can in turn be used to investigate their input domains.
Revised: The representations learned by these models can in turn be used to investigate their input domains.

Original: Language models have been used to study language processing by examining the semantic and syntactic content of the embeddings (Tenney et al., 2019;
Revised: Language models have been used to study language processing by examining the semantic and syntactic content of the embeddings (Tenney et al., 2019;

Original: Roger et al., 2022) and the dimensionality of the input space (Hernandez & Andreas, 2021).
Revised: Roger et al., 2022) and the dimensionality of the input space (Hernandez & Andreas, 2021).

Original: Our work opens the door for similar studies of neural processing, and in this work we investigate the intrinsic dimensionality of neural computation.
Revised: Our work opens the doors for similar studies of neural processing. Here we investigate the intrinsic dimensionality of neural computation.

Original: Neural decoding
Revised: Neural decoding

Original: Intracranial probes, such as SEEG, have been proposed as key components in the next-generation of brain machine interfaces (Jackson & Hall, 2016; Herff et al., 2020).
Revised: Intracranial probes, such as SEEG, have been proposed as key components in the next-generation of brain machine interfaces (Jackson & Hall, 2016; Herff et al., 2020).

Original: Our work is the first to provide a modern representation learning approach to feature extraction for intracranial signals.
Revised: Our work is the first to provide a modern representation learning approach to feature extraction for intracranial signals.

Original: Previous work gives approaches for other modalities, including fMRI data (Malkiel et al., 2021), and EEG (Kostas et al., 2021;
Revised: Previous work gives approaches for other modalities, including fMRI data (Malkiel et al., 2021), and EEG (Kostas et al., 2021;

Original: Banville et al., 2021).
Revised: Banville et al., 2021).

Original: But the signal obtained for these modalities have fundamentally different characteristics.
Revised: But the signal obtained for these modalities have fundamentally different characteristics.

Original: Importantly, they lack the temporal resolution and spatial precision of intracranial electrodes, which directly measure electrophysiological brain activity from a small population of neurons.
Revised: Importantly, they lack the temporal resolution and spatial precision of intracranial electrodes, which directly measure electrophysiological brain activity from a small population of neurons.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present a self-supervised approach for learning representations of neurophysiological data.
Revised: We present a self-supervised approach for learning representations of neurophysiological data.

Original: The resulting representations have three chief benefits.
Revised: The resulting representations have three chief benefits.

Original: First, they improve the accuracy and dataefficiency of neural decoding.
Revised: First, they improve the accuracy and dataefficiency of neural decoding.

Original: This is no small matter: many core results in neuroscience hinge on whether a linear decoder can perform a certain task.
Revised: This is no small matter: many core results in neuroscience hinge on whether a linear decoder can perform a certain task.

Original: Moreover, neuroscience tends to be in the small data regime where gathering data for even small experiments is a significant investment;  a large reduction in the amount of data required to decode something can make a large difference.
Revised: Moreover, neuroscience tends to be in the small data regime where gathering data for even small experiments is a significant investment; here, a large reduction in the amount of data required to decode something can make a large difference.

Original: Such decoding is also critical to building the next generation of brain machine interfaces.
Revised: Such decoding is also critical to building the next generation of brain machine interfaces.

Original: Second, they provide this performance improvement while maintaining the benefits of linear decoders, i.e., explainability, with the performance of complex decoders.
Revised: Second, they provide this performance improvement while maintaining the benefits of linear decoders, i.e., explainability, with the performance of complex decoders.

Original: This is a direct upgrade of one of the most popular techniques in neuroscience.
Revised: This is a direct upgrade of one of the most popular techniques in neuroscience.

Original: Finally, by providing task-agnostic embeddings that can then be analyzed, they open the door for new investigations of cognitive processes.
Revised: Finally, by providing task-agnostic embeddings that can then be analyzed, they open the door for new investigations of cognitive processes.

Original: The changes in representations across time may reveal mechanisms and dynamics underlying sleep and other brain state changes in a completely data-driven manner.
Revised: The changes in representations across time may reveal mechanisms and dynamics underlying sleep and other brain state changes in a completely data-driven manner.

Original: To further refine the representations that BrainBERT builds, in the future, we intend to train much larger variants on continuous 24/7 recordings from numerous subjects.
Revised: To further refine the representations that BrainBERT builds, in the future, we intend to train much larger variants on continuous 24/7 recordings from numerous subjects.

Original: Fields like NLP have been revolutionized by such models; we hope that neuroscience will see similar benefits.
Revised: Fields like NLP have been revolutionized by such models; we hope that neuroscience will see similar benefits.

Original: BrainBERT will be made available as a resource to the community along with the data and scripts to reproduce the results presented here.
Revised: BrainBERT is available as a resource to the community along with the data and scripts to reproduce the results presented here.

Revised: It should be noted that the pretrained weights that we release have been trained on activity induced by a particular stimulus, i.e., passive movie viewing.

Revised: If the downstream use case departs significantly from this setting, then we recommend pretraining BrainBERT from scratch to improve performance.

Original: Groups investigating intracranial recordings should be able to use BrainBERT as a drop-in addition to their existing pipelines.
Revised: Otherwise, most groups investigating intracranial recordings should be able to use BrainBERT as a drop-in addition to their existing pipelines.

Original: E THICS STATEMENT
Revised: A CKNOWLEDGEMENTS

Revised: This work was supported by the Center for Brains, Minds, and Machines, NSF STC award CCF1231216, the NSF award 2124052, the MIT CSAIL Systems that Learn Initiative, the MIT CSAIL

Revised: Machine Learning Applications Initiative, the MIT-IBM Watson AI Lab, the CBMM-Siemens Graduate Fellowship, the DARPA Artificial Social Intelligence for Successful Teams (ASIST) program, the DARPA Knowledge Management at Scale and Speed (KMASS) program, the United States Air

Revised: Force Research Laboratory and United States Air Force Artificial Intelligence Accelerator under

Revised: Cooperative Agreement Number FA8750-19-2-1000, the Air Force Office of Scientific Research (AFOSR) under award number FA9550-21-1-0014, and the Office of Naval Research under award number N00014-20-1-2589 and award number N00014-20-1-2643.

Revised: The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government.

Revised: The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein We would also like to thank Colin Conwell and Hector Luis Penagos-Vargas for helpful comments and discussion.

Revised: E THICS STATEMENT

Original: Experiments that contributed to this work were approved by IRB.
Revised: Experiments that contributed to this work were approved by IRB.

Original: All subjects consented to participate.
Revised: All subjects consented to participate.

Original: All electrodes were placed according to clinical not research concerns.
Revised: All electrode locations were exclusively dictated by clinical considerations.

Original: We believe that accelerating research in neuroscience and in brain-computer interfaces can have significant positive benefits for society.

Original: It should be noted that the pretrained weights that we release have been pretrained on data from a specific population using activity induced by a particular stimulus, i e., passive movie viewing.

Original: If the downstream use case departs significantly from this setting, then it is possible that pretraining BrainBERT from scratch may improve performance

Original: R EPRODUCIBILITY STATEMENT
Revised: R EPRODUCIBILITY STATEMENT

Original: Code to train models and reproduce the results was submitted as part of the supplementary materials. BrainBERT models will be made public along with the data used to train them.
Revised: Code to train models and reproduce the results was submitted as part of the supplementary materials and can be accessed here: https://github.com/czlwang/BrainBERT .

Original: A S UPERLETS
Revised: A S UPERLETS

Original: Superlet transforms are a technique for time-frequency analysis introduced by Moca et al.
Revised: Superlet transforms are a technique for time-frequency analysis introduced by Moca et al.

Original: The superlet transform is formed from a composite Morlet wavelet transforms.
Revised: The superlet transform is formed from a composite Morlet wavelet transforms.

Original: In contrast to traditional time-frequency techniques, such as the Short-time Fourier Transform (STFT) which has a fixed time-frequency resolution trade-off for all frequencies, the Morlet wavelet has lower temporal resolution at lower frequencies.
Revised: In contrast to traditional time-frequency techniques, such as the Short-time Fourier Transform (STFT) which has a fixed time-frequency resolution trade-off for all frequencies, the Morlet wavelet has lower temporal resolution at lower frequencies.

Original: This variable trade-off allows the resulting representation to better capture the dynamics of neural signal, for which high frequency oscillations often occur in short bursts and low frequency oscillations persist for longer durations.
Revised: This variable trade-off allows the resulting representation to better capture the dynamics of neural signal, for which high frequency oscillations often occur in short bursts and low frequency oscillations persist for longer durations.

Original: However, the Morlet transform is not Pareto optimal, and the superlet transform makes improvements to both time and frequency resolution simultaneously in order to better capture the self-similar oscillations across different frequencies which are often present in neural signal.
Revised: However, the Morlet transform is not Pareto optimal, and the superlet transform makes improvements to both time and frequency resolution simultaneously in order to better capture the self-similar oscillations across different frequencies which are often present in neural signal.

Original: In this section, we briefly summarize the content of Moca et al.
Revised: In this section, we briefly summarize the content of Moca et al.

Original: (2021) relevant to our work.
Revised: (2021) relevant to our work.

Original: Consider the following formulation of the Morlet wavelet for a given frequency of interest f :
Revised: Consider the following formulation of the Morlet wavelet for a given frequency of interest f :

Original: [Equation]
Revised: [Equation]

Original: Here, c is the number of cycles, which is a parameter that controls the time-frequency resolution trade-off.
Revised: Here, c is the number of cycles, which is a parameter that controls the time-frequency resolution trade-off.

Original: The expression for the wavelet on the right side of the equation eq.
Revised: The expression for the wavelet on the right side of the equation eq.

Original: (4) can be understood as being composed of two terms: the first term can be thought of as a complex wave function (Euler’s formula), and the second term can be understood as a Gaussian windowing function that modulates the amplitude of the wave.
Revised: (4) can be understood as being composed of two terms: the first term can be thought of as a complex wave function (Euler’s formula), and the second term can be understood as a Gaussian windowing function that modulates the amplitude of the wave.

Original: Note that the width σ of the windowing function falls off with frequency.
Revised: Note that the width σ of the windowing function falls off with frequency.

Original: For a given frequency f , the superlet transform of order o is the geometric mean of Morlet transforms ψ c,f for a range of values c ∈ [1 , o ] .
Revised: For a given frequency f , the superlet transform of order o is the geometric mean of Morlet transforms ψ c,f for a range of values c ∈ [1 , o ] .

Original: The superlet representation at a frequency of interest f for a signal x is then:
Revised: The superlet representation at a frequency of interest f for a signal x is then:

Original: [Equation]
Revised: [Equation]

Original: where ∗ is the complex convolution operator.
Revised: where ∗ is the complex convolution operator.

Original: Although, compared to the Morlet wavelet transform, the overall resolution is improved, the f in the denominator of eq.
Revised: Although, compared to the Morlet wavelet transform, the overall resolution is improved, the f in the denominator of eq.

Original: (5) means that there continues to be relatively lower time resolution at lower frequencies.
Revised: (5) means that there continues to be relatively lower time resolution at lower frequencies.

Original: In other words, there is more temporal smearing at the bottom of the spectrogram than at the top.
Revised: In other words, there is more temporal smearing at the bottom of the spectrogram than at the top.

Original: To account for this, we use an adaptive masking strategy (see section 2: Pretraining).
Revised: To account for this, we use an adaptive masking strategy (see section 2: Pretraining).

Original: B T IME - FREQUENCY REPRESENTATION PARAMETERS
Revised: B T IME - FREQUENCY REPRESENTATION PARAMETERS

Original: For the STFT, we use a window size of 400 samples ( ≈ 200 ms ) with an overlap of 350 samples ( ≈ 175 ms ) with frequency channels evenly spaced from 0 to 200Hz.
Revised: For the STFT, we use a window size of 400 samples ( ≈ 200 ms ) with an overlap of 350 samples ( ≈ 175 ms ) with frequency channels evenly spaced from 0 to 200Hz.

Original: For the superlet, we use c 1 = 1 with orders o = 3 − 30 .
Revised: For the superlet, we use c 1 = 1 with orders o = 3 − 30 .

Original: The frequencies of interest are evenly spaced from 0.1 to 200Hz.
Revised: The frequencies of interest are evenly spaced from 0.1 to 200Hz.

Original: To match the down-sampling rate of the STFT, the superlet representation is decimated by a factor of 50.
Revised: To match the down-sampling rate of the STFT, the superlet representation is decimated by a factor of 50.

Original: Finally, for both representations, we remove 5 columns from each array on either side ( ≈ 250 ms ) to account for edge-effects.
Revised: Finally, for both representations, we remove 5 columns from each array on either side ( ≈ 250 ms ) to account for edge-effects.

Original: Superlets take the composite of many Morlet wavelets.
Revised: Superlets take the composite of many Morlet wavelets.

Original: Thus, a superlet is simply a collection of
Revised: Thus, a superlet is simply a collection of

Original: Morlet wavlets with different c :
Revised: Morlet wavlets with different c :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where o is the order of the superlet.
Revised: where o is the order of the superlet.

Original: For this work, we use the multiplicative, adaptive superlet transform.
Revised: For this work, we use the multiplicative, adaptive superlet transform.

Original: In the multiplicative version of the transform, c i = c 1 · i .
Revised: In the multiplicative version of the transform, c i = c 1 · i .

Original: And in the adaptive version, the order of the superlet changes according to frequency:
Revised: And in the adaptive version, the order of the superlet changes according to frequency:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: C M ASKING TIME AND FREQUENCY
Revised: C M ASKING TIME AND FREQUENCY

Original: During pre-training, BrainBERT receives an augmented version of the spectrogram, from which random time and frequency bands have been removed.
Revised: During pre-training, BrainBERT receives an augmented version of the spectrogram, from which random time and frequency bands have been removed.

Original: In this work, we use two masking strategies: static masking, which is suited for spectrograms of fixed time-frequency resolution, and adaptive masking, which is appropriate for spectrograms with a variable trade-off in time-frequency resolution.
Revised: In this work, we use two masking strategies: static masking, which is suited for spectrograms of fixed time-frequency resolution, and adaptive masking, which is appropriate for spectrograms with a variable trade-off in time-frequency resolution.

Original: Static masking
Revised: Static masking

Original: For our static masking procedure, we adapt the work of (Liu et al., 2020; 2021).
Revised: For our static masking procedure, we adapt the work of (Liu et al., 2020; 2021).

Original: The width of the temporal mask is randomly selected from the range [1 , 5] , and the width of the frequency mask is randomly selected from the range [1 , 2] .
Revised: The width of the temporal mask is randomly selected from the range [1 , 5] , and the width of the frequency mask is randomly selected from the range [1 , 2] .

Original: Adaptive masking
Revised: Adaptive masking

Original: For the superlet transform, we use adaptive temporal and frequency masks.
Revised: For the superlet transform, we use adaptive temporal and frequency masks.

Original: The temporal width of the adaptive mask as a function of frequency f is given by:
Revised: The temporal width of the adaptive mask as a function of frequency f is given by:

Original: [Equation]
Revised: [Equation]

Original:  where m is the minimum width of the mask, which is randomly selected from { 1 , 2 } per example.
Revised: (9) where m is the minimum width of the mask, which is randomly selected from { 1 , 2 } per example.

Original: See fig.
Revised: See fig.

Original: 7 for visualization.
Revised: 7 for visualization.

Original: The width of the frequency masks is given as a function of frequency f :
Revised: The width of the frequency masks is given as a function of frequency f :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: The parameters of these equations are selected to roughly match the width of the masks used for the
Revised: The parameters of these equations are selected to roughly match the width of the masks used for the

Original: STFT.
Revised: STFT.

Original: D M ASKING PROCEDURE
Revised: D M ASKING PROCEDURE

Original: The masking procedure is described in algorithm 1 (see section 2: Pretraining).
Revised: The masking procedure is described in algorithm 1 (see section 2: Pretraining).

Original: Note that intervals are masked without overlap.
Revised: Note that intervals are masked without overlap.

Original: The procedure is adapted from Liu et al.
Revised: The procedure is adapted from Liu et al.

Original: (2021) and Devlin et al.
Revised: (2021) and Devlin et al.

Original: During pre-training, we use p mask = 0 .
Revised: During pre-training, we use p mask = 0 .

Original: 05 , p ID = 0 .
Revised: 05 , p ID = 0 .

Original: 1 , and p replace = 0 .
Revised: 1 , and p replace = 0 .

Original: The purpose of letting some segments go unaugmented is to make the distribution seen at train time and test time more similar.
Revised: The purpose of letting some segments go unaugmented is to make the distribution seen at train time and test time more similar.

Original: The purpose of replacing some segments with random signal is to prevent the model from simply learning the identity function.
Revised: The purpose of replacing some segments with random signal is to prevent the model from simply learning the identity function.

Original: Algorithm Y ← n × m spectrogram i ←while i ≤ m do p ∼ Unif (0 , 1) if p < p mask then l ∼ ⌊ Unif ( step min , step max + 1) ⌋ q ∼ Unif (0 , 1) if q < p ID then pass else if p ID ≤ q < p ID + p replace then j ← Unif (0 , m − l )
Revised: Y ← n × m spectrogram i ←while i ≤ m do p ∼ Unif (0 , 1) if p < p mask then l ∼ ⌊ Unif ( step min , step max + 1) ⌋ q ∼ Unif (0 , 1) if q < p ID then pass else if p ID ≤ q < p ID + p replace then j ← Unif (0 , m − l )

Original: Y [: , i : i + l ] ← Y [: , j : j + l ] else
Revised: Y [: , i : i + l ] ← Y [: , j : j + l ] else

Original: Y [: , i : i + l ] ←end if i ← i + l end if end while
Revised: Y [: , i : i + l ] ←end if i ← i + l end if end while

Original: E B ASELINES Linear baselines (time domain)
Revised: E B ASELINES Linear baselines (time domain)

Original: We train two feed forward networks with layer dimensions [ d input , 1] and sigmoid activations.
Revised: We train two feed forward networks with layer dimensions [ d input , 1] and sigmoid activations.

Original: The first model, Linear (5s time domain), takes 5s of time domain input, sampled at 2048 Hz, so d input = 10 , 240 .
Revised: The first model, Linear (5s time domain), takes 5s of time domain input, sampled at 2048 Hz, so d input = 10 , 240 .

Original: The second model, Linear (0.25s time domain), takes 5s of input, so d input = 
Revised: The second model, Linear (0.25s time domain), takes 0.25s of input, so d input = 512 .

Original: Deep neural network (time domain)
Revised: Deep neural network (time domain)

Original: We train a deep neural network, Deep NN (5s time domain), which consists of 5 stacked feed forward layers with dimensions
Revised: We train a deep neural network, Deep NN (5s time domain), which consists of 5 stacked feed forward layers with dimensions

Original: [ d input , 1024 , 512 , 256 , 128] and
Revised: [ d input , 1024 , 512 , 256 , 128] and

Original: ReLU activations on the hidden layers and a sigmoid activation on the output layer.
Revised: ReLU activations on the hidden layers and a sigmoid activation on the output layer.

Original: The network takes 5s of time domain input, sampled at a rate of 2048 Hz, so d input = 10 , 240 .
Revised: The network takes 5s of time domain input, sampled at a rate of 2048 Hz, so d input = 10 , 240 .

Original: Linear baselines (time-frequency domain)
Revised: Linear baselines (time-frequency domain)

Original: We train two feed forward linear networks with dimension [ d input , 1] and sigmoid activations that take the time-frequency representations as input.
Revised: We train two feed forward linear networks with dimension [ d input , 1] and sigmoid activations that take the time-frequency representations as input.

Original: Linear (.25s STFT) and Linear (.25s superlet) receive the time-frequency representation, averaged across time in a ≈ 244 ms interval, centered on the example.
Revised: Linear (.25s STFT) and Linear (.25s superlet) receive the time-frequency representation, averaged across time in a ≈ 244 ms interval, centered on the example.

Original: More precisely, given a time-frequency representation Y ∈ R n × 2 l , the model receives the vector obtained by averaging Y : ,l − 5 ,l +5 across the time axis.
Revised: More precisely, given a time-frequency representation Y ∈ R n × 2 l , the model receives the vector obtained by averaging Y : ,l − 5 ,l +5 across the time axis.

Original: There are n = 40 frequency bands, so d input = 40 .
Revised: There are n = 40 frequency bands, so d input = 40 .

Original: This is the same form as the input that our classification network receives from BrainBERT (see section 3).
Revised: This is the same form as the input that our classification network receives from BrainBERT (see section 3).

Original: F P REPROCESSING
Revised: F P REPROCESSING

Original: Data was high-pass filtered at 0.1Hz.
Revised: Data was high-pass filtered at 0.1Hz.

Original: Line noise at 60Hz and its harmonics were removed.
Revised: Line noise at 60Hz and its harmonics were removed.

Original: Each electrode was re-referenced by subtracting out the mean signal from the two adjacent electrodes on the same shaft (Laplacian re-referencing).
Revised: Each electrode was re-referenced by subtracting out the mean signal from the two adjacent electrodes on the same shaft (Laplacian re-referencing).

Original: This decreases the cross-correlation between electrodes (Li et al., 2018).
Revised: This decreases the cross-correlation between electrodes (Li et al., 2018).

Original: Only electrodes which can be Laplacian re-referenced, i.e., have neighbors, are included in the pretraining data.
Revised: Only electrodes which can be Laplacian re-referenced, i.e., have neighbors, are included in the pretraining data.

Original: Finally, signals from all electrodes are visually inspected, and recordings which show obvious signs of corruption are removed.
Revised: Finally, signals from all electrodes are visually inspected, and recordings which show obvious signs of corruption are removed.

Original: In total, data from 1,249 electrodes ( ≈ 74% of total electrodes; 4,551 electrode-hours) are used for pretraining.
Revised: In total, data from 1,249 electrodes ( ≈ 74% of total electrodes; 4,551 electrode-hours) are used for pretraining.

Original: G P RETRAINING PARAMETERS
Revised: G P RETRAINING PARAMETERS

Original: All layers ( N = 6 ) in the encoder stack are set with the following parameters:
Revised: All layers ( N = 6 ) in the encoder stack are set with the following parameters:

Original: d h = 768 , H = 12 , and p dropout = 0 .
Revised: d h = 768 , H = 12 , and p dropout = 0 .

Original: We pretrain the BrainBERT model with the LAMB optimizer (You et al., 2019) and lr = 1 e − 4 .
Revised: We pretrain the BrainBERT model with the LAMB optimizer (You et al., 2019) and lr = 1 e − 4 .

Original: We use a batch size of n batch = 256 , train for 500k steps, and record the validation performance every 1000 steps.
Revised: We use a batch size of n batch = 256 , train for 500k steps, and record the validation performance every 1,000 steps.

Original: Then, the weights with the best validation performance are retained.
Revised: Then, the weights with the best validation performance are retained.

Original: During pretraining, the BrainBERT representations are passed to a spectrogram prediction head which consists of a two feed forward linear layers.
Revised: During pretraining, the BrainBERT representations are passed to a spectrogram prediction head which consists of a two feed forward linear layers.

Original: The first layer, a hidden layer, has dimension 
Revised: The first layer, a hidden layer, has dimension d = 768 and a GeLU activation. The second layer, the layer, has dimension d = the interest see appendix B.

Original: As part of our ablation test, we also include a BrainBERT with randomly initialized weights.
Revised: As part of our ablation test, we also include a BrainBERT with randomly initialized weights.

Original: For this, weights are initialized according to the uniform Xavier scheme (Glorot & Bengio, 2010).
Revised: For this, weights are initialized according to the uniform Xavier scheme (Glorot & Bengio, 2010).

Original: H F INE - TUNING TRAINING PARAMETERS
Revised: H F INE - TUNING TRAINING PARAMETERS

Original: The classifier is a fully connected linear layer with d in = 768 , d out = 1 and a sigmoid activation.
Revised: The classifier is a fully connected linear layer with d in = 768 , d out = 1 and a sigmoid activation.

Original: The model is trained using a binary cross entropy loss.
Revised: The model is trained using a binary cross entropy loss.

Original: When fine-tuning, we use the AdamW optimizer, with lr = 1 e − 3 for the classification head and lr = 1 e − 4 for the BrainBERT weights.
Revised: When fine-tuning, we use the AdamW optimizer, with lr = 1 e − 3 for the classification head and lr = 1 e − 4 for the BrainBERT weights.

Original: When training with frozen BrainBERT weights, we use the AdamW optimizer with lr = 1 e − 3 .
Revised: When training with frozen BrainBERT weights, we use the AdamW optimizer with lr = 1 e − 3 .

Original: All models, both this classifier and the baseline models, are trained for 1,000 updates.
Revised: All models, both this classifier and the baseline models, are trained for 1,000 updates.

Original: Validation performance is computed every 100 updates, and test accuracy is reported using the weights with the best validation performance.
Revised: Validation performance is computed every 100 updates, and test accuracy is reported using the weights with the best validation performance.

Original: I I NTRINSIC D IMENSION
Revised: I I NTRINSIC D IMENSION

Original: We use a standard method for determining intrinsic dimension (Fan et al., 2010).
Revised: We use a standard method for determining intrinsic dimension (Fan et al., 2010).

Original: For a given threshold β , the intrinsic dimension is the d ∈ N such that the ratio of explained variance for d dimensions of a N dimensional PCA is above β :
Revised: For a given threshold β , the intrinsic dimension is the d ∈ N such that the ratio of explained variance for d dimensions of a N dimensional PCA is above β :

Original: [Equation]
Revised: [Equation]

Original: We use a PCA of N = 200 and a threshold of β = 0 .
Revised: We use a PCA of N = 200 and a threshold of β = 0 .

Original: For each session in the held out data, we segment the neural recordings into 5s intervals, for which the spectrogram is a matrix Y ∈ R n × m .
Revised: For each session in the held out data, we segment the neural recordings into 5s intervals, for which the spectrogram is a matrix Y ∈ R n × m .

Original: We produce the BrainBERT embeddings E ∈ R n × m for each spectrogram, and average across the time-dimension to obtain a single n -dimensional vector representation of the interval.
Revised: We produce the BrainBERT embeddings E ∈ R n × m for each spectrogram, and average across the time-dimension to obtain a single n -dimensional vector representation of the interval.

Original: For each electrode, we find the intrinsic dimension of the manifold on which these BrainBERT embeddings lie.
Revised: For each electrode, we find the intrinsic dimension of the manifold on which these BrainBERT embeddings lie.

Original: We find the regions with the highest intrinsic dimension, by first taking the electrodes in the top 10-th percentile.
Revised: We find the regions with the highest intrinsic dimension, by first taking the electrodes in the top 10-th percentile.

Original: Among these electrodes, most of them lie in the precentral gyrus (13.5%), the lateral orbitofrontal cortex (10.1%) and the pars opercularis (9.0%).
Revised: Among these electrodes, most of them lie in the precentral gyrus (13.5%), the lateral orbitofrontal cortex (10.1%), and the pars opercularis (9.0%).

Original: However, it should be noted that the electrodes are not distributed evenly across the cortex, so to get a sense of which regions have the highest ID, we should normalize across the number of electrodes in a region.
Revised: However, it should be noted that the electrodes are not distributed evenly across the cortex, so to get a sense of which regions have the highest ID, we should normalize across the number of electrodes in a region.

Original: We find that the regions with the highest mean ID are the supramarginal gyrus, the lateral orbitofrontal cortex, and the amygdala.
Revised: We find that the regions with the highest mean ID are the supramarginal gyrus, the lateral orbitofrontal cortex, and the amygdala.

Original: I. T ASKS Sentence onset
Revised: I. T ASKS Sentence onset

Original: Once the movie transcript has been aligned to the brain activity, the brain activity which corresponds to the sentence onsets can be collected, each embedded in 5s of context.
Revised: Once the movie transcript has been aligned to the brain activity, the brain activity which corresponds to the sentence onsets can be collected, each embedded in 5s of context.

Original: These intervals form the set of positive examples.
Revised: These intervals form the set of positive examples.

Original: Speech vs. non-speech
Revised: Speech vs. non-speech

Original: For this task, the positive examples are formed from the 5s of context surrounding any word.
Revised: For this task, the positive examples are formed from the 5s of context surrounding any word.

Original: For both this task and the sentence onset task, the negative examples are formed by finding 1s intervals which do not overlap with any intervals of speech audio.
Revised: For both this task and the sentence onset task, the negative examples are formed by finding 1s intervals which do not overlap with any intervals of speech audio.

Original: Each example is embedded in 5s of context.
Revised: Each example is embedded in 5s of context.

Original: Volume
Revised: Volume

Original: For each word in the transcript, the volume is measured as the root-mean-square for a 500ms interval starting from the word onset.
Revised: For each word in the transcript, the volume is measured as the root-mean-square for a 500ms interval starting from the word onset.

Original: Those words which lie one standard deviation above the mean volume are labeled as “high-volume”.
Revised: Those words which lie one standard deviation above the mean volume are labeled as “high-volume”.

Original: Those words which lie one standard deviation below the mean volume are labeled as ”low-volume”.
Revised: Those words which lie one standard deviation below the mean volume are labeled as ”low-volume”.

Original: Pitch
Revised: Pitch

Original: For each word in the transcript, the pitch is extracted using librosa’s piptrack function over a Mel-spectrogram (sampling rate 48,000 Hz, FFT window length of 2048, hop length of 512, and 128 mel filters).
Revised: For each word in the transcript, the pitch is extracted using librosa’s piptrack function over a Mel-spectrogram (sampling rate 48,000 Hz, FFT window length of 2048, hop length of 512, and 128 mel filters).

Original: Those words which lie one standard deviation above the mean pitch are labeled as “high-pitch”.
Revised: Those words which lie one standard deviation above the mean pitch are labeled as “high-pitch”.

Original: Those words which lie one standard deviation below the mean pitch are labeled as “low-vpitch”.
Revised: Those words which lie one standard deviation below the mean pitch are labeled as “low-vpitch”.

Original: J D ATASET STATISTICS
Revised: J D ATASET STATISTICS

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: Guardians

Original: [Table]
Revised: [Table]

Original: K E LECTRODE VISUALIZATION
Revised: K E LECTRODE VISUALIZATION

Original: For each subject, pre-operative T1 MRI scans without contrast were processed with FreeSurfer Fischl et al. (2004), which performed skull stripping, white matter segmentation, and surface generation.
Revised: For each subject, pre-operative T1 MRI scans without contrast were processed with FreeSurfer (Fischl et al., 2004), which performed skull stripping, white matter segmentation, and surface generation.

Original: iELVis
Revised: iELVis

Original: Groppe et al. (2017) was used to co-register a post-operative fluoroscopy scan to the preoperative MRI.
Revised: (Groppe et al., 2017) was used to co-register a post-operative fluoroscopy scan to the preoperative MRI.

Original: Electrodes were manually identified using BioImageSuite Joshi et al. (2011), and then assigned to one of 94 regions (according to the Desikan-Killiany atlas (Desikan et al., 2006)) using FreeSurfer’s automatic parcellation.
Revised: Electrodes were manually identified using BioImageSuite (Joshi et al., 2011), and then assigned to one of 94 regions (according to the Desikan-Killiany atlas (Desikan et al., 2006)) using FreeSurfer’s automatic parcellation.

Original: The alignment to the atlas was manually verified for each subject.
Revised: The alignment to the atlas was manually verified for each subject.

Original: Depth electrodes in the white matter, if they were within 1.5 mm of the gray-white matter boundary, were projected to the nearest point on that boundary, and were labeled as coming from that region (for the purposes of region analyses).
Revised: Depth electrodes in the white matter, if they were within 1.5 mm of the gray-white matter boundary, were projected to the nearest point on that boundary, and were labeled as coming from that region (for the purposes of region analyses).

Original: This procedure is very similar to the post brain-shift correction methods used for electrocorticography electrodes Yang et al.
Revised: This procedure is very similar to the post brain-shift correction methods used for electrocorticography electrodes (Yang et al., 2012).

Original: For solely visualization purposes, all electrodes identified to lie in the gray matter or on the gray-white matter boundary were first projected to the pial surface (using nearest neighbors), and then mapped to an average brain (using Freesurfer’s fsaverage atlas) for the visualizations shown in fig.
Revised: For solely visualization purposes, all electrodes identified to lie in the gray matter or on the gray-white matter boundary were first projected to the pial surface (using nearest neighbors), and then mapped to an average brain (using Freesurfer’s fsaverage atlas) for the visualizations shown in fig.

Original: 3 and fig.
Revised: 3 and fig.

Original: L P RETRAINING PERFORMANCE
Revised: L P RETRAINING PERFORMANCE

Original: [Figure]
Revised: [Figure]

Original: N S UPPLEMENTARY R ESULTS
Revised: N S UPPLEMENTARY R ESULTS

Original: [Table]
Revised: [Table]

Revised: , averaged across all tasks, between BrainBERT pretrained with and without a particular subject subject.

Revised: See table 6 for

Original: Sentenceonset
Revised: Sentenceonset

Original: Speech/Nonspeech
Revised: Speech/Nonspeech

Original: Pitch
Revised: Pitch

Original: Volume
Revised: Volume

Original: Average pretrain w/ subject.
Revised: Average pretrain w/ subject.

Original: 74 ± .. 85 ± .. 79 ± .. 92 ± ..
Revised: 74 ± .. 85 ± .. 79 ± .. 92 ± ..

Original: 82 ± .pretrain w/o subject.
Revised: 82 ± .pretrain w/o subject.

Original: 73 ± .. 85 ± .. 77 ± .. 92 ± .. 82 ± .
Revised: 73 ± .. 85 ± .. 77 ± .. 92 ± .. 82 ± .

Original: Linear (5s) .
Revised: Linear (5s) .

Original: 62 ± .. 56 ± .. 52 ± .. 62 ± ..
Revised: 62 ± .. 56 ± .. 52 ± .. 62 ± ..

Original: 58 ± .pretrain w/ subject.
Revised: 58 ± .pretrain w/ subject.

Original: 61 ± .. 83 ± ..
Revised: 61 ± .. 83 ± ..

Original: 74 ± .. 73 ± ..
Revised: 74 ± .. 73 ± ..

Original: 73 ± .pretrain w/o subject.
Revised: 73 ± .pretrain w/o subject.

Original: 62 ± .. 85 ± .. 77 ± ..
Revised: 62 ± .. 85 ± .. 77 ± ..

Original: 82 ± .. 77 ± .
Revised: 82 ± .. 77 ± .

Original: Linear (5s) .
Revised: Linear (5s) .

Original: 59 ± .. 53 ± .. 52 ± .. 61 ± ..
Revised: 59 ± .. 53 ± .. 52 ± .. 61 ± ..

Original: 56 ± .pretrain w/ subject.
Revised: 56 ± .pretrain w/ subject.

Original: 85 ± .. 81 ± .. 65 ± .. 86 ± .. 79 ± .pretrain w/o subject.
Revised: 85 ± .. 81 ± .. 65 ± .. 86 ± .. 79 ± .pretrain w/o subject.

Original: 84 ± .. 78 ± .. 61 ± .. 88 ± .. 78 ± .
Revised: 84 ± .. 78 ± .. 61 ± .. 88 ± .. 78 ± .

Original: Linear (5s) .
Revised: Linear (5s) .

Original: 64 ± .. 57 ± .. 54 ± .. 56 ± ..
Revised: 64 ± .. 57 ± .. 54 ± .. 56 ± ..

Original: 58 ± .pretrain w/ subject.
Revised: 58 ± .pretrain w/ subject.

Original: 57 ± .. 94 ± .. 66 ± .. 69 ± .. 72 ± .pretrain w/o subject.
Revised: 57 ± .. 94 ± .. 66 ± .. 69 ± .. 72 ± .pretrain w/o subject.

Original: 60 ± .. 91 ± .. 66 ± ..
Revised: 60 ± .. 91 ± .. 66 ± ..

Original: 75 ± .. 73 ± .
Revised: 75 ± .. 73 ± .

Original: Linear (5s) .
Revised: Linear (5s) .

Original: 58 ± .. 54 ± .. 61 ± .. 64 ± ..
Revised: 58 ± .. 54 ± .. 61 ± .. 64 ± ..

Original: 59 ± .pretrain w/ subject.
Revised: 59 ± .pretrain w/ subject.

Original: 63 ± ..
Revised: 63 ± ..

Original: 76 ± .. 69 ± ..
Revised: 76 ± .. 69 ± ..

Original: 70 ± ..
Revised: 70 ± ..

Original: 70 ± .pretrain w/o subject.
Revised: 70 ± .pretrain w/o subject.

Original: 64 ± .. 73 ± ..
Revised: 64 ± .. 73 ± ..

Original: 76 ± .. 78 ± .. 73 ± .
Revised: 76 ± .. 78 ± .. 73 ± .

Original: Linear (5s) .
Revised: Linear (5s) .

Original: 56 ± .. 53 ± .. 53 ± ..
Revised: 56 ± .. 53 ± .. 53 ± ..

Original: 58 ± .. 55 ± .pretrain w/ subject.
Revised: 58 ± .. 55 ± .pretrain w/ subject.

Original: 74 ± .. 91 ± .. 68 ± .. 83 ± .. 79 ± .pretrain w/o subject.
Revised: 74 ± .. 91 ± .. 68 ± .. 83 ± .. 79 ± .pretrain w/o subject.

Original: 75 ± .. 86 ± .. 66 ± .. 85 ± .. 78 ± .
Revised: 75 ± .. 86 ± .. 66 ± .. 85 ± .. 78 ± .

Original: Linear (5s) .
Revised: Linear (5s) .

Original: 63 ± .. 57 ± ..
Revised: 63 ± .. 57 ± ..

Original: 58 ± .. 53 ± ..
Revised: 58 ± .. 53 ± ..

Original: 58 ± .pretrain w/ subject.
Revised: 58 ± .pretrain w/ subject.

Original: 71 ± .. 75 ± ..
Revised: 71 ± .. 75 ± ..

Original: 65 ± .. 84 ± ..
Revised: 65 ± .. 84 ± ..

Original: 74 ± .pretrain w/o subject.
Revised: 74 ± .pretrain w/o subject.

Original: 74 ± .. 81 ± ..
Revised: 74 ± .. 81 ± ..

Original: 65 ± .. 89 ± .. 77 ± .
Revised: 65 ± .. 89 ± .. 77 ± .

Original: Linear (5s) .
Revised: Linear (5s) .

Original: 56 ± .. 53 ± .. 52 ± .. 55 ± ..
Revised: 56 ± .. 53 ± .. 52 ± .. 55 ± ..

Original: 54 ± .pretrain w/ subject.
Revised: 54 ± .pretrain w/ subject.

Original: 68 ± .. 89 ± .. 66 ± ..
Revised: 68 ± .. 89 ± .. 66 ± ..

Original: 76 ± .. 74 ± .pretrain w/o subject.
Revised: 76 ± .. 74 ± .pretrain w/o subject.

Original: 70 ± .. 85 ± .. 72 ± .. 75 ± ..
Revised: 70 ± .. 85 ± .. 72 ± .. 75 ± ..

Original: 76 ± .
Revised: 76 ± .

Original: Linear (5s) .
Revised: Linear (5s) .

Original: 61 ± .. 54 ± .. 59 ± .. 57 ± ..
Revised: 61 ± .. 54 ± .. 59 ± .. 57 ± ..

Original: 58 ± .pretrain w/ subject.
Revised: 58 ± .pretrain w/ subject.

Original: 75 ± .. 80 ± ..
Revised: 75 ± .. 80 ± ..

Original: 75 ± .. 59 ± .. 72 ± .pretrain w/o subject.
Revised: 75 ± .. 59 ± .. 72 ± .pretrain w/o subject.

Original: 78 ± .. 80 ± ..
Revised: 78 ± .. 80 ± ..

Original: 76 ± .. 56 ± .. 72 ± .
Revised: 76 ± .. 56 ± .. 72 ± .

Original: Linear (5s) .
Revised: Linear (5s) .

Original: 55 ± .. 53 ± .. 53 ± .. 80 ± .
Revised: 55 ± .. 53 ± .. 53 ± .. 80 ± .

Original: [Table]
Revised: [Table]