Original: Toward a realistic model of speech processing in the brain with self-supervised learning
Revised: Toward a realistic model of speech processing in the brain with self-supervised learning

Original: Abstract
Revised: Abstract

Original: Several deep neural networks have recently been shown to generate activations similar to those of the brain in response to the same input.
Revised: Several deep neural networks have recently been shown to generate activations similar to those of the brain in response to the same input.

Original: These algorithms, however, remain largely implausible: they require (1) extraordinarily large amounts of data, (2) unobtainable supervised labels, (3) textual rather than raw sensory input, and / or (4) implausibly large memory (e.g. thousands of contextual words).
Revised: These algorithms, however, remain largely implausible: they require (1) extraordinarily large amounts of data, (2) unobtainable supervised labels, (3) textual rather than raw sensory input, and / or (4) implausibly large memory (e.g. thousands of contextual words).

Original: These elements highlight the need to identify algorithms that, under these limitations, would suffice to account for both behavioral and brain responses.
Revised: These elements highlight the need to identify algorithms that, under these limitations, would suffice to account for both behavioral and brain responses.

Original: Focusing on speech processing, we here hypothesize that self-supervised algorithms trained on the raw waveform constitute a promising candidate.
Revised: Focusing on speech processing, we here hypothesize that self-supervised algorithms trained on the raw waveform constitute a promising candidate.

Original: Specifically, we compare a recent self-supervised model, wav2vec 2.0, to the brain activity of 412 English,
Revised: Specifically, we compare a recent self-supervised model, wav2vec 2.0, to the brain activity of 412 English,

Original: French, and Mandarin individuals recorded with functional Magnetic Resonance
Revised: French, and Mandarin individuals recorded with functional Magnetic Resonance

Original: Imaging (fMRI), while they listened to approximately one hour of audio books.
Revised: Imaging (fMRI), while they listened to approximately one hour of audio books.

Original: First, we show that this algorithm learns brain-like representations with as little as 600 hours of unlabelled speech – a quantity comparable to what infants can be exposed to during language acquisition.
Revised: First, we show that this algorithm learns brain-like representations with as little as 600 hours of unlabelled speech – a quantity comparable to what infants can be exposed to during language acquisition.

Original: Second, its functional hierarchy aligns with the cortical hierarchy of speech processing.
Revised: Second, its functional hierarchy aligns with the cortical hierarchy of speech processing.

Original: Third, different training regimes reveal a functional specialization akin to the cortex: wav2vec 2.0 learns sound-generic, speech-specific and language-specific representations similar to those of the prefrontal and temporal cortices.
Revised: Third, different training regimes reveal a functional specialization akin to the cortex: wav2vec 2.0 learns sound-generic, speech-specific and language-specific representations similar to those of the prefrontal and temporal cortices.

Original: Fourth, we confirm the similarity of this specialization with the behavior of 386 additional participants.
Revised: Fourth, we confirm the similarity of this specialization with the behavior of 386 additional participants.

Original: These elements, resulting from the largest neuroimaging benchmark to date, show how self-supervised learning can account for a rich organization of speech processing in the brain, and thus delineate a path to identify the laws of language acquisition which shape the human brain.
Revised: These elements, resulting from the largest neuroimaging benchmark to date, show how self-supervised learning can account for a rich organization of speech processing in the brain, and thus delineate a path to identify the laws of language acquisition which shape the human brain.

Original:  Introduction
Revised:  Introduction

Original: The performance of deep neural networks has taken off over the past decade.
Revised: The performance of deep neural networks has taken off over the past decade.

Original: Algorithms trained on object classification, text translation, and speech recognition are starting to reach human-level performance [Xu et al., 2020].
Revised: Algorithms trained on object classification, text translation, and speech recognition are starting to reach human-level performance [Xu et al., 2020].

Original: Furthermore, the representations generated by these algorithms have repeatedly been shown to correlate with those of the brain [Kriegeskorte, 2015, Yamins and DiCarlo, 2016, Kietzmann et al., 2018, Kell and McDermott, 2019, Cichy and Kaiser, 2019, Toneva and
Revised: Furthermore, the representations generated by these algorithms have repeatedly been shown to correlate with those of the brain [Kriegeskorte, 2015, Yamins and DiCarlo, 2016, Kietzmann et al., 2018, Kell and McDermott, 2019, Cichy and Kaiser, 2019, Toneva and

Original: Wehbe, 2019, Millet and King, 2021, Caucheteux and King, 2022], suggesting that these algorithms converge to brain-like computations.
Revised: Wehbe, 2019, Millet and King, 2021, Caucheteux and King, 2022], suggesting that these algorithms converge to brain-like computations.

Original: Such convergence, however, should not obscure the major differences that remain between these deep learning models and the brain.
Revised: Such convergence, however, should not obscure the major differences that remain between these deep learning models and the brain.

Original: In particular, the above comparisons derive from models trained with (1) extraordinarily large amounts of data (40GB for GPT-

Revised: In particular, the above comparisons derive from models trained with (1) extraordinarily large amounts of data (40GB for GPT-

Original: [Radford et al., 2019], the equivalent of multiple lifetimes of reading), (2) supervised labels, which is rarely the case for humans (e.g.
Revised: [Radford et al., 2019], the equivalent of multiple lifetimes of reading), (2) supervised labels, which is rarely the case for humans (e.g.

Original: [Yamins and DiCarlo, 2016]), (3) data in a textual rather than a raw sensory format, and/or (4) considerable memory (e.g., language models typically have parallel access to thousands of context words to process text).
Revised: [Yamins and DiCarlo, 2016]), (3) data in a textual rather than a raw sensory format, and/or (4) considerable memory (e.g., language models typically have parallel access to thousands of context words to process text).

Original: These differences highlight the pressing necessity to identify architectures and learning objectives which, subject to these four constraints, would be sufficient to account for both behavior and brain responses.
Revised: These differences highlight the pressing necessity to identify architectures and learning objectives which, subject to these four constraints, would be sufficient to account for both behavior and brain responses.

Original: Here, we hypothesize that the latest self-supervised architectures trained on raw sensory data constitute promising candidates [Borgholt et al., 2022, Bardes et al., 2021, Baevski et al., 2020].
Revised: Here, we hypothesize that the latest self-supervised architectures trained on raw sensory data constitute promising candidates [Borgholt et al., 2022, Bardes et al., 2021, Baevski et al., 2020].

Original: We focus on wav2vec 2.0 [Baevski et al., 2020], an architecture that stacks convolutional and transformer layers to predict a quantization of the latent representations of speech waveforms.
Revised: We focus on wav2vec 2.0 [Baevski et al., 2020], an architecture that stacks convolutional and transformer layers to predict a quantization of the latent representations of speech waveforms.

Original: We train wav2vec 2.0 on 600 h of effective speech – a quantity roughly comparable to what infants are exposed to during early language acquisition (speech only makes up a small fraction of infants’ daily experience)
Revised: We train wav2vec 2.0 on 600 h of effective speech – a quantity roughly comparable to what infants are exposed to during early language acquisition (speech only makes up a small fraction of infants’ daily experience)

Original: [Dupoux, 2018, Hart and Risley, 1992, Gilkerson et al., 2017].
Revised: [Dupoux, 2018, Hart and Risley, 1992, Gilkerson et al., 2017].

Original: We use standard encoding analyses [Naselaris et al., 2011, Huth et al., 2016, Yamins and DiCarlo, 2016, Kell et al., 2018]
Revised: We use standard encoding analyses [Naselaris et al., 2011, Huth et al., 2016, Yamins and DiCarlo, 2016, Kell et al., 2018]

Original: (Figure 1) to compare this model to the brains of 412 healthy volunteers (
Revised: (Figure 1) to compare this model to the brains of 412 healthy volunteers (

Original: English speakers, 28 French speakers, and 33 Mandarin speakers) recorded with functional magnetic resonance imaging (fMRI) while they passively listened to approximately one hour of audio books in their native language [Nastase et al., 2020, Li et al., 2021] (8.5 hours of distinct audio materials in total).
Revised: English speakers, 28 French speakers, and 33 Mandarin speakers) recorded with functional magnetic resonance imaging (fMRI) while they passively listened to approximately one hour of audio books in their native language [Nastase et al., 2020, Li et al., 2021] (8.5 hours of distinct audio materials in total).

Original: To better understand the similarities between wav2vec 2.0 and the brain, we compare brain activity to each layer of this model, as well as to several variants, namely (1) a random (untrained) wav2vec 2.0 model, (2) a model trained on 600 h of non-speech sounds, (3) a model trained on 600 h of nonnative speech (for example, a model trained on English speech and mapped onto the brain responses to French-speaking participants), (4) a model trained on 600 h of native speech (for example, a model trained on English speech and mapped onto the brain responses to English participants), and (5) a model trained directly on speech-to-text (i.e., a supervised learning scheme) on the native language of the participants.
Revised: To better understand the similarities between wav2vec 2.0 and the brain, we compare brain activity to each layer of this model, as well as to several variants, namely (1) a random (untrained) wav2vec 2.0 model, (2) a model trained on 600 h of non-speech sounds, (3) a model trained on 600 h of nonnative speech (for example, a model trained on English speech and mapped onto the brain responses to French-speaking participants), (4) a model trained on 600 h of native speech (for example, a model trained on English speech and mapped onto the brain responses to English participants), and (5) a model trained directly on speech-to-text (i.e., a supervised learning scheme) on the native language of the participants.

Original: Our results provide four main contributions.
Revised: Our results provide four main contributions.

Original: First, self-supervised learning leads wav2vec 2.0 to learn latent representations of the speech waveform similar to those of the human brain.
Revised: First, self-supervised learning leads wav2vec 2.0 to learn latent representations of the speech waveform similar to those of the human brain.

Original: Second, the functional hierarchy of its transformer layers aligns with the cortical hierarchy of speech in the brain, and reveals the whole-brain organisation of speech processing with an unprecedented clarity.
Revised: Second, the functional hierarchy of its transformer layers aligns with the cortical hierarchy of speech in the brain, and reveals the whole-brain organisation of speech processing with an unprecedented clarity.

Original: Third, the auditory-, speech-, and language-specific representations learned by the model converge to those of the human brain.
Revised: Third, the auditory-, speech-, and language-specific representations learned by the model converge to those of the human brain.

Original: Fourth, behavioral comparisons to 386 supplementary participants’ results on a speech sound discrimination task confirm this common language specialization.
Revised: Fourth, behavioral comparisons to 386 supplementary participants’ results on a speech sound discrimination task confirm this common language specialization.

Original:  Methods 2.1 Models
Revised:  Methods 2.1 Models

Original: We train several variants of wav2vec 2.0 [Baevski et al., 2020] from scratch on different speech datasets using two different learning objectives (a self-supervised and a supervised objective)
Revised: We train several variants of wav2vec 2.0 [Baevski et al., 2020] from scratch on different speech datasets using two different learning objectives (a self-supervised and a supervised objective)

Original: . 2.1.1 Architecture
Revised: . 2.1.1 Architecture

Original: Wav2vec 2.0 consists of three main modules.
Revised: Wav2vec 2.0 consists of three main modules.

Original: First, a feature encoder composed of seven blocks of temporal convolutions (output dimension 512) transforms the speech input S (raw mono waveform at 16 kHz) into a latent representation z (output dimension of 512, frequency 49 Hz, stride of 20 ms between each frame, receptive field of 25 ms).
Revised: First, a feature encoder composed of seven blocks of temporal convolutions (output dimension 512) transforms the speech input S (raw mono waveform at 16 kHz) into a latent representation z (output dimension of 512, frequency 49 Hz, stride of 20 ms between each frame, receptive field of 25 ms).

Original: Second, a quantization module discretizes z into q , a dictionary of discrete and latent representations of sounds.
Revised: Second, a quantization module discretizes z into q , a dictionary of discrete and latent representations of sounds.

Original: Third, z is input to a “context network” consisting of 12 transformer blocks (model dimension 768, inner dimension 3072, and 8 attention heads), which together yield a contextualized embedding c , of the same dimensionality of q .
Revised: Third, z is input to a “context network” consisting of 12 transformer blocks (model dimension 768, inner dimension 3072, and 8 attention heads), which together yield a contextualized embedding c , of the same dimensionality of q .

Original: 2.1.2 Learning objective
Revised: 2.1.2 Learning objective

Original: Self-supervised learning.
Revised: Self-supervised learning.

Original: In this training paradigm, the model optimizes two losses.
Revised: In this training paradigm, the model optimizes two losses.

Original: The first loss is contrastive and requires the model to predict the quantized representation q of some masked input using c , from a finite set of quantized representations drawn from the input sample.
Revised: The first loss is contrastive and requires the model to predict the quantized representation q of some masked input using c , from a finite set of quantized representations drawn from the input sample.

Original: The second loss ensures that the quantized representations are diverse.
Revised: The second loss ensures that the quantized representations are diverse.

Original: See Section A.2 and [Baevski et al., 2020] for details.
Revised: See Section A.2 and [Baevski et al., 2020] for details.

Original: Supervised learning.
Revised: Supervised learning.

Original: In this training paradigm, the quantization module is discarded and a linear layer mapping c to phonemes is added at the end of the pipeline.
Revised: In this training paradigm, the quantization module is discarded and a linear layer mapping c to phonemes is added at the end of the pipeline.

Original: The model is randomly initialized and all layers (including the feature encoder) are trained using a Connectionist Temporal Classification (CTC)
Revised: The model is randomly initialized and all layers (including the feature encoder) are trained using a Connectionist Temporal Classification (CTC)

Original: [Graves, 2012] loss to perform phone recognition.
Revised: [Graves, 2012] loss to perform phone recognition.

Original: For both training paradigms, we extract the activations of each layer from both the feature encoder (outputting z ) and the context network (outputting c ).
Revised: For both training paradigms, we extract the activations of each layer from both the feature encoder (outputting z ) and the context network (outputting c ).

Original: We extract the representations of the convolutional and transformer blocks using an input window of 10 s of raw waveform (stride = 5 s).
Revised: We extract the representations of the convolutional and transformer blocks using an input window of 10 s of raw waveform (stride = 5 s).

Original: 2.1.3 Training
Revised: 2.1.3 Training

Original: Datasets.
Revised: Datasets.

Original: We successively train different wav2vec 2.0 models using each of four datasets: (i) the French and (ii) English CommonVoice corpora [Ardila et al., 2020], (iii) the MAGICDATA
Revised: We successively train different wav2vec 2.0 models using each of four datasets: (i) the French and (ii) English CommonVoice corpora [Ardila et al., 2020], (iii) the MAGICDATA

Original: Mandarin Chinese Read Speech Corpus [Co., 2019], and (iv) a non-speech subset of the Audioset dataset [Gemmeke et al., 2017], which contains recordings of various acoustic scenes.
Revised: Mandarin Chinese Read Speech Corpus [Co., 2019], and (iv) a non-speech subset of the Audioset dataset [Gemmeke et al., 2017], which contains recordings of various acoustic scenes.

Original: Preprocessing.
Revised: Preprocessing.

Original: All the audio datasets were randomly subsampled to have an approximate size of 600 hours, downsampled to 16 kHz and converted to mono with the Sox software 1 .
Revised: All the audio datasets were randomly subsampled to have an approximate size of 600 hours, downsampled to 16 kHz and converted to mono with the Sox software 1 .

Original: We randomly split the datasets into a training (80%), a validation (10%) and a test set (10%).
Revised: We randomly split the datasets into a training (80%), a validation (10%) and a test set (10%).

Original: The audio recordings we use from the Audioset dataset are filtered so that they do not contain speech or any sounds produced by humans, such as laughter or singing.
Revised: The audio recordings we use from the Audioset dataset are filtered so that they do not contain speech or any sounds produced by humans, such as laughter or singing.

Original: For the speech datasets, we also use their corresponding annotations (in the supervised settings).
Revised: For the speech datasets, we also use their corresponding annotations (in the supervised settings).

Original: We phonemize these annotations using eSpeakNG 2 .
Revised: We phonemize these annotations using eSpeakNG 2 .

Original: The number of different phoneme symbols in these annotations is similar for French (32),
Revised: The number of different phoneme symbols in these annotations is similar for French (32),

Original: English (39) and Mandarin Chinese (33).
Revised: English (39) and Mandarin Chinese (33).

Original: Implementation.
Revised: Implementation.

Original: We train all of our models using the fairseq implementation of wav2vec 2.0 3 using default hyperparameters.
Revised: We train all of our models using the fairseq implementation of wav2vec 2.0 3 using default hyperparameters.

Original: We also analyze a model whose parameters were randomly initialised (“untrained” model).
Revised: We also analyze a model whose parameters were randomly initialised (“untrained” model).

Original: We use self-supervised learning to train four models: three on the speech datasets (French, English, and Mandarin) and one on the acoustic scenes dataset.
Revised: We use self-supervised learning to train four models: three on the speech datasets (French, English, and Mandarin) and one on the acoustic scenes dataset.

Original: In each case, the training was performed using the same configuration file (namely, the base configuration provided in the fairseq repository for pretraining wav2vec 2.0 on LibriSpeech [Panayotov et al., 2015]).
Revised: In each case, the training was performed using the same configuration file (namely, the base configuration provided in the fairseq repository for pretraining wav2vec 2.0 on LibriSpeech [Panayotov et al., 2015]).

Original: We train the models for 400k updates and select the ones with the best validation loss.
Revised: We train the models for 400k updates and select the ones with the best validation loss.

Original: We also use the supervised training paradigm to train three models, on the French, English, and
Revised: We also use the supervised training paradigm to train three models, on the French, English, and

Original: Mandarin datasets, respectively.
Revised: Mandarin datasets, respectively.

Original: Each training was performed using the same configuration file, which was identical to the configuration provided in the fairseq repository for fine-tuning wav2vec 2.0 on the 960 hour Voxpopuli corpus [Wang et al., 2021], except that parameters were not frozen ( freeze_finetune_updates = 0 ) and learning was performed on all parameters of the models using the CTC loss ( feature_grad_mult = 0 . 1 ).
Revised: Each training was performed using the same configuration file, which was identical to the configuration provided in the fairseq repository for fine-tuning wav2vec 2.0 on the 960 hour Voxpopuli corpus [Wang et al., 2021], except that parameters were not frozen ( freeze_finetune_updates = 0 ) and learning was performed on all parameters of the models using the CTC loss ( feature_grad_mult = 0 . 1 ).

Original: We train the models for 400k updates and we use the ones with the best word error rate (WER) on the validation set.
Revised: We train the models for 400k updates and we use the ones with the best word error rate (WER) on the validation set.

Original: The French model obtains 13.
Revised: The French model obtains 13.

Original: WER, the English model 28.6 WER, and the Mandarin model 4.6 WER, on their respective test sets.
Revised: WER, the English model 28.6 WER, and the Mandarin model 4.6 WER, on their respective test sets.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: 2.2 Functional MRI
Revised: 2.2 Functional MRI

Original: We analyse a composite set of fMRI recordings aggregated from the Little Prince [Li et al., 2021] and the Narratives public datasets [Nastase et al., 2020].
Revised: We analyse a composite set of fMRI recordings aggregated from the Little Prince [Li et al., 2021] and the Narratives public datasets [Nastase et al., 2020].

Original: Narratives.
Revised: Narratives.

Original: This dataset 4 contains the fMRI recordings of 345 native English-speaking participants listening to English narratives (4.6 hours of unique audio in total).
Revised: This dataset 4 contains the fMRI recordings of 345 native English-speaking participants listening to English narratives (4.6 hours of unique audio in total).

Original: The participants listened to different stories varying from 7 to 98 min (mean = 26 min).
Revised: The participants listened to different stories varying from 7 to 98 min (mean = 26 min).

Original: Following [Nastase et al., 2020], we (1) focus on fifteen representative stories and ignore the narratives that have been modified by scrambling and (2) exclude eight participants because of noisy recordings.
Revised: Following [Nastase et al., 2020], we (1) focus on fifteen representative stories and ignore the narratives that have been modified by scrambling and (2) exclude eight participants because of noisy recordings.

Original: Overall, this selection results in a curated dataset of 303 participants listening to fifteen stories ranging from 3 min to 56 min, for a total of 4 hours of unique audio (36,018 words from a vocabulary of 4,004 unique words).
Revised: Overall, this selection results in a curated dataset of 303 participants listening to fifteen stories ranging from 3 min to 56 min, for a total of 4 hours of unique audio (36,018 words from a vocabulary of 4,004 unique words).

Original: The Little Prince.
Revised: The Little Prince.

Original: This dataset 5 contains fMRI recordings of 48 English native speakers, 33 Mandarin native speakers, and 28 French native speakers listening to The Little Prince in their respective native language.
Revised: This dataset 5 contains fMRI recordings of 48 English native speakers, 33 Mandarin native speakers, and 28 French native speakers listening to The Little Prince in their respective native language.

Original: The experiment itself was divided into nine runs of approximately 10 min of passive listening.
Revised: The experiment itself was divided into nine runs of approximately 10 min of passive listening.

Original: For each language condition, the story was read by a single native speaker.
Revised: For each language condition, the story was read by a single native speaker.

Original: The
Revised: The

Original: English, Mandarin, and French audiobooks last 94, 90 and 97 minutes respectively.
Revised: English, Mandarin, and French audiobooks last 94, 90 and 97 minutes respectively.

Original: Preprocessing.
Revised: Preprocessing.

Original: For Narratives, we did not perform additional preprocessing: we use the public preprocessing of the dataset already projected on the surface space (“fsaverage6”) without spatial smoothing (labelled “afni-nosmooth” in the data repository).
Revised: For Narratives, we did not perform additional preprocessing: we use the public preprocessing of the dataset already projected on the surface space (“fsaverage6”) without spatial smoothing (labelled “afni-nosmooth” in the data repository).

Original: In contrast, the Little Prince dataset is only provided in a volumetric space.
Revised: In contrast, the Little Prince dataset is only provided in a volumetric space.

Original: Consequently, for each language condition separately, we subselected the cortical voxels by computing a brain mask using the average of all participants’ fMRI data realigned onto a common template brain via Freesurfer [Fischl, 2012].
Revised: Consequently, for each language condition separately, we subselected the cortical voxels by computing a brain mask using the average of all participants’ fMRI data realigned onto a common template brain via Freesurfer [Fischl, 2012].

Original: These voxels are then projected onto a brain surface using nilearn’s vol_to_surf function with defaults parameters [Abraham et al., 2014] and a ‘fsaverage6‘ template surface [Fischl, 2012].
Revised: These voxels are then projected onto a brain surface using nilearn’s vol_to_surf function with defaults parameters [Abraham et al., 2014] and a ‘fsaverage6‘ template surface [Fischl, 2012].

Original: For both Narratives and
Revised: For both Narratives and

Original: The Little Prince , fMRI signals are normalized across the time dimension to have a mean of 0 and a variance of 1, for each participant, surface voxel and session independently.
Revised: The Little Prince , fMRI signals are normalized across the time dimension to have a mean of 0 and a variance of 1, for each participant, surface voxel and session independently.

Original: Brain parcellation.
Revised: Brain parcellation.

Original: For the purposes of certain analyses, we group the fMRI voxels into regions of interest using the Destrieux Atlas [Destrieux et al., 2010].
Revised: For the purposes of certain analyses, we group the fMRI voxels into regions of interest using the Destrieux Atlas [Destrieux et al., 2010].

Original: This parcellation results in 75 brain regions in each hemisphere.
Revised: This parcellation results in 75 brain regions in each hemisphere.

Original: For simplicity, we label the regions as follows: A1 and A2 represents
Revised: For simplicity, we label the regions as follows: A1 and A2 represents

Original: Heschl gyrus, which is the anatomical location of the primary and secondary auditory cortices, STG and STS are the superior temporal gyrus and sulcus, and IFG is the inferior frontal gyrus.
Revised: Heschl gyrus, which is the anatomical location of the primary and secondary auditory cortices, STG and STS are the superior temporal gyrus and sulcus, and IFG is the inferior frontal gyrus.

Original: 2.3 Brain score (R)
Revised: 2.3 Brain score (R)

Original: To quantify the similarity between the network’s activations X and the brain recordings Y , we use a standard linear encoding model
Revised: To quantify the similarity between the network’s activations X and the brain recordings Y , we use a standard linear encoding model

Original: [Huth et al., 2016, Yamins and DiCarlo, 2016].
Revised: [Huth et al., 2016, Yamins and DiCarlo, 2016].

Original: For each subject, we split the data into train and test sets using a five-fold cross-validation setting.
Revised: For each subject, we split the data into train and test sets using a five-fold cross-validation setting.

Original: For each train split, a linear mapping W is fitted to predict the brain response Y train given X train .
Revised: For each train split, a linear mapping W is fitted to predict the brain response Y train given X train .

Original: W combines a temporal alignment function with fixed weight, and a trained penalized linear regression.
Revised: W combines a temporal alignment function with fixed weight, and a trained penalized linear regression.

Original: Temporal alignment.
Revised: Temporal alignment.

Original: The sampling frequency of the model’s activations (between 49 and 200 Hz) differs from the sampling frequency of fMRI BOLD signals (0.5 Hz).
Revised: The sampling frequency of the model’s activations (between 49 and 200 Hz) differs from the sampling frequency of fMRI BOLD signals (0.5 Hz).

Original: Furthermore, the BOLD signals have delayed responses spanning over several seconds.
Revised: Furthermore, the BOLD signals have delayed responses spanning over several seconds.

Original: Thus, we first convolve the model activations with a standard hemodynamic response function (HRF) using nistats [Abraham et al., 2014] compute_regressor function with the ‘glover’ model and default parameters.
Revised: Thus, we first convolve the model activations with a standard hemodynamic response function (HRF) using nistats [Abraham et al., 2014] compute_regressor function with the ‘glover’ model and default parameters.

Original: This results in the convolved activations X ′ train with the same sampling frequency as the fMRI Y train (see A.3).
Revised: This results in the convolved activations X ′ train with the same sampling frequency as the fMRI Y train (see A.3).

Original: Penalised linear regression.
Revised: Penalised linear regression.

Original: Once temporally aligned, we fit an ℓ 2 -penalised linear regression that predicts the brain signals Y train given the activations X train .
Revised: Once temporally aligned, we fit an ℓ 2 -penalised linear regression that predicts the brain signals Y train given the activations X train .

Original: We use the RidgeCV function from scikit-learn [Pedregosa et al., 2011], with the penalization hyperparameter λ varying betweenand 10 8 (20 values scaled logarithmically) chosen independently for each dimension with a nested cross-validation over the training set (see A.4).
Revised: We use the RidgeCV function from scikit-learn [Pedregosa et al., 2011], with the penalization hyperparameter λ varying betweenand 10 8 (20 values scaled logarithmically) chosen independently for each dimension with a nested cross-validation over the training set (see A.4).

Original: Evaluation.
Revised: Evaluation.

Original: We evaluate the linear mapping W on the held out sets by measuring Pearson’s correlation between predicted and actual brain responses: R = corr (cid:0) Y test , W · X test (cid:1) .
Revised: We evaluate the linear mapping W on the held out sets by measuring Pearson’s correlation between predicted and actual brain responses: R = corr (cid:0) Y test , W · X test (cid:1) .

Original: Finally, we average the correlation scores across test splits to obtain the final “brain score”.
Revised: Finally, we average the correlation scores across test splits to obtain the final “brain score”.

Original: To report the average layer k ∗ with the highest brain score for each voxel (Figure 3), while being robust to regressionto-the-mean biases, we first find the best layer k s for each participant s and each voxel independently and then compute a circular mean across the N = 412 participants and the K = 19 layers:
Revised: To report the average layer k ∗ with the highest brain score for each voxel (Figure 3), while being robust to regressionto-the-mean biases, we first find the best layer k s for each participant s and each voxel independently and then compute a circular mean across the N = 412 participants and the K = 19 layers:

Original: k ∗ = angle (cid:16) 1 N (cid:80)
Revised: k ∗ = angle (cid:16) 1 N (cid:80)

Original: Ns =1 exp (cid:16) 

Revised: Ns =1 exp (cid:16)

Original: iπk s K +1 (cid:17)(cid:17)
Revised: iπk s K +1 (cid:17)(cid:17)

Original: Statistics.
Revised: Statistics.

Original: We assess the reliability of brain scores with second-level analyses across participants thanks to a Wilcoxon signed-rank test across participants.
Revised: We assess the reliability of brain scores with second-level analyses across participants thanks to a Wilcoxon signed-rank test across participants.

Original: Thus, the resulting p-values are not affected by fMRI auto-correlation within participants.
Revised: Thus, the resulting p-values are not affected by fMRI auto-correlation within participants.

Original: We perform statistical correction for multiple comparisons with Benjamini–Hochberg False Discovery Rate (FDR) across voxels [Benjamini, 2010].
Revised: We perform statistical correction for multiple comparisons with Benjamini–Hochberg False Discovery Rate (FDR) across voxels [Benjamini, 2010].

Original: 2.4 Behavioral experiment
Revised: 2.4 Behavioral experiment

Original: To compare the phonetic representations of our models to those of humans, we compare the forcedchoice discrimination judgements of online participants 6 to an analogous method applied to wav2vec 2.0 [Schatz, 2016].
Revised: To compare the phonetic representations of our models to those of humans, we compare the forcedchoice discrimination judgements of online participants 6 to an analogous method applied to wav2vec 2.0 [Schatz, 2016].

Original: Specifically, for each triplet of sound “ABX”, participants judged whether the stimulus X was more similar to A or B. Analogously, we computed the Euclidean distance in the most discriminative layer of wav2vec 2.0 (here transformer layer 5) to determine whether X was closer to A or B. Additional data, analyses and model-human comparison can be found in [Millet and Dunbar, 2022].
Revised: Specifically, for each triplet of sound “ABX”, participants judged whether the stimulus X was more similar to A or B. Analogously, we computed the Euclidean distance in the most discriminative layer of wav2vec 2.0 (here transformer layer 5) to determine whether X was closer to A or B. Additional data, analyses and model-human comparison can be found in [Millet and Dunbar, 2022].

Original: We focus on the French and English stimuli, which represent ≈ 6,000 ABX triplets (testing 508 English and 524 French phone pairs), with 386 participants in total (193 from each language group).
Revised: We focus on the French and English stimuli, which represent ≈ 6,000 ABX triplets (testing 508 English and 524 French phone pairs), with 386 participants in total (193 from each language group).

Original: In Figure 4-A, we report the ABX accuracy of English- and French-speaking participants in both their native and non-native language (either English or French).
Revised: In Figure 4-A, we report the ABX accuracy of English- and French-speaking participants in both their native and non-native language (either English or French).

Original: We first average results per phone pair, and then average over phone pairs to obtain the ABX discrimination accuracy.
Revised: We first average results per phone pair, and then average over phone pairs to obtain the ABX discrimination accuracy.

Original: Similarly, in
Revised: Similarly, in

Original: Figure 4-B, we compute the ABX accuracy of our wav2vec 2.0 models on the same evaluation sets as the participants, using the parameters described in [Millet and Dunbar, 2022].
Revised: Figure 4-B, we compute the ABX accuracy of our wav2vec 2.0 models on the same evaluation sets as the participants, using the parameters described in [Millet and Dunbar, 2022].

Original: English and
Revised: English and

Original: French models are evaluated on the same (‘native’) or different (‘non-native’) language stimuli as their training.
Revised: French models are evaluated on the same (‘native’) or different (‘non-native’) language stimuli as their training.

Original: The random and non-speech models are evaluated on both French and English speech stimuli.
Revised: The random and non-speech models are evaluated on both French and English speech stimuli.

Original:  Results
Revised:  Results

Original: Wav2vec 2.0 maps onto brain responses to speech.
Revised: Wav2vec 2.0 maps onto brain responses to speech.

Original: We estimate whether the activations of wav2vec 2.0 models linearly map onto the human brain activity of 412 individuals listening to audio books in the fMRI scanner.
Revised: We estimate whether the activations of wav2vec 2.0 models linearly map onto the human brain activity of 412 individuals listening to audio books in the fMRI scanner.

Original: For this, we first independently train three models with 600 h of
Revised: For this, we first independently train three models with 600 h of

Original: French, English, or Mandarin, respectively, and compute the brain scores ( R ) with the corresponding participants.
Revised: French, English, or Mandarin, respectively, and compute the brain scores ( R ) with the corresponding participants.

Original: Specifically, we (1) convolve the activations ( X ) of the model with a hemodynamic response function (HRF), (2) train a ℓ 2 -penalized linear regression on a training split to map them to brain activity Y , and (3) compute the Pearson correlation coefficient between (i) the true fMRI activity and (ii) the predicted activations on a test split.
Revised: Specifically, we (1) convolve the activations ( X ) of the model with a hemodynamic response function (HRF), (2) train a ℓ 2 -penalized linear regression on a training split to map them to brain activity Y , and (3) compute the Pearson correlation coefficient between (i) the true fMRI activity and (ii) the predicted activations on a test split.

Original: The models’ activations significantly predict brain activity in nearly all cortical areas, reaching the highest R scores in the primary and secondary auditory cortices (Figure 2-A B).
Revised: The models’ activations significantly predict brain activity in nearly all cortical areas, reaching the highest R scores in the primary and secondary auditory cortices (Figure 2-A B).

Original: These scores are significantly higher than those obtained with a randomly initialised model ( p <
Revised: These scores are significantly higher than those obtained with a randomly initialised model ( p <

Original: 10 − 50 on average across voxels), and this comparison is robust across language groups (all p < 10 − 5 ).
Revised: 10 − 50 on average across voxels), and this comparison is robust across language groups (all p < 10 − 5 ).

Original: Comparison of self-supervised to supervised models.
Revised: Comparison of self-supervised to supervised models.

Original: Does self-supervision reach representations that are as brain-like as those obtained with supervised learning?
Revised: Does self-supervision reach representations that are as brain-like as those obtained with supervised learning?

Original: To address this issue, we trained wav2vec 2.0 with an alternative, supervised objective, namely, predicting phonetic annotations from the same 600 hours of effective speech sounds.
Revised: To address this issue, we trained wav2vec 2.0 with an alternative, supervised objective, namely, predicting phonetic annotations from the same 600 hours of effective speech sounds.

Original: We then implemented the R score analyses described above.
Revised: We then implemented the R score analyses described above.

Original: The results show that self-supervised learning in fact leads to modestly but significantly better R scores than supervised learning (Figure 2-C): ∆ R = 0 .
Revised: The results show that self-supervised learning in fact leads to modestly but significantly better R scores than supervised learning (Figure 2-C): ∆ R = 0 .

Original: 002 , p < 10 6 .
Revised: 002 , p < 10 6 .

Original: The hierarchy of wav2vec 2.0 maps onto the hierarchy of the cortex.
Revised: The hierarchy of wav2vec 2.0 maps onto the hierarchy of the cortex.

Original: To compare the speech hierarchy in the brain with the functional hierarchy learned by wav2vec 2.0, we evaluate the R score of each layer of the model (Figure 3).
Revised: To compare the speech hierarchy in the brain with the functional hierarchy learned by wav2vec 2.0, we evaluate the R score of each layer of the model (Figure 3).

Original: First, we observe that convolutional layers are less predictive than transformer layers.
Revised: First, we observe that convolutional layers are less predictive than transformer layers.

Original: Second, within the transformers, the hierarchy of representations aligns with the expected cortical hierarchy [Hickok and Poeppel, 2007]: while low-level areas (A1, A2) are best predicted by the first transformer layers, higher level areas (IFG, STS) are best predicted by deeper layers.
Revised: Second, within the transformers, the hierarchy of representations aligns with the expected cortical hierarchy [Hickok and Poeppel, 2007]: while low-level areas (A1, A2) are best predicted by the first transformer layers, higher level areas (IFG, STS) are best predicted by deeper layers.

Original: Remarkably, this hierarchy extends to supplementary motor and motor areas in both hemispheres (Figure 3-A).
Revised: Remarkably, this hierarchy extends to supplementary motor and motor areas in both hemispheres (Figure 3-A).

Original: Language specificity in phone discrimination tasks.
Revised: Language specificity in phone discrimination tasks.

Original: The acoustic features underlying speech (fricatives, vowels, and so on) may also characterize non-speech sounds (the sound of tree leaves in the wind, of a stone falling, and so on).
Revised: The acoustic features underlying speech (fricatives, vowels, and so on) may also characterize non-speech sounds (the sound of tree leaves in the wind, of a stone falling, and so on).

Original: Does the model show commonalities merely with general auditory processing in the brain, or does it capture speech-specific processing?
Revised: Does the model show commonalities merely with general auditory processing in the brain, or does it capture speech-specific processing?

Original: If so, does it show commonalities with brain representations that are specific to the native language of the participants, or merely to general speech processing?
Revised: If so, does it show commonalities with brain representations that are specific to the native language of the participants, or merely to general speech processing?

Original: We first evaluate the specialization of humans’ perception to their native language using an ABX behavioral task (Section 2.4).
Revised: We first evaluate the specialization of humans’ perception to their native language using an ABX behavioral task (Section 2.4).

Original: Specifically, we compare
Revised: Specifically, we compare

Original: French and English participants on their ability to distinguish native and non-native phones.
Revised: French and English participants on their ability to distinguish native and non-native phones.

Original: As expected [Bohn, 2017, Kuhl et al., 2005], participants were better at discriminating native sounds than non-native ones (across phone pairs: p < 10 − 18 , Figure 4-A).
Revised: As expected [Bohn, 2017, Kuhl et al., 2005], participants were better at discriminating native sounds than non-native ones (across phone pairs: p < 10 − 18 , Figure 4-A).

Original: Second, applying the same test to our self-supervised French and English models shows that, like humans, models best discriminate sounds from their ‘native’ language (i.e., the French model better distinguishes French stimuli than
Revised: Second, applying the same test to our self-supervised French and English models shows that, like humans, models best discriminate sounds from their ‘native’ language (i.e., the French model better distinguishes French stimuli than

Original: English ones, across phone pairs, and vice versa: p < 0 . 05 ).
Revised: English ones, across phone pairs, and vice versa: p < 0 . 05 ).

Original: Interestingly, the ABX accuracy of the model is significantly higher than participants’.
Revised: Interestingly, the ABX accuracy of the model is significantly higher than participants’.

Original: This quantitative difference may be partially explained by the fact that participants – and online participants in particular – undergo fluctuating attention, and adopt strategies which can negatively impact performance [Humphreys, 1939].
Revised: This quantitative difference may be partially explained by the fact that participants – and online participants in particular – undergo fluctuating attention, and adopt strategies which can negatively impact performance [Humphreys, 1939].

Original: Finally, as expected, the random and acoustic models obtain the worst ABX accuracy.
Revised: Finally, as expected, the random and acoustic models obtain the worst ABX accuracy.

Original: Overall, These results confirm that 600 h of self-supervised learning on effective speech suffices for wav2vec 2.0 to learn language-specific representations (Figure 4-B).
Revised: Overall, These results confirm that 600 h of self-supervised learning on effective speech suffices for wav2vec 2.0 to learn language-specific representations (Figure 4-B).

Original: Wav2vec 2.0 and the brain learn language specific representations.
Revised: Wav2vec 2.0 and the brain learn language specific representations.

Original: Next, we compare the brain scores of random, non-speech, non-native and native models (Figure 4-C D).
Revised: Next, we compare the brain scores of random, non-speech, non-native and native models (Figure 4-C D).

Original: First, our results show that the non-speech model attains higher R scores than the random model (on average across voxels, ∆ R = 0 .
Revised: First, our results show that the non-speech model attains higher R scores than the random model (on average across voxels, ∆ R = 0 .

Original: 006 , p = 10 − 31 ) confirming the importance of learning to generate brainlike representations.
Revised: 006 , p = 10 − 31 ) confirming the importance of learning to generate brainlike representations.

Original: Second, non-native models attain higher R scores than the non-speech model ( ∆ R = 0 .
Revised: Second, non-native models attain higher R scores than the non-speech model ( ∆ R = 0 .

Original: 002 , p = 10 − 9 ), confirming that wav2vec 2.0 learns speech-specific representations of sounds when trained on speech.
Revised: 002 , p = 10 − 9 ), confirming that wav2vec 2.0 learns speech-specific representations of sounds when trained on speech.

Original: Finally, the native model attains higher R scores than non-native models ( ∆ R = 0 .
Revised: Finally, the native model attains higher R scores than non-native models ( ∆ R = 0 .

Original: 002 , p = 10 − 15 ).
Revised: 002 , p = 10 − 15 ).

Original:  Discussion
Revised:  Discussion

Original: Human infants acquire language with little to no supervision:
Revised: Human infants acquire language with little to no supervision:

Original: A few hundred hours of speech suffices for their young brain to learn to discretize phonemes, segment morphemes, and assemble words in the language(s) of their social group [Dupoux, 2018, Gilkerson et al., 2017].
Revised: A few hundred hours of speech suffices for their young brain to learn to discretize phonemes, segment morphemes, and assemble words in the language(s) of their social group [Dupoux, 2018, Gilkerson et al., 2017].

Original: However, the learning principle that allows this unique feat remains, to date, unknown.
Revised: However, the learning principle that allows this unique feat remains, to date, unknown.

Original: Here, we test whether self-supervised learning applied to a limited amount of speech effectively accounts for the organization of speech processing in the human brain as measured with fMRI.
Revised: Here, we test whether self-supervised learning applied to a limited amount of speech effectively accounts for the organization of speech processing in the human brain as measured with fMRI.

Original: For this, we train several variants of wav2vec 2.0 [Baevski et al., 2020] with three curated datasets of
Revised: For this, we train several variants of wav2vec 2.0 [Baevski et al., 2020] with three curated datasets of

Original: French, English, and Mandarin, and compare their activations to those of a large group of French,
Revised: French, English, and Mandarin, and compare their activations to those of a large group of French,

Original: English, and Mandarin speakers recorded with fMRI while passively listening to audio stories.
Revised: English, and Mandarin speakers recorded with fMRI while passively listening to audio stories.

Original: Our results show that this self-supervised model learns (i) representations that linearly map onto a remarkably distributed set of cortical regions (Figure 2), (ii) a computational hierarchy that aligns with the cortical hierarchy (Figure 3), and (iii) features specific to the language of the participants (Figure 4).
Revised: Our results show that this self-supervised model learns (i) representations that linearly map onto a remarkably distributed set of cortical regions (Figure 2), (ii) a computational hierarchy that aligns with the cortical hierarchy (Figure 3), and (iii) features specific to the language of the participants (Figure 4).

Original: Towards a biologically-plausible learning principle.
Revised: Towards a biologically-plausible learning principle.

Original: These results extend recent findings on the similarities between the brain and a variety of deep learning models trained with biologicallyimplausible objectives and data.
Revised: These results extend recent findings on the similarities between the brain and a variety of deep learning models trained with biologicallyimplausible objectives and data.

Original: First, fMRI
Revised: First, fMRI

Original: [Kell et al., 2018, Millet and King, 2021, Thompson et al., 2021], electroencephalography [Huang et al., 2018], and multi- or single-unit responses to sounds [Koumura et al., 2019, Begus et al., 2022] have been shown to be linearly predicted by the activations of deep convolutional networks trained on supervised auditory tasks.
Revised: [Kell et al., 2018, Millet and King, 2021, Thompson et al., 2021], electroencephalography [Huang et al., 2018], and multi- or single-unit responses to sounds [Koumura et al., 2019, Begus et al., 2022] have been shown to be linearly predicted by the activations of deep convolutional networks trained on supervised auditory tasks.

Original: For example, [Millet and King, 2021] showed that a supervised speech-to-text model better accounted for brain responses to speech in 102 individuals when it was trained on speech recognition rather than auditory scene classification.
Revised: For example, [Millet and King, 2021] showed that a supervised speech-to-text model better accounted for brain responses to speech in 102 individuals when it was trained on speech recognition rather than auditory scene classification.

Original: Similarly, [Kell et al., 2018] showed that eight participants listening to brief speech and non-speech sounds demonstrated fMRI responses in the temporal lobe that aligned with those of a deep convolutional neural network trained on a binary auditory classification task.
Revised: Similarly, [Kell et al., 2018] showed that eight participants listening to brief speech and non-speech sounds demonstrated fMRI responses in the temporal lobe that aligned with those of a deep convolutional neural network trained on a binary auditory classification task.

Original: Our results, based on up to 50 times more fMRI recordings of the entire cortex show that such representational similarities hold with a self-supervised objective [Lerner et al., 2011, Berezutskaya et al., 2017, Caucheteux et al., 2021c,b].
Revised: Our results, based on up to 50 times more fMRI recordings of the entire cortex show that such representational similarities hold with a self-supervised objective [Lerner et al., 2011, Berezutskaya et al., 2017, Caucheteux et al., 2021c,b].

Original: Second, a growing series of MEG [Toneva and Wehbe, 2019,
Revised: Second, a growing series of MEG [Toneva and Wehbe, 2019,

Original: Caucheteux and King, 2022], fMRI [Mitchell et al., 2008, Qian et al., 2016, Pereira et al., 2018,
Revised: Caucheteux and King, 2022], fMRI [Mitchell et al., 2008, Qian et al., 2016, Pereira et al., 2018,

Original: Schwartz et al., 2019, Antonello et al., 2021, Jain and Huth, 2018] and electro-physiology studies [Schrimpf et al., 2021, Goldstein et al., 2022] showed that text-based language models trained on very large corpora generate brain-like representations too.
Revised: Schwartz et al., 2019, Antonello et al., 2021, Jain and Huth, 2018] and electro-physiology studies [Schrimpf et al., 2021, Goldstein et al., 2022] showed that text-based language models trained on very large corpora generate brain-like representations too.

Original: While these results suggest elements of convergence between language models and the brain [Caucheteux and King, 2022], they also remain biologically implausible: not only are these algorithms pre-equipped with abstract linguistic units such as characters and words, but they are trained on corpora that no one would ever be able to read in their lifetime.
Revised: While these results suggest elements of convergence between language models and the brain [Caucheteux and King, 2022], they also remain biologically implausible: not only are these algorithms pre-equipped with abstract linguistic units such as characters and words, but they are trained on corpora that no one would ever be able to read in their lifetime.

Original: In contrast, wav2vec 2.0 is here trained with a reasonable amount of raw speech waveforms
Revised: In contrast, wav2vec 2.0 is here trained with a reasonable amount of raw speech waveforms

Original: [Hart and Risley, 1992, Gilkerson et al., 2017, Dupoux, 2018].
Revised: [Hart and Risley, 1992, Gilkerson et al., 2017, Dupoux, 2018].

Original: The functional similarity between wav2vec 2.0 and the brain thus opens the way to clarify how humans learn to process speech.
Revised: The functional similarity between wav2vec 2.0 and the brain thus opens the way to clarify how humans learn to process speech.

Original: The emergence of a brain-like hierarchy of speech processing.
Revised: The emergence of a brain-like hierarchy of speech processing.

Original: The present study reveals the hierarchical organization of speech processing with remarkable clarity.
Revised: The present study reveals the hierarchical organization of speech processing with remarkable clarity.

Original: First, the functional hierarchy learnt by wav2vec 2.0 is aligned with the anatomy: e.g. the superior temporal sulcus and the temporal pole are known to project to the ventral and dorsal part of the inferofrontal gyrus, respectively [Petkov et al., 2015].
Revised: First, the functional hierarchy learnt by wav2vec 2.0 is aligned with the anatomy: e.g. the superior temporal sulcus and the temporal pole are known to project to the ventral and dorsal part of the inferofrontal gyrus, respectively [Petkov et al., 2015].

Original: Second, the identification of functional gradients within the prefrontal cortex, and down to the motor areas typically associated with larynx and mouth control [Dichter et al., 2018] reinforces the relevance of motor processes to speech perception [Kellis et al., 2010,
Revised: Second, the identification of functional gradients within the prefrontal cortex, and down to the motor areas typically associated with larynx and mouth control [Dichter et al., 2018] reinforces the relevance of motor processes to speech perception [Kellis et al., 2010,

Original: Mugler et al., 2014, Shamma et al., 2021].
Revised: Mugler et al., 2014, Shamma et al., 2021].

Original: Finally, the existence of multiple levels of representations around the inferofrontal cortex is consistent with the idea that Broca’s area may be responsible for merging linguistic units [Chomsky, 2000, Friederici, 1999, Hagoort, 2005, Poeppel et al., 2012].
Revised: Finally, the existence of multiple levels of representations around the inferofrontal cortex is consistent with the idea that Broca’s area may be responsible for merging linguistic units [Chomsky, 2000, Friederici, 1999, Hagoort, 2005, Poeppel et al., 2012].

Original: It should be noted, however, that our results aggregate a large cohort of individuals which could mask a more modular organization at the individual level.
Revised: It should be noted, however, that our results aggregate a large cohort of individuals which could mask a more modular organization at the individual level.

Original: Interpreting the neural representations of speech.
Revised: Interpreting the neural representations of speech.

Original: Interpreting neural representations is a notoriously difficult challenge to both AI and neuroscience.
Revised: Interpreting neural representations is a notoriously difficult challenge to both AI and neuroscience.

Original: Here, we first investigate language specificity and show that the neural representations specific to the native models are primarily represented in the superior temporal sulcus and middle temporal gyrus (Figure 4D): areas known to represent phonetic features [Mesgarani et al., 2014].
Revised: Here, we first investigate language specificity and show that the neural representations specific to the native models are primarily represented in the superior temporal sulcus and middle temporal gyrus (Figure 4D): areas known to represent phonetic features [Mesgarani et al., 2014].

Original: However, these effect are relatively modest (Figure 4): the random model and the non-speech model reach, in STS and STG, 67% and 87% of the brain scores obtained by the “native” model, respectively.
Revised: However, these effect are relatively modest (Figure 4): the random model and the non-speech model reach, in STS and STG, 67% and 87% of the brain scores obtained by the “native” model, respectively.

Original: While this high baseline initially surprised us, this phenomenon could be explained by the fact that the auditory cortex is continuously bombarded by – and should thus be tuned to – non-speech input.
Revised: While this high baseline initially surprised us, this phenomenon could be explained by the fact that the auditory cortex is continuously bombarded by – and should thus be tuned to – non-speech input.

Original: Second, our probing analyses show that the models trained with self-supervised learning learn relevant acoustic and linguistic representations (Supplementary Figure S1).
Revised: Second, our probing analyses show that the models trained with self-supervised learning learn relevant acoustic and linguistic representations (Supplementary Figure S1).

Original: This result, consistent with Vaidya et al.
Revised: This result, consistent with Vaidya et al.

Original: [2022] and Stephenson et al.
Revised: [2022] and Stephenson et al.

Original: [2019], suggests that the difference of brain scores observed between the random, non-native and native models (Figure 4) may be partly driven by the corresponding spectro-temporal, phonetic, word and sentence-level representations, respectively.
Revised: [2019], suggests that the difference of brain scores observed between the random, non-native and native models (Figure 4) may be partly driven by the corresponding spectro-temporal, phonetic, word and sentence-level representations, respectively.

Original: These elements of interpretation remain, however, scarce, and a systematic interpretation of the representations shared between wav2vec 2.0 and the brain remains necessary.
Revised: These elements of interpretation remain, however, scarce, and a systematic interpretation of the representations shared between wav2vec 2.0 and the brain remains necessary.

Original: Scope of the study.
Revised: Scope of the study.

Original: It is important to stress that the scope of the present study could be broadened in several ways.
Revised: It is important to stress that the scope of the present study could be broadened in several ways.

Original: First, our study focuses on adult speakers, whose cultural and educational background is not representative of the population [Henrich et al., 2010].
Revised: First, our study focuses on adult speakers, whose cultural and educational background is not representative of the population [Henrich et al., 2010].

Original: Second, we focus on the passive listening of three languages.
Revised: Second, we focus on the passive listening of three languages.

Original: Third, we focus on one self-supervised learning architecture [Baevski et al., 2020], and its functional alignment with fMRI, whose temporal resolution is notoriously limited.
Revised: Third, we focus on one self-supervised learning architecture [Baevski et al., 2020], and its functional alignment with fMRI, whose temporal resolution is notoriously limited.

Original: Generalizing the present approach to more languages [Malik-Moraleda et al., 2022], a larger spectrum of children and adult participants recorded with a variety of electrophysiological and neuroimaging devices will thus be essential to confirm, precise, and/or mitigate the present findings.
Revised: Generalizing the present approach to more languages [Malik-Moraleda et al., 2022], a larger spectrum of children and adult participants recorded with a variety of electrophysiological and neuroimaging devices will thus be essential to confirm, precise, and/or mitigate the present findings.

Original: The remaining gap between brain and speech models.
Revised: The remaining gap between brain and speech models.

Original: Several major gaps can be evidenced between wav2vec 2.0 and the brain.
Revised: Several major gaps can be evidenced between wav2vec 2.0 and the brain.

Original: First, the transformer layers are not temporally constrained: each layer can access all elements within the contextual window.
Revised: First, the transformer layers are not temporally constrained: each layer can access all elements within the contextual window.

Original: This differs from the necessarily recurrent nature of processing in the brain.
Revised: This differs from the necessarily recurrent nature of processing in the brain.

Original: Second, wav2vec 2.0 behaves differently to humans in specific tasks.
Revised: Second, wav2vec 2.0 behaves differently to humans in specific tasks.

Original: In particular, it is overly-sensitive to band-pass filtering, non-robustly exploit fine temporal structures [Weerts et al., 2021] and fails to display the expected categorical responses [Millet et al., 2021].
Revised: In particular, it is overly-sensitive to band-pass filtering, non-robustly exploit fine temporal structures [Weerts et al., 2021] and fails to display the expected categorical responses [Millet et al., 2021].

Original: Third, recent studies show that wav2vec 2.0 encodes significantly less semantic information than text-based models [Pasad et al., 2021, Vaidya et al., 2022].
Revised: Third, recent studies show that wav2vec 2.0 encodes significantly less semantic information than text-based models [Pasad et al., 2021, Vaidya et al., 2022].

Original: While our analyses suggest that learning allows wav2vec 2.0 to capture some lexical features in its deep layers (Figure
Revised: While our analyses suggest that learning allows wav2vec 2.0 to capture some lexical features in its deep layers (Figure

Original: S1, Table S4), it remains unclear whether these layers also capture complex syntactic structures, such as recursive syntactic trees [Lakretz et al., 2021, Caucheteux et al., 2021a].
Revised: S1, Table S4), it remains unclear whether these layers also capture complex syntactic structures, such as recursive syntactic trees [Lakretz et al., 2021, Caucheteux et al., 2021a].

Original: We speculate that these limitations may be due to the time scales of wav2vec 2.0 which, unlike humans, learns very short-term representations of speech.
Revised: We speculate that these limitations may be due to the time scales of wav2vec 2.0 which, unlike humans, learns very short-term representations of speech.

Original: In any case, these differences likely explain why the brain scores of wav2vec 2.0 remain substantially lower than our noise-ceiling (19% on average, and up to 74% in Heschl’s gyrus and sulcus, Table S1, Figure S2).
Revised: In any case, these differences likely explain why the brain scores of wav2vec 2.0 remain substantially lower than our noise-ceiling (19% on average, and up to 74% in Heschl’s gyrus and sulcus, Table S1, Figure S2).

Original: Overall, the complexity of the human brain is often thought to be incompatible with a simple theory: “Even if there were enough data available about the contents of each brain area, there probably would not be a ready set of equations to describe them, their relationships, and the ways they change over time”
Revised: Overall, the complexity of the human brain is often thought to be incompatible with a simple theory: “Even if there were enough data available about the contents of each brain area, there probably would not be a ready set of equations to describe them, their relationships, and the ways they change over time”

Original: [Gallant, 2013].
Revised: [Gallant, 2013].

Original: By showing how the equations of self-supervised learning give rise to brain-like processes, this work contributes to challenge this view.
Revised: By showing how the equations of self-supervised learning give rise to brain-like processes, this work contributes to challenge this view.

Original: Acknowledgments
Revised: Acknowledgments

Original: This project was funded, in part, by the Bettencourt-Schueller Foundation, the Philippe Foundation, and FrontCog grant ANR-17-EURE-0017 to JRK for his work at PSL.
Revised: This project was funded, in part, by the Bettencourt-Schueller Foundation, the Philippe Foundation, and FrontCog grant ANR-17-EURE-0017 to JRK for his work at PSL.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] See Section(b)
Revised: [Yes] See Section(b)

Original: Did you describe the limitations of your work?
Revised: Did you describe the limitations of your work?

Original: [Yes] See Section(c)
Revised: [Yes] See Section(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] We have, and since we use results of human experiments that were previously made, and that respect these review guidelines, we conform to them.
Revised: [Yes] We have, and since we use results of human experiments that were previously made, and that respect these review guidelines, we conform to them.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: No theoretical results (b) Did you include complete proofs of all theoretical results?
Revised: No theoretical results (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: No theoretical results 3.
Revised: No theoretical results 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] The code will be provided upon request.
Revised: [Yes] The code will be provided upon request.

Original: Otherwise for the rest, see Section 2.2 and 2.1.
Revised: Otherwise for the rest, see Section 2.2 and 2.1.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 2.1.3.
Revised: [Yes] See Section 2.1.3.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] This is the case for all our figures, see Figure 2, 3 and (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] This is the case for all our figures, see Figure 2, 3 and (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [No] The models we train are not new: the original papers already give that type of information.
Revised: [No] The models we train are not new: the original papers already give that type of information.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 2.2, 2.1.3, and 2.4.
Revised: [Yes] See Section 2.2, 2.1.3, and 2.4.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [No] They are all under CC0 for the English and French training audio data, the Perceptimatic Benchmark, The Little Prince dataset and the Narratives dataset.
Revised: [No] They are all under CC0 for the English and French training audio data, the Perceptimatic Benchmark, The Little Prince dataset and the Narratives dataset.

Original: Magic data is under CC BY-NC-ND 4.0, Audioset is under CC BY 4.0 for the audio recordings, and CC BY-SA 4.0 for the ontology, Narratives’ preprocessing code is under GNU GENERAL PUBLIC LICENSE and fairseq code is under MIT License.
Revised: Magic data is under CC BY-NC-ND 4.0, Audioset is under CC BY 4.0 for the audio recordings, and CC BY-SA 4.0 for the ontology, Narratives’ preprocessing code is under GNU GENERAL PUBLIC LICENSE and fairseq code is under MIT License.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] The code will be given upon request (d) Did you discuss whether
Revised: [No] The code will be given upon request (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] See Section 2.2 and 2.4.
Revised: [Yes] See Section 2.2 and 2.4.

Original: More details on the way they gave their consent can be seen in the original papers/websites that provided the experimental results.
Revised: More details on the way they gave their consent can be seen in the original papers/websites that provided the experimental results.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: The data were anonymized before we used them.
Revised: The data were anonymized before we used them.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: We did not conduct the experiments ourselves (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: We did not conduct the experiments ourselves (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: We did not conduct the experiments ourselves (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: We did not conduct the experiments ourselves (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: We did not conduct the experiments ourselves
Revised: We did not conduct the experiments ourselves

Original: A Appendix A.1 Self-supervised loss formula

Original: Wav2vec 2.0, when trained in a self-supervised way, uses a loss ( L ) which is the weighted combination of two losses: one diversity loss ( L d ), which pushes the quantization module to contain representations that are as diverse as possible, and one Contrastive Predictive Coding loss ( L m ), which pushes the model to choose, from the context network output c , the right quantized representation ( q ) of some masked input, among other possible representations.

Original: L m has the following formula, for some masked time step t :

Original: [Equation]

Original: with sim( a , b ) = a T b / ∥ a ∥∥ b ∥ , κ the temperature, which is constant during training, Q t the set of

Original: K + 1 quantized candidate the model has to choose from, including the right one, i.e. q t .

Original: L d is included to encourage the equal use of the V possible entries of each of the G codebooks of the quantization module.

Original: The goal is to maximize the entropy of the averaged softmax distribution over the codebook entries for each codebook ¯ p g , across a set of utterances:

Original: [Equation]

Original: A.2 Supervised loss formula

Original: When trained in a supervised way, wav2vec 2.0 is trained to optimise a Connectionist Temporal

Original: Classification loss parameterized over θ :

Original: [Equation]

Original: [Equation]

Original: where m θ ( U ) ∈ R d τ × d v are the probabilistic predictions of the model at each τ time sample given the input raw waveform

Original: U ∈ R d τ × d u , V ∈ R d t × d v are the true transcriptions of U , and a U,V is the set of all possible alignments between U and V .

Original: A.3 Preprocessing of the model’s activations

Original: The activations of the network X ∈ R d ˆ t × d x are first normalized to be between [0 , 1] for each listening session.

Original: Then, we use nistats [Abraham et al., 2014] compute_regressor function with the ‘glover’ model to temporally convolve ( h ∈ R d ˆ t ) and temporally down-sample ( using g : R d ˆ t → R d t ) each artificial neuron j :

Original: [Equation]

Original: A.4 Penalized linear model - Ridge regression

Original: For each split s , we fit an ℓ 2 -penalized linear model V ∈ R d x × d z trained to predict the transformed

Original: BOLD time series from the model activations for each dimension independently.

Original: The formula of the optimization is the following:

Original: [Equation]

Original: A.5 Probing the linguistic features encoded in wav2vec2 activations

Original: Interpreting the representations of deep learning models is notoriously difficult.

Original: To address this issue, [Pasad et al., 2021] explored the encoding of local acoustic features, phone identity, word identity and word meaning across layers.

Original: Similarly, [Millet et al., 2021] compared representations to human behavioural data to assess whether they better captured listeners’ perception of higher-level phonemic properties or of lower-level subphonemic properties of speech stimuli.

Original: Finally, [Vaidya et al., 2022] recent study explores filter banks, spectrograms, phonemes and words across layers.

Original: Here, we complement these analyses by showing that self-supervised learning allows wav2vec 2.

Original: to learn represents, along its hierarchy the representations of MEL spectrograms, phonetic categories and word embeddings (Figure S1).

Original: For this, we perform a ridge regression on the Timit dataset 7 to predict five auditory and linguistic features from the activation functions of each layer and model of the present paper.

Original: We study the following features:

Original: • the MEL spectrogram of the audio, computed using librosa (d=128) • the phonemes (categorical features).

Original: We use the transcripts and alignments provided in Timit.

Original: • the word embedding and part-of-speech of the words.

Original: The time alignments for words are provided by Timit.

Original: We use spaCy to compute the word embedding (medium model, d=300), and their part-of-speech (categorical feature, d=19).

Original: • the sentence embedding of each sample, provided by Laser.

Original: We use a subset of 1,680 samples from Timit, each sample being an audio recording

Original: of a

Original: short sen tence

Original: (<

Original: seconds

Original: ) from

Original: 24 speakers

Original: The model’s activations were mean-pooled to the sampling rate of each feature.

Original: The results show that the layers of wav2vec 2.0 partially follow the hierarchy predicted from neurolinguistics [Hickok and Poeppel, 2007]

Original: (Table S4): the first layers of the transformer best account for the spectro-temporal information, whereas deeper layers best account for the phonetic, word-level and sentence level information.

Original: While all of these features emerge with training (Figure S1), only the highest-level features (phone, word and sentence-level) appear to be specific to speech and to the language with which wav2vec 2.0 was trained (Figure S1).

Original: Interestingly, the word and sentence-level features are encoded deeper in the supervised network (best layer=18 in Table S4) compared to the unsupervised network (best layer=14), which suggests that self-supervised learning generates a reservoir representations in its middle layers, reservoir which may partly overlap with the labels used in supervised learning.

Original: Together with our ABX tests, and layer-wise tuning of each voxel (Figure 3), these elements suggest that the representations of speech shaped by our experience are learnt and instantiated in the superior temporal gyrus and sulcus.

Original: These elements, consistent with previous electrophysiological studies [Mesgarani et al., 2014], thus provide a coherent spectrum of evidence for the location of acquired speech representations in the brain.

Original: A.6 Noise ceiling analysis

Original: The noise in fMRI recordings is inevitable.

Original: To estimate the maximum explainable signal given this level of noise, we follow previous studies and employ a shared-response model, or "noise ceiling"

Original: [Huth et al., 2016, Caucheteux and King, 2022, Caucheteux et al., 2022].

Original: Precisely, we predict the brain signals of one subject given the brain activity of the other subjects, in response to the same audio recording.

Original: In practice, we apply the same evaluation as Equation (2.3), for one subject s and one voxel v , but we use the average brain signals of other subjects’ brains Y ( s ) = 1 |S| (cid:80) s ′ ̸

Original: = s Y ( s ′ ) instead of the activations X .

Original: As a result, the "noise ceiling" of one subject ( s ) and one voxel ( v ) is computed as follows:

Original: [Equation]

Original: 6) where W is an ℓ 2 -penalized linear regression fitted on separate train data, using a cross validation setting with five test folds.

Original: We compute such noise ceiling on 290 subjects of the Narrative dataset listening to the same stories (Figure S2).

Original: We report the noise ceiling across voxels in Figure S2, and, in Table S1, the brain scores of the networks studied in the main paper normalised by the noise ceiling.

Original: Precisely, for each voxel, we divide the average brain scores by the noise ceiling for this particular voxel.

Original: While low on average, the unsupervised wav2vec2 model reaches 74% of the noise ceiling in Heschel and more than 20% in STS, STS and IFG.

Original: [Table]

Original: Random wav2vec

Original: [Table]

Original: [Table]

Original: Non-Speech

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: Non-Native

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: Native, Supervised

Original: [Table]

Original: Below, we report the brain scores of our models, normalised by such noise ceiling.

Original: Precisely, we compute the brain scores for each subject and voxels

Original: [Table]

Original: [Table]

Original: [Table]

Original: Random wav2vec

Original: [Table]

Original: [Table]

Original: Acoustic wav2vec 12.

Original: [Table]

Original: [Table]

Original: Mandarin wav2vec 9.

Original: French wav2vec 8.

Original: Dutch wav2vec 18.

Original: English wav2vec 8.

Original: English wav2vec2 (supervised) 8.

Original: [Table]

Original: [Table]

Original: , averaged across the 10 cross-validation folds.