Original: Toward a realistic model of speech processing in the brain with self-supervised learning
Revised: Toward a realistic model of speech processing in the brain with self-supervised learning

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Several deep neural networks have recently been shown to generate activationssimilar to those of the brain in response to the same input.
Revised: Several deep neural networks have recently been shown to generate activationssimilar to those of the brain in response to the same input.

Original: These algorithms,however, remain largely implausible: they require (1) extraordinarily large amountsof data, (2) unobtainable supervised labels, (3) textual rather than raw sensory input,and / or (4) implausibly large memory (e.g. thousands of contextual words).
Revised: These algorithms, however, remain largely implausible: they require (1) extraordinarily large amounts ofdata, (2) unobtainable supervised labels, (3) textual rather than raw sensory input,and / or (4) implausibly large memory (e.g. thousands of contextual words).

Original: Theseelements highlight the need to identify algorithms that, under these limitations,would suffice to account for both behavioral and brain responses.
Revised: Theseelements highlight the need to identify algorithms that, under these limitations,would suffice to account for both behavioral and brain responses.

Original: Focusing on theissue of speech processing, we here hypothesize that self-supervised algorithmstrained on the raw waveform constitute a promising candidate.
Revised: Focusing on theissue of speech processing, we here hypothesize that self-supervised algorithmstrained on the raw waveform constitute a promising candidate.

Original: Specifically, wecompare a recent self-supervised architecture, Wav2Vec 2.0, to the brain activity of412 English, French, and Mandarin individuals recorded with functional Magnetic Resonance Imaging (fMRI), while they listened to ≈ 1 h of audio books.
Revised: Specifically, wecompare a recent self-supervised architecture, Wav2Vec 2.0, to the brain activityof 412 English, French, and Mandarin individuals recorded with functional Magnetic Resonance Imaging (fMRI), while they listened to ≈ 1 h of audio books.

Original: Our resultsare four-fold.
Revised: Ourresults are four-fold.

Original: First, we show that this algorithm learns brain-like representationswith as little as 600 hours of unlabelled speech – a quantity comparable to whatinfants can be exposed to during language acquisition.
Revised: First, we show that this algorithm learns brain-like representations with as little as 600 hours of unlabelled speech – a quantity comparable towhat infants can be exposed to during language acquisition.

Original: Second, its functionalhierarchy aligns with the cortical hierarchy of speech processing.
Revised: Second, its functionalhierarchy aligns with the cortical hierarchy of speech processing.

Original: Third, differenttraining regimes reveal a functional specialization akin to the cortex: Wav2Vec 2.0learns sound-generic, speech-specific and language-specific representations similarto those of the prefrontal and temporal cortices.
Revised: Third, differenttraining regimes reveal a functional specialization akin to the cortex: Wav2Vec2.0 learns sound-generic, speech-specific and language-specific representationssimilar to those of the prefrontal and temporal cortices.

Original: Fourth, we confirm the similarityof this specialization with the behavior of 386 additional participants.
Revised: Fourth, we confirm thesimilarity of this specialization with the behavior of 386 additional participants.

Original: Theseelements, resulting from the largest neuroimaging benchmark to date, show howself-supervised learning can account for a rich organization of speech processingin the brain, and thus delineate a path to identify the laws of language acquisitionwhich shape the human brain.
Revised: These elements, resulting from the largest neuroimaging benchmark to date, showhow self-supervised learning can account for a rich organization of speech processing in the brain, and thus delineate a path to identify the laws of languageacquisition which shape the human brain.

Original:  Introduction
Revised:  Introduction

Original: The performance of deep neural networks has taken off over the past decade.
Revised: The performance of deep neural networks has taken off over the past decade.

Original: Algorithms trainedon object classification, text translation and speech recognition are starting to reach human-levelperformance [Xu et al., 2021].
Revised: Algorithms trainedon object classification, text translation, and speech recognition are starting to reach human-levelperformance Xu et al.

Original: Furthermore, the representations of these algorithms have repeatedlyshown to correlate with those of the brain [Kriegeskorte, 2015, Yamins and DiCarlo, 2016, Kietzmannet al., 2018, Kell and McDermott, 2019, Cichy and Kaiser, 2019, Toneva and Wehbe, 2019, Milletand King, 2021, Caucheteux and King, 2022], suggesting that these algorithms converge to brain-likecomputations.
Revised: Furthermore, the representations generated by these algorithms haverepeatedly been shown to correlate with those of the brain Kriegeskorte [2015], Yamins and DiCarlo[2016], Kietzmann et al. [2018], Kell and McDermott [2019], Cichy and Kaiser [2019], Tonevaand Wehbe [2019], Millet and King [2021], Caucheteux and King [2022], suggesting that thesealgorithms converge to brain-like computations.

Original: Such convergence, however, should not obscure the major differences that remain between these deeplearning models and the brain.
Revised: Such convergence, however, should not obscure the major differences that remain between thesedeep learning models and the brain.

Original: In particular, the above comparisons derive from models trained with(1) extraordinarily large amounts of data (40GB for [-GPT-
-]
Revised: In particular, the above comparisons derive from models trainedwith (1) extraordinarily large amounts of data (40GB for [-GPT-
-]GPT-2 Radford et al.

Original: [Radford et al., 2019], the equivalent of multiple lifetimes of reading)
Revised: [2019], the equivalentof multiple lifetimes of reading), (2) supervised labels, which is rarely the case for humans (e.g.

Original: (2) supervised labels which are rare in human experience (e.g. [Yamins and DiCarlo, 2016]), (3) data in a textual rather than a raw sensory format, and/or (4) considerablememory (e.g., language models typically have parallel access to thousands of context words toprocess text).
Revised: Yamins and DiCarlo [2016]), (3) data in a textual rather than a raw sensory format, and/or (4)considerable memory (e.g., language models typically have parallel access to thousands of contextwords to process text).

Original: These differences highlight the pressing necessity to identify architectures and learningobjectives which, subject to these four constraints, would be sufficient to account for both behaviorand brain responses.
Revised: These differences highlight the pressing necessity to identify architecturesand learning objectives which, subject to these four constraints, would be sufficient to account forboth behavior and brain responses.

Original: Here, we hypothesize that the latest self-supervised architectures trained on raw sensory data constitutepromising candidates [Borgholt et al., 2022, Bardes et al., 2021, Baevski et al., 2020].
Revised: Here, we hypothesize that the latest self-supervised architectures trained on raw sensory data constitute promising candidates Borgholt et al. [2022], Bardes et al. [2021], Baevski et al.

Original: We focus on 
Revised: Wefocus on Wav2Vec 2.0 Baevski et al.

Original: Wav2Vec 2.0 [Baevski et al., 2020], an architecture that stacks convolutional and transformer layersto predict a quantization of the latent representations of speech waveforms.
Revised: [2020], an architecture that stacks convolutional and transformer layers to predict a quantization of the latent representations of speech waveforms.

Original: We train Wav2Vec 2.0 on600 h of  speech—a quantity roughly comparable to what infants are exposed to during early languageacquisition [Dupoux, 2018].
Revised: We train Wav2Vec 2.0 on 600 h of effective speech—a quantity roughly comparable to what infants are exposed to during early language acquisition (speech only makes up a small fraction of infants’ dailyexperience) Dupoux [2018].

Original: We use standard encoding analyses [Naselaris et al., 2011, Huth et al., 2016, Yamins and DiCarlo,2016, Kell et al., 2018]
Revised: We use standard encoding analyses Naselaris et al. [2011], Huth et al. [2016], Yamins and DiCarlo[2016], Kell et al.

Original:  (see Figure 1) to compare this model to the brains of 412 healthy volunteers(351 English speakers, 28 French speakers, and 33 Mandarin speakers) recorded with functionalmagnetic resonance imaging (fMRI) while they passively listened to approximately one hour ofaudio books in their native language [Nastase et al., 2020, Li et al., 2021] (8.5 hours of distinct audiomaterials in total).
Revised: [2018] (see Figure 1) to compare this model to the brains of 412 healthy volunteers(351 English speakers, 28 French speakers, and 33 Mandarin speakers) recorded with functionalmagnetic resonance imaging (fMRI) while they passively listened to approximately one hour ofaudio books in their native language Nastase et al. [2020], Li et al. [2021] (8.5 hours of distinctaudio materials in total).

Original: To better understand the similarities between Wav2Vec 2.0 and the brain, we compare brain activity toeach layer of this model, as well as to several variants, namely (1) a random (untrained) Wav2Vec 2.0model, (2) a model trained on 600 h of non-speech sounds, (3) a model trained on 600 h of non-nativespeech (for example, a model trained on English speech and mapped onto the brain responses to French-speaking participants), (4) a model trained on 600 h of native speech (for example, a modeltrained on English speech and mapped onto the brain responses to English participants), and (5) amodel trained directly on speech-to-text (i.e., a supervised learning scheme) on the native languageof the participants.
Revised: To better understand the similarities between Wav2Vec 2.0 and the brain, we compare brain activityto each layer of this model, as well as to several variants, namely (1) a random (untrained) Wav2Vec2.0 model, (2) a model trained on 600 h of non-speech sounds, (3) a model trained on 600 h of nonnative speech (for example, a model trained on English speech and mapped onto the brain responsesto French-speaking participants), (4) a model trained on 600 h of native speech (for example, a modeltrained on English speech and mapped onto the brain responses to English participants), and (5) amodel trained directly on speech-to-text (i.e., a supervised learning scheme) on the native languageof the participants.

Original: Our results provide four main contributions.
Revised: Our results provide four main contributions.

Original: First, self-supervised learning leads Wav2Vec 2.0 tolearn latent representations of the speech waveform similar to those of the human brain.
Revised: First, self-supervised learning leads Wav2Vec 2.0 tolearn latent representations of the speech waveform similar to those of the human brain.

Original: Second, thefunctional hierarchy of its transformer layers aligns with the cortical hierarchy of speech in the brain,and reveals the whole-brain organisation of speech processing with an unprecedented clarity.
Revised: Second,the functional hierarchy of its transformer layers aligns with the cortical hierarchy of speech in thebrain, and reveals the whole-brain organisation of speech processing with an unprecedented clarity.

Original: Third,the auditory-, speech-, and language-specific representations learned by the model converge to thoseof the human brain.
Revised: Third, the auditory-, speech-, and language-specific representations learned by the model convergeto those of the human brain.

Original: Fourth, behavioral comparisons to 386 supplementary participants’ results on aspeech sound discrimination task confirm this common language specialization.
Revised: Fourth, behavioral comparisons to 386 supplementary participants’results on a speech sound discrimination task confirm this common language specialization.

Original:  Methods2.1 Models
Revised:  Methods2.1 Models

Original: We train several variants of Wav2Vec 2.0 [Baevski et al., 2020] from scratch on different datasetsusing two different learning objectives (a self-supervised and a supervised objective).
Revised: We train several variants of Wav2Vec 2.0 Baevski et al. [2020] from scratch on different datasetsusing two different learning objectives (a self-supervised and a supervised objective).

Original: 2.1.1 Architecture
Revised: 2.1.1 Architecture

Original: Wav2Vec 2.0 consists of three main modules.
Revised: Wav2Vec 2.0 consists of three main modules.

Original: First, a feature encoder composed of seven blocks oftemporal convolutions (output dimension 512) transforms the speech input S (raw mono waveformat 16 kHz) into a latent representation z (output dimension of 512, frequency 49 Hz, stride of 20 msbetween each frame, receptive field of 25 ms).
Revised: First, a feature encoder composed of seven blocks oftemporal convolutions (output dimension 512) transforms the speech input S (raw mono waveformat 16 kHz) into a latent representation z (output dimension of 512, frequency 49 Hz, stride of 20 msbetween each frame, receptive field of 25 ms).

Original: Second, a quantization module discretizes z into q , adictionary of discrete and latent representations of sounds.
Revised: Second, a quantization module discretizes z into q , adictionary of discrete and latent representations of sounds.

Original: Third, z is input to a “context network”consisting of 12 transformer blocks (model dimension 768, inner dimension 3072, and 8 attentionheads), which together yield a contextualized embedding c , of the same dimensionality of q .
Revised: Third, z is input to a “context network”consisting of 12 transformer blocks (model dimension 768, inner dimension 3072, and 8 attentionheads), which together yield a contextualized embedding c , of the same dimensionality of q .

Original: 2.1.2 Learning objective
Revised: 2.1.2 Learning objective

Original: Self-supervised learning.
Revised: Self-supervised learning.

Original: In this training paradigm, the model optimizes two losses.
Revised: In this training paradigm, the model optimizes two losses.

Original: The firstlossis contrastive and requires the model to predict the quantized representation q of some masked inputusing c , from a finite drawn input sample second loss
Revised: The first loss

Original: Supervised learning.
Revised: Supervised learning.

Original: In this training paradigm, the quantization module is discarded and a linearlayer mapping c to phonemes is added at the end of the pipeline.
Revised: In this training paradigm, the quantization module is discarded and a linearlayer mapping c to phonemes is added at the end of the pipeline.

Original: The model is randomly initializedand all layers (including the feature encoder) are trained using a Connectionist Temporal Classification(CTC) [Graves, 2012] loss to perform phone recognition.
Revised: The model is randomly initializedand all layers (including the feature encoder) are trained using a Connectionist Temporal Classification (CTC) Graves [2012] loss to perform phone recognition.

Original: For both training paradigms we extract the activations of each layer from both the feature encoder(outputting z ) and the context network (outputting c ).
Revised: For both training paradigms, we extract the activations of each layer from both the feature encoder(outputting z ) and the context network (outputting c ).

Original: Thus, we study both the convolutional andtransformer blocks.
Revised: Thus, we study both the convolutional andtransformer blocks.

Original: We extract these internal representations using input windows consisting of 10 sof raw waveform, with a stride of 5 s. 2.1.3 Training
Revised: We extract these internal representations using input windows consisting of 10 sof raw waveform, with a stride of 5 s.2.1.3 Training

Original: Datasets.
Revised: Datasets.

Original: We successively train different Wav2Vec 2.0 models using each of four datasets: (i) the
Revised: We successively train different Wav2Vec 2.0 models using each of four datasets: (i) the

Original: French and (ii) English CommonVoice corpora [Ardila et al., 2020], (iii) the MAGICDATA Mandarin
Revised: French and (ii) English CommonVoice corpora Ardila et al. [2020], (iii) the MAGICDATA Mandarin

Original: Chinese Read Speech Corpus [Co., 2019] and (iv) a non-speech subset of the Audioset dataset[Gemmeke et al., 2017], which contains recordings of various acoustic scenes.
Revised: Chinese Read Speech Corpus Co. [2019], and (iv) a non-speech subset of the Audioset dataset Gemmeke et al. [2017], which contains recordings of various acoustic scenes.

Original: Preprocessing.
Revised: Preprocessing.

Original: All the audio datasets were randomly subsampled to have an approximate size of600 hours, downsampled to 16 kHz and converted to mono with the Sox software 1 .
Revised: All the audio datasets were randomly subsampled to have an approximate sizeof 600 hours, downsampled to 16 kHz and converted to mono with the Sox software 1 .

Original: We randomly splitthe datasets into a training (80%), a validation (10%) and a test set (10%).
Revised: We randomly split the datasets into a training (80%), a validation (10%) and a test set (10%).

Original: The audio recordings weuse from the Audioset dataset are filtered so that they do not contain speech or any sounds producedby humans, such as laughter or singing.
Revised: The audiorecordings we use from the Audioset dataset are filtered so that they do not contain speech or anysounds produced by humans, such as laughter or singing.

Original: For the speech datasets, we also use their correspondingannotations (in the supervised settings).
Revised: For the speech datasets, we also usetheir corresponding annotations (in the supervised settings).

Original: We phonemize these annotations using eSpeakNG 2 .
Revised: We phonemize these annotations using eSpeakNG (https://github.com/espeak-ng/espeak-ng).

Original: Note that the number of different phoneme symbols in these annotations is similar for French (32), English(39), and Mandarin Chinese (33).
Revised: Note that the number of different phonemesymbols in these annotations is similar for French (32), English (39), and Mandarin Chinese (33).

Original: Implementation.
Revised: Implementation.

Original: We train all of our models using the fairseq implementation of Wav2Vec 2.0 3using default hyperparameters.
Revised: We train all of our models using the fairseq implementation of Wav2Vec2.0 (https://github.com/pytorch/fairseq/tree/main/examples/wav2vec) using default hyperparameters.

Original: We also analyze a model whose parameters were randomly initialised(“untrained” model).
Revised: We also analyze a model whose parameters were randomly initialised (“untrained” model).

Original: We use self-supervised learning to train four models: three on the speech datasets (French, English,and Mandarin) and one on the acoustic scenes dataset.
Revised: We use self-supervised learning to train four models: three on the speech datasets (French, English,and Mandarin) and one on the acoustic scenes dataset.

Original: In each case, the training was performed usingthe same configuration file (namely, the base configuration provided in the fairseq repository forpretraining Wav2Vec 2.0 on LibriSpeech [Panayotov et al., 2015]).
Revised: In each case, the training was performedusing the same configuration file (namely, the base configuration provided in the fairseq repositoryfor pretraining Wav2Vec 2.0 on LibriSpeech Panayotov et al.

Original: We train the models for 400kupdates and select the ones with the best validation loss.
Revised: We train the models for 400kupdates and select the ones with the best validation loss.

Original: We use the supervised training paradigm to train three models, on the French, English, and Mandarin datasets, respectively.
Revised: We use the supervised training paradigm to train three models, on the French, English, and Mandarin datasets, respectively.

Original: Each training was performed using the same configuration file, whichwas identical to the configuration provided in the fairseq repository for fine-tuning Wav2Vec
Revised: Each training was performed using the same configuration file, whichwas identical to the configuration provided in the fairseq repository for fine-tuning Wav2Vec

Original: 2.0on the 960 hour Voxpopuli corpus [Wang et al., 2021], except that parameters were not frozen( freeze_finetune_updates = 0 ) and learning was performed on all parameters of the modelsusing the CTC loss ( feature_grad_mult = 0 . 1 ).
Revised: 2.0on the 960 hour Voxpopuli corpus Wang et al. [2021], except that parameters were not frozen( freeze_finetune_updates = 0 ) and learning was performed on all parameters of the modelsusing the CTC loss ( feature_grad_mult = 0 . 1 ).

Original: We train the models for 400k updates and we usethe ones with the best word error rate (WER) on the validation set.
Revised: We train the models for 400k updates and we usethe ones with the best word error rate (WER) on the validation set.

Original: The French model obtains 13.

Revised: The French model obtains 13.

Original: WER, the English model 28.6 WER, and the Mandarin model 4.6 WER, on their respective test sets.
Revised: WER, the English model 28.6 WER, and the Mandarin model 4.6 WER, on their respective test sets.

Original: 2.2 Functional MRI
Revised: 2.2 Functional MRI

Original: We analyse a composite set of fMRI recordings aggregated from the Little Prince [Li et al., 2021]and the Narratives public datasets [Nastase et al., 2020].
Revised: We analyse a composite set of fMRI recordings aggregated from the Little Prince Li et al. [2021]and the Narratives public datasets Nastase et al.

Original: Narratives.
Revised: Narratives.

Original: This is a public dataset consisting of the fMRI recordings of 345 native Englishspeaking participants listening to English narratives (4.6 hours of unique audio in total). 
Revised: This is a public dataset consisting of the fMRI recordings of 345 native Englishspeaking participants listening to English narratives (4.6 hours of unique audio in total). Following

Original: Followingthe original manuscript and repository [Nastase et al., 2020], we (1) focus on fifteen representative stories and ignore the narratives that have been modified by scrambling and (2) exclude eightparticipants because of noisy recordings.
Revised: Nastase et al. [2020], we (1) focus on fifteen representative stories and ignore the narratives that havebeen modified by scrambling and (2) exclude eight participants because of noisy recordings.

Original: Overall, this selection results in a curated dataset of 303participants listening to fifteen stories ranging from 3 min to 56 min, for a total of 4 hours of uniqueaudio, 36018 words, and 4004 unique words.
Revised: Overall, this selection results in a curated dataset of 303 participants listening to fifteen stories ranging from3 min to 56 min, for a total of 4 hours of unique audio, 36018 words, and 4004 unique words.

Original: The Little Prince.
Revised: The Little Prince.

Original: This dataset  containsthe fMRI data acquired for 48 English native [-speakers, 
-] Mandarin native speakers, and 28 French native speakers, each listening to The Little Prince in theirrespective native language.
Revised: This dataset 2 contains fMRI data acquired while 48 English native [-speakers, 
-]speakers,33 Mandarin native speakers, and 28 French native speakers, were listening to The Little Prince intheir respective native language.

Original: The experiment itself was divided into nine runs of approximately 10 minof passive listening.
Revised: The experiment itself was divided into nine runs of approximately10 min of passive listening.

Original: For each language condition, the story was read by a single native speaker.
Revised: For each language condition, the story was read by a single nativespeaker.

Original: The English, Mandarin, and French audiobooks last 94, 90 and 97 minutes respectively.
Revised: The English, Mandarin, and French audiobooks last 94, 90 and 97 minutes respectively.

Original: Preprocessing.
Revised: Preprocessing.

Original: For Narratives, we did not perform additional preprocessing: we use the publicpreprocessing of the dataset already projected on the surface space (“fsaverage6”) without spatialsmoothing (labelled “afni-nosmooth” in the data repository).
Revised: For Narratives, we did not perform additional preprocessing: we use the publicpreprocessing of the dataset already projected on the surface space (“fsaverage6”) without spatialsmoothing (labelled “afni-nosmooth” in the data repository).

Original: In contrast, the Little Prince datasetis only provided in a volumetric space.
Revised: In contrast, the Little Prince datasetis only provided in a volumetric space.

Original: Consequently, for each language condition separately, wesubselected the cortical voxels by computing a brain mask using the average of all participants’fMRI data realigned onto a common template brain via Freesurfer [Fischl, 2012].
Revised: Consequently, for each language condition separately, wesubselected the cortical voxels by computing a brain mask using the average of all participants’fMRI data realigned onto a common template brain via Freesurfer Fischl [2012].

Original: These voxels arethen projected onto a brain surface using nilearn’s vol_to_surf function with defaults parameters[Abraham et al., 2014] and a ‘fsaverage6‘ template surface [Fischl, 2012].
Revised: These voxels arethen projected onto a brain surface using nilearn’s vol_to_surf function with defaults parameters Abraham et al. [2014] and a ‘fsaverage6‘ template surface Fischl [2012].

Original: For both Narratives and The Little Prince , fMRI signals are normalized across the time dimension tohave a mean of 0 and a variance of 1, for each participant, surface voxel and session independently.
Revised: For both Narratives and The Little Prince , fMRI signals are normalized across the time dimension tohave a mean of 0 and a variance of 1, for each participant, surface voxel and session independently.

Original: Brain parcellation.
Revised: Brain parcellation.

Original: For the purposes of certain analyses, we group the fMRI voxels into regionsof interest using the Destrieux Atlas [Destrieux et al., 2010].
Revised: For the purposes of certain analyses, we group the fMRI voxels into regionsof interest using the Destrieux Atlas Destrieux et al.

Original: This parcellation results in 75 brain regions in each hemisphere.
Revised: This parcellation results in 75 brainregions in each hemisphere.

Original: For simplicity, we label the regions as follows: A1 and A2 represents
Revised: For simplicity, we label the regions as follows: A1 and A2 represents

Original: Heschl gyrus, which is the anatomical location of the primary and secondary auditory cortices, STGand STS are the superior temporal gyrus and sulcus, and IFG is the inferior frontal gyrus.
Revised: Heschl gyrus, which is the anatomical location of the primary and secondary auditory cortices, STGand STS are the superior temporal gyrus and sulcus, and IFG is the inferior frontal gyrus.

Original: 2.3 Neural Predictivity score (R)
Revised: 2.3 Neural Predictivity score (R)

Original: To quantify the similarity between the network’s activations X and the brain recordings Y , we use astandard linear encoding model
Revised: To quantify the similarity between the network’s activations X and the brain recordings Y , we usea standard linear encoding model

Original: [Huth et al., 2016, Yamins and DiCarlo, 2016].
Revised: Huth et al. [2016], Yamins and DiCarlo [2016].

Original: For each subject, wesplit the data into train and test sets using a five-fold cross-validation setting.
Revised: For each subject,we split the data into train and test sets using a five-fold cross-validation setting.

Original: For each train split, alinear mapping W is fitted to predict the brain response Y train given X train .
Revised: For each trainsplit, a linear mapping W is fitted to predict the brain response Y train given X train .

Original: W combines a temporalalignment function with fixed weight, and a trained penalized linear regression.
Revised: W combines atemporal alignment function with fixed weight, and a trained penalized linear regression.

Original: Temporal alignment.
Revised: Temporal alignment.

Original: The sampling frequency of the model’s activations (between 49 and 200 Hz)differs from the sampling frequency of fMRI BOLD signals (0.5 Hz).
Revised: The sampling frequency of the model’s activations (between 49 and 200 Hz)differs from the sampling frequency of fMRI BOLD signals (0.5 Hz).

Original: Furthermore, the BOLDsignals have delayed responses spanning over several seconds.
Revised: Furthermore, the BOLDsignals have delayed responses spanning over several seconds.

Original: Thus, we first convolve the modelactivations with a standard hemodynamic response function (HRF) using nistats [Abraham et al. ,2014] compute_regressor function with the ‘glover’ model and default parameters.
Revised: Thus, we first convolve the modelactivations with a standard hemodynamic response function (HRF) using nistats Abraham et al.[2014] compute_regressor function with the ‘glover’ model and default parameters.

Original: This results inthe convolved activations X ′ train with the same sampling frequency as the fMRI Y train (see A.3).
Revised: This resultsin the convolved activations X ′ train with the same sampling frequency as the fMRI Y train (see A.3).

Original: Penalised linear regression.
Revised: Penalised linear regression.

Original: Once temporally aligned, we fit an ℓ 2 -penalised linear regressionthat predicts the brain signals Y train given the activations X train .
Revised: Once temporally aligned, we fit an ℓ 2 -penalised linear regressionthat predicts the brain signals Y train given the activations X train .

Original: We use the RidgeCV function fromscikit-learn [Pedregosa et al., 2011], with the penalization hyperparameter λ varying between 10 and 10 8 (20 values scaled logarithmically) chosen independently for each dimension with a nestedcross-validation over the training set (see A.4).
Revised: We use the RidgeCV function fromscikit-learn Pedregosa et al. [2011], with the penalization hyperparameter λ varying between 10and 10 8 (20 values scaled logarithmically) chosen independently for each dimension with a nestedcross-validation over the training set (see A.4).

Original: Evaluation.
Revised: Evaluation.

Original: We evaluate the linear mapping W on the held out sets by measuring Pearson’scorrelation between predicted and actual brain responses:
Revised: We evaluate the linear mapping W on the held out sets by measuring Pearson’s correlation between predicted and actual brain responses:

Original: [Equation]
Revised: [Equation]

Original: Finally, we average the correlation scores across test splits to obtain the final Neural Predictivityscore ( R ).
Revised: Finally, we average the correlation scores across test splits to obtain the final Neural Predictivityscore ( R ).

Original: 2.4 Behavioral experiment
Revised: 2.4 Behavioral experiment

Original: To study the representation specific to each language, we compare the way humans and modelsare able to discriminate native and non-native speech sounds.
Revised: To study the representation specific to each language, we compare the way humans and modelsare able to discriminate native and non-native speech sounds.

Original: We use “ABX” tasks, which arediscrimination tests during which participants have to listen to three speech sounds: A (e.g., /pop/), B(e.g., /pap/) and X (e.g., /pop/)
Revised: We use “ABX” tasks, which arediscrimination tests during which participants have to listen to three speech sounds: A (e.g., /pop/), B (e.g., /pap/) and X (e.g., /pop/)

Original: [Schatz, 2016].
Revised: Schatz [2016].

Original: X is of the same category as A or B, and the participanthas to decide which of A or B is the closest to X. Once the results are aggregated across participantsof one given language group, the discrimination accuracy indicates whether they can differentiate thecategories A and B belong to (in our example, whether they can differentiate the phone pair /o/-/a/).
Revised: X is of the same category as A or B, and theparticipant has to decide which of A or B is the closest to X. Once the results are aggregated acrossparticipants of one given language group, the discrimination accuracy indicates whether they candifferentiate the categories A and B belong to (in our example, whether they can differentiate thephone pair /o/-/a/).

Original: For this, we use the Perceptimatic human benchmark 4 that contains the ABX results of French and
Revised: For this, we use the Perceptimatic human benchmark 3 that contains the ABX results of French and

Original: English-speaking participants on speech stimuli from various languages.
Revised: English-speaking participants on speech stimuli from various languages.

Original: We focus on the French and English stimuli, which represents ≈ 6,000 ABX triplets (testing 508 English and 524 French phonepairs), with 386 participants in total (193 from each language group).
Revised: We focus on the Frenchand English stimuli, which represents ≈ 6,000 ABX triplets (testing 508 English and 524 Frenchphone pairs), with 386 participants in total (193 from each language group).

Original: In Figure 4-A, we report the ABX accuracy of English- and French-speaking participants in both theirnative and non-native language (either English or French).
Revised: In Figure 4-A, we report the ABX accuracy of English- and French-speaking participants in boththeir native and non-native language (either English or French).

Original: We first average results per phone pair,and then average over phone pairs to obtain the ABX discrimination accuracy.
Revised: We first average results per phonepair, and then average over phone pairs to obtain the ABX discrimination accuracy.

Original: Similarly, in Figure4-B, we compute the ABX accuracy of our Wav2Vec 2.0 models on the same evaluation sets as theparticipants, using the parameters described in [Millet and Dunbar, 2022].
Revised: Similarly, in Figure 4-B, we compute the ABX accuracy of our Wav2Vec 2.0 models on the same evaluation setsas the participants, using the parameters described in Millet and Dunbar [2022].

Original: English and French modelsare evaluated on the same (‘native’) or different (‘non-native’) language stimuli as their training.
Revised: English and Frenchmodels are evaluated on the same (‘native’) or different (‘non-native’) language stimuli as theirtraining.

Original: Theuntrained and non-speech models are evaluated on both French and English speech stimuli.
Revised: The untrained and non-speech models are evaluated on both French and English speechstimuli.

Original: 2.5 Layer specificity
Revised: 2.5 Layer specificity

Original: To report the average layer k ∗ with the highest brain score for each voxel (Figure 3) whilebeing robust to regression-to-the-mean biases, we first find the best layer k s for each subjects and each voxel independently and then compute a circular mean across participants:
Revised: To report the average layer k ∗ with the highest brain score for each voxel (Figure 3) whilebeing robust to regression-to-the-mean biases, we first find the best layer k s for each subjects and each voxel independently and then compute a circular mean across participants:

Original: k ∗ =angle (cid:16) 1 N (cid:80)
Revised: k ∗ =angle (cid:16) 1 N (cid:80)

Original: Ns =1 exp (cid:16) 

Revised: Ns =1 exp (cid:16)

Original: iπk s K +1 (cid:17)(cid:17) with K = 19 layers, and N = 412 participants.
Revised: iπk s K +1 (cid:17)(cid:17) with K = 19 layers, and N = 412 participants.

Original: 2.6 Statistics
Revised: 2.6 Statistics

Original: We assess the reliability of brain scores with second-level analyses across participants, by applying a
Revised: We assess the reliability of brain scores with second-level analyses across participants, by applying a

Original: Wilcoxon signed-rank test across participants.
Revised: Wilcoxon signed-rank test across participants.

Original: Thus, the resulting p-values are not affected by fMRIauto-correlation within participants.
Revised: Thus, the resulting p-values are not affected by fMRIauto-correlation within participants.

Original: We perform statistical correction for multiple comparisons with
Revised: We perform statistical correction for multiple comparisons with

Original: Benjamini–Hochberg False Discovery Rate (FDR) across voxels Benjamini [2010].
Revised: Benjamini–Hochberg False Discovery Rate (FDR) across voxels Benjamini [2010].

Original:  Results
Revised:  Results

Original: Wav2Vec 2.0 maps onto brain responses to speech.
Revised: Wav2Vec 2.0 maps onto brain responses to speech.

Original: We estimate whether the activations of
Revised: We estimate whether the activations of

Original: Wav2Vec 2.0 models linearly map onto the human brain activity of 412 individuals listening toaudio books in the fMRI scanner.
Revised: Wav2Vec 2.0 models linearly map onto the human brain activity of 412 individuals listening toaudio books in the fMRI scanner.

Original: For this, we first independently train three models with 600 h of French, English, or Mandarin, respectively, and compute the Neural Predictivity score ( R ) with thecorresponding participants.
Revised: For this, we first independently train three models with 600 hof French, English, or Mandarin, respectively, and compute the Neural Predictivity score ( R ) withthe corresponding participants.

Original: Specifically, we (1) convolve the activations ( X ) of the model with ahemodynamic response function (HRF), (2) train a ℓ 2 -penalized linear regression on a training split tomap them to brain activity Y , and (3) compute the Pearson correlation coefficient between (i) the truefMRI activity and (ii) the predicted activations on a test split.
Revised: Specifically, we (1) convolve the activations ( X ) of the model witha hemodynamic response function (HRF), (2) train a ℓ 2 -penalized linear regression on a trainingsplit to map them to brain activity Y , and (3) compute the Pearson correlation coefficient between(i) the true fMRI activity and (ii) the predicted activations on a test split.

Original: The models’ activations significantly predict brain activity in nearly all cortical areas, reaching the highest R scores in the primary andsecondary auditory cortices (Figure 2-A B).
Revised: The models’ activationssignificantly predict brain activity in nearly all cortical areas, reaching the highest R scores in the primary and secondary auditory cortices (Figure 2-A B).

Original: These scores are significantly higher than those obtainedwith a randomly initialised model ( p < 10 − 50 on average across voxels), and this comparison isrobust across language groups (all p <
Revised: These scores are significantly higher thanthose obtained with a randomly initialised model ( p < 10 − 50 on average across voxels), and thiscomparison is robust across language groups (all p <

Original: 10 − 5 ).
Revised: 10 − 5 ).

Original: Comparison of self-supervised to supervised models.
Revised: Comparison of self-supervised to supervised models.

Original: Does self-supervision reach representationsthat are as brain-like as those obtained with supervised learning?
Revised: Does self-supervision reach representations that are as brain-like as those obtained with supervised learning?

Original: To address this issue, we trained Wav2Vec 2.0 with an alternative, supervised objective, namely, predicting phonetic annotations fromthe same 600 hours of speech sounds.
Revised: To address this issue, wetrained Wav2Vec 2.0 with an alternative, supervised objective, namely, predicting phonetic annotations from the same 600 hours of speech sounds.

Original: We then implemented the R score analyses described above.
Revised: We then implemented the R score analysesdescribed above.

Original: The results show that self-supervised learning in fact leads to modestly but significantly better Rscores than supervised learning (Figure 2-C): ∆ R = 0 .
Revised: The results show that self-supervised learning in fact leads to modestly but significantly better R scores than supervised learning (Figure 2-C): ∆ R = 0 .

Original: 002 , p < 10 6 .
Revised: 002 , p < 10 6 .

Original: The hierarchy of Wav2Vec 2.0 maps onto the hierarchy of the cortex.
Revised: The hierarchy of Wav2Vec 2.0 maps onto the hierarchy of the cortex.

Original: To compare the speechhierarchy in the brain with the functional hierarchy learned by Wav2Vec 2.0, we compare the R scoreof each layer of the model (Figure 3).
Revised: To compare the speechhierarchy in the brain with the functional hierarchy learned by Wav2Vec 2.0, we compare the Rscore of each layer of the model (Figure 3).

Original: First, we observe that convolutional layers are less predictivethan transformer layers.
Revised: First, we observe that convolutional layers are lesspredictive than transformer layers.

Original: Second, within the transformers, the hierarchy of representations alignswith the expected cortical hierarchy [Hickok and Poeppel, 2007]: while low-level areas (A1, A2)are best predicted by the first transformer layers, higher level areas (IFG, STS) are best predicted bydeeper layers.
Revised: Second, within the transformers, the hierarchy of representationsaligns with the expected cortical hierarchy Hickok and Poeppel [2007]: while low-level areas (A1, A2) are best predicted by the first transformer layers, higher level areas (IFG, STS) are best predictedby deeper layers.

Original: Remarkably, this hierarchy extends to supplementary motor and motor areas in bothhemispheres (Figure 3-A).
Revised: Remarkably, this hierarchy extends to supplementary motor and motor areas inboth hemispheres (Figure 3-A).

Original: Language specificity in phone discrimination tasks.
Revised: Language specificity in phone discrimination tasks.

Original: The acoustic features underlying speech(fricatives, vowels, and so on) may also characterize non-speech sounds (the sound of tree leaves inthe wind, of a stone falling, and so on).
Revised: The acoustic features underlying speech(fricatives, vowels, and so on) may also characterize non-speech sounds (the sound of tree leaves inthe wind, of a stone falling, and so on).

Original: Does the model show commonalities merely with generalauditory processing in the brain, or does it capture speech-specific processing?
Revised: Does the model show commonalities merely with generalauditory processing in the brain, or does it capture speech-specific processing?

Original: If so, does it showcommonalities with brain representations that are specific to the native language of the participants,or merely to general speech processing?
Revised: If so, does it showcommonalities with brain representations that are specific to the native language of the participants,or merely to general speech processing?

Original: We first evaluate the specialization of humans’ perceptionto their native language using an ABX behavioral task (Section 2.4).
Revised: We first evaluate the specialization of humans’ perceptionto their native language using an ABX behavioral task (Section 2.4).

Original: Specifically, we compare 

Revised: Specifically, we compare

Original: French and English participants on their ability to distinguish native and non-native phones.
Revised: French and English participants on their ability to distinguish native and non-native phones.

Original: Asexpected [Bohn, 2017, Kuhl et al., 2005], participants were better at discriminating native soundsthan non-native ones (across phone pairs: p < 10 − 18 , Figure 4-A).
Revised: Asexpected Bohn [2017], Kuhl et al. [2005], participants were better at discriminating native soundsthan non-native ones (across phone pairs: p < 10 − 18 , Figure 4-A).

Original: Second, applying the same test toour self-supervised French and English models shows that, like humans, models best discriminatesounds from their ‘native’ language (i.e., the French model better distinguishes French stimuli than
Revised: Second, applying the same testto our self-supervised French and English models shows that, like humans, models best discriminatesounds from their ‘native’ language (i.e., the French model better distinguishes French stimuli than

Original: English ones, across phone pairs, and vice versa: p < 0 . 05 ).
Revised: English ones, across phone pairs, and vice versa: p < 0 . 05 ).

Original: Finally, the random and acoustic modelsobtain the worst ABX accuracy.
Revised: Finally, the random and acousticmodels obtain the worst ABX accuracy.

Original: These results confirm that 600 h of self-supervised learning sufficesfor Wav2Vec 2.0 to learn language-specific representations (Figure 4-B).
Revised: These results confirm that 600 h of self-supervised learningon effective speech suffices for Wav2Vec 2.0 to learn language-specific representations (Figure 4-B).

Original: Wav2Vec 2.0 and the brain learn language specific representations.
Revised: Wav2Vec 2.0 and the brain learn language specific representations.

Original: Next, we compare the
Revised: Next, we compare the

Original: Neural Predictivity scores of random, non-speech, non-native and native models (Figure 4-C D).
Revised: Neural Predictivity scores of random, non-speech, non-native and native models (Figure 4-C D).

Original: First, our results show that the non-speech model attains higher R scores than the random model (onaverage across voxels, ∆ R = 0 . 006 , p= 10 − 31 ) confirming the importance of learning to generatebrain-like representations.
Revised: First, our results show that the non-speech model attains higher R scores than the random model (onaverage across voxels, ∆ R = 0 . 006 , p= 10 − 31 ) confirming the importance of learning to generatebrain-like representations.

Original: Second, non-native models attain higher R scores than the non-speechmodel ( ∆ R = 0 .
Revised: Second, non-native models attain higher R scores than the non-speechmodel ( ∆ R = 0 .

Original: 002 , p = 10 − 9 ), confirming that Wav2Vec 2.0 learns speech-specific representationsof sounds when trained on speech.
Revised: 002 , p = 10 − 9 ), confirming that Wav2Vec 2.0 learns speech-specific representations of sounds when trained on speech.

Original: Finally, the native model attains higher R scores than non-nativemodels ( ∆ R = 0 .
Revised: Finally, the native model attains higher R scores thannon-native models ( ∆ R = 0 .

Original: 002 , p = 10 − 15 ).
Revised: 002 , p = 10 − 15 ).

Original: The specialization for native sounds shared between models andthe brain appears to be mainly localized around the superior temporal sulcus and the middle temporalgyrus (Figure 4-D), whereas higher-level regions like IFG and the angular gyrus only show moregeneral specializations (i.e., speech rather than native-language sounds).
Revised: The specialization for native sounds shared betweenmodels and the brain appears to be mainly localized around the superior temporal sulcus and themiddle temporal gyrus (Figure 4-D), whereas higher-level regions like IFG and the angular gyrusonly show more general specializations (i.e., speech rather than native-language sounds).

Original:  Discussion
Revised:  Discussion

Original: Human infants acquire language with little to no supervision.
Revised: Human infants acquire language with little to no supervision.

Original: A few hundred hours of speech sufficesfor young brains to learn to discretize phonemes, segment morphemes, and assemble words in thelanguage(s) of their social group [Dupoux, 2018].
Revised: A few hundred hours of speech sufficesfor young brains to learn to discretize phonemes, segment morphemes, and assemble words in thelanguage(s) of their social group Dupoux [2018].

Original: Here, we test whether self-supervised learning ona limited amount of speech suffices to yield a model functionally equivalent to speech perception inthe human brain.
Revised: Here, we test whether self-supervised learning ona limited amount of speech suffices to yield a model functionally equivalent to speech perception inthe human brain.

Original: We train several variants of Wav2Vec 2.0 [Baevski et al., 2020] with three curateddatasets of French, English, and Mandarin, and compare their activations to those of a large groupof French, English, and Mandarin speakers recorded with fMRI while passively listening to audio stories.
Revised: We train several variants of Wav2Vec 2.0 Baevski et al. [2020] with three curateddatasets of French, English, and Mandarin, and compare their activations to those of a large groupof French, English, and Mandarin speakers recorded with fMRI while passively listening to audiostories.

Original: Our results show that this self-supervised model learns (1) representations that linearly maponto a remarkably distributed set of cortical regions (Figure 2), (2) a hierarchy that aligns with thecortical hierarchy (Figure 3), and (3) features specific to the language of the participants (Figure 4).
Revised: Our results show that this self-supervised model learns (1) representations that linearly map onto a remarkably distributed set of cortical regions (Figure 2), (2) a hierarchy that aligns with thecortical hierarchy (Figure 3), and (3) features specific to the language of the participants (Figure 4).

Original: These results extend recent findings on the similarities between brain responses to speech and deeplearning models trained with biologically-implausible objectives and data.
Revised: These results extend recent findings on the similarities between brain responses to speech and deeplearning models trained with biologically-implausible objectives and data.

Original: First, fMRI [Kell et al.
Revised: First, fMRI Kell et al.[2018], Millet and King [2021], Thompson et al.

Original: ,2018, Millet and King, 2021, Thompson et al., 2021], electroencephalography [Huang et al., 2018],and multi- or single-unit responses to sounds [Koumura et al., 2019, Begus et al., 2022] have beenshown to be linearly predicted by the activations of deep convolutional networks trained on supervisedauditory tasks.
Revised: [2021], electroencephalography Huang et al.[2018], and multi- or single-unit responses to sounds Koumura et al. [2019], Begus et al. [2022]have been shown to be linearly predicted by the activations of deep convolutional networks trainedon supervised auditory tasks.

Original: For example, [Millet and King, 2021] showed that a supervised model, DeepSpeech2.0 [Amodei et al., 2016], better accounted for brain responses to speech in 102 individuals when itwas trained on speech recognition rather than auditory scene classification.
Revised: For example, Millet and King [2021] showed that a supervised model, DeepSpeech 2.0 Amodei et al. [2016], better accounted for brain responses to speech in 102 individuals when it was trained on speech recognition rather than auditory scene classification.

Original: Similarly, [Kell et al.,2018] showed that eight participants listening to brief speech and non-speech sounds demonstratedfMRI responses in the temporal lobe that aligned with those of a deep convolutional neural networktrained on a dual auditory classification task.
Revised: Similarly, Kell et al. [2018] showed that eight participants listening to brief speech and non-speech soundsdemonstrated fMRI responses in the temporal lobe that aligned with those of a deep convolutionalneural network trained on a dual auditory classification task.

Original: Our results, based on up to 50 times more fMRIrecordings of the entire cortex, not only show that such representational similarities hold with a selfsupervised objective, but reveal the remarkably similar hierarchical organisation of speech processingbetween these two systems [Lerner et al., 2011, Berezutskaya et al., 2017, Caucheteux et al., 2021b,a].
Revised: Our results, based on up to 50 timesmore fMRI recordings of the entire cortex, not only show that such representational similaritieshold with a self-supervised objective, but reveal the remarkably similar hierarchical organisationof speech processing between these two systems Lerner et al. [2011], Berezutskaya et al. Caucheteux et al. [2021b,a].

Original: Second, a growing series of MEG [Toneva and Wehbe, 2019, Caucheteux and King, 2022], fMRI[Qian et al., 2016, Pereira et al., 2018, Schwartz et al., 2019, Antonello et al., 2021] and electrophysiology studies [Schrimpf et al., 2021, Goldstein et al., 2022] show that text-based languagemodels trained on very large corpora generate brain-like representations too.
Revised: Second, a growing series of MEG Toneva and Wehbe [2019], Caucheteux and King [2022], fMRI Qian et al. [2016], Pereira et al. [2018], Schwartz et al.

Revised: [2019], Antonello et al.

Original:  While these resultssuggest elements of convergence between language models and the brain, they remain largelybiologically implausible: not only are these algorithms pre-equipped with abstract linguistic unitssuch as characters and words, but they are trained on corpora that no one would ever be able to readin their lifetime.
Revised: [2021] and electrophysiology studies Schrimpf et al. [2021], Goldstein et al. [2022] show that text-based languagemodels trained on very large corpora generate brain-like representations too. While these resultssuggest elements of convergence between language models and the brain, they remain largely biologically implausible: not only are these algorithms pre-equipped with abstract linguistic units suchas characters and words, but they are trained on corpora that no one would ever be able to read intheir lifetime.

Original: In contrast, Wav2Vec 2.0 is here trained with raw speech waveforms, in a quantitythat is comparable to what human infants require for early language acquisition [Dupoux, 2018].
Revised: In contrast, Wav2Vec 2.0 is here trained with raw speech waveforms, in a quantitythat is comparable to what human infants require for early language acquisition Dupoux [2018].

Original: The present study reveals the neural hierarchies and specializations underlying speech comprehensionin the brain with remarkable clarity.
Revised: The present study reveals the neural hierarchies and specializations underlying speech comprehension in the brain with remarkable clarity.

Original: First, this hierarchy is aligned with the anatomy: e.g. thesuperior temporal sulcus and the temporal pole are known to project to the ventral and dorsal part ofthe inferofrontal gyrus, respectively [Petkov et al., 2015].
Revised: First, this hierarchy is aligned with the anatomy: e.g. thesuperior temporal sulcus and the temporal pole are known to project to the ventral and dorsal partof the inferofrontal gyrus, respectively Petkov et al.

Original: Second, the identification of functionalgradients within the prefrontal cortex, and down to the motor areas typically associated with larynxand mouth control [Dichter et al., 2018] reinforces the relevance of motor processes to speechperception [Kellis et al., 2010, Mugler et al., 2014, Shamma et al., 2021].
Revised: Second, the identification of functionalgradients within the prefrontal cortex, and down to the motor areas typically associated with larynx and mouth control Dichter et al. [2018] reinforces the relevance of motor processes to speechperception Kellis et al. [2010], Mugler et al. [2014], Shamma et al.

Original: Finally, the existenceof multiple levels of representations around the inferofrontal cortex is consistent with the idea that
Revised: Finally, the existenceof multiple levels of representations around the inferofrontal cortex is consistent with the idea that

Original: Broca’s area may be responsible for merging linguistic units [Chomsky, 2000, Friederici, 1999,
Revised: Broca’s area may be responsible for merging linguistic units Chomsky [2000], Friederici [1999],

Original: Hagoort, 2005, Poeppel et al., 2012].
Revised: Hagoort [2005], Poeppel et al.

Original: It should be noted, however, that our results aggregate a largecohort of individuals which could mask a more modular organization at the individual level.
Revised: It should be noted, however, that our results aggregate a largecohort of individuals which could mask a more modular organization at the individual level.

Original: Still, several major gaps remain between self-supervised speech models like Wav2Vec 2.0 and thebrain.
Revised: Still, several major gaps remain between self-supervised speech models like Wav2Vec 2.0 and thebrain.

Original: First, its transformer layers are not temporally constrained: each layer can access all elementswithin the contextual window.
Revised: First, its transformer layers are not temporally constrained: each layer can access all elementswithin the contextual window.

Original: This differs from the necessarily recurrent nature of processing inthe brain.
Revised: This differs from the necessarily recurrent nature of processing in thebrain.

Original: Second, Wav2Vec 2.0 shows differences with humans in recent behavioral studies: itshows a heightened sensitivity to band-pass filtering and an under-reliance on fine temporal structures[Weerts et al., 2021].
Revised: Second, Wav2Vec 2.0 shows differences with humans in recent behavioral studies: it showsa heightened sensitivity to band-pass filtering and an under-reliance on fine temporal structures Weerts et al.

Original: It also fails to predict categorical effects on perception [Millet et al., 2021].
Revised: It also fails to predict categorical effects on perception Millet et al.

Original: Third, recent studies show that Wav2Vec 2.0 encodes significantly less semantic information thantext-based models [Pasad et al., 2021].
Revised: Third, recent studies show that Wav2Vec 2.0 encodes significantly less semantic information thantext-based models Pasad et al.

Original: These limitations may be due to the time scales of Wav2Vec2.0 which, unlike humans, learns very short-term representations of speech.
Revised: These limitations may be due to the time scales of Wav2Vec2.0 which, unlike humans, learns very short-term representations of speech.

Original: Overall, given thatthe human brain remains the best known system for speech processing, our results highlight theimportance of systematically evaluating self-supervised models on their convergence to human-likespeech representations.
Revised: Overall, given thatthe human brain remains the best known system for speech processing, our results highlight theimportance of systematically evaluating self-supervised models on their convergence to human-likespeech representations.

Original: The complexity of the human brain is often thought to be incompatible with a simple theory: “Evenif there were enough data available about the contents of each brain area, there probably would notbe a ready set of equations to describe them, their relationships, and the ways they change over time”[Gallant, 2013].
Revised: The complexity of the human brain is often thought to be incompatible with a simple theory: “Evenif there were enough data available about the contents of each brain area, there probably would not bea ready set of equations to describe them, their relationships, and the ways they change over time” Gallant [2013].

Original: By showing how the equations of self-supervised learning give rise to brain-likeprocesses, this work is an important challenge to this view.
Revised: By showing how the equations of self-supervised learning give rise to brain-likeprocesses, this work is an important challenge to this view.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes ] See Section[-
-]
Revised: [Yes] See Section[-
-]

Original: (b) Did you describe the limitations of your work
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section
Revised: [Yes] See Section

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms

Original: to them?
Revised: to them?

Original: [Yes] We have, and since we use results of human experiments that were
Revised: [Yes] We have, and since we use results of human experiments that were

Original: previously made, and that respect these review guidelines, we conform to them.
Revised: previously made, and that respect these review guidelines, we conform to them.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: No theoretical
Revised: No theoretical

Original: results
Revised: results

Original: b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: No theoretical results
Revised: No theoretical results

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?

Original: [Yes] The code will
Revised: [Yes] The code will

Original: [Table]

Original: be provided upon request.
Revised: be provided upon request.

Original: Otherwise for the rest, see Section 2.2 and 2.1.3(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: Otherwise for the rest, see Section 2.2 and 2.1. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Section 2.1.3.(c)
Revised: [Yes] See Section 2.1.3.

Original:  Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] This is the case for all our figures, see Figure 2, 3 and4(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [Yes] This is the case for all our figures, see Figure 2, 3 and (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [No] The models we train are not new:
Revised: [No] The models we train are not new:

Original: [Table]

Original: the original papers already give that type of information.
Revised: the original papers already give that type of information.

Original: [Table]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...

Original: [Table]

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 2.2, 2.1.3,and 2.4.(b)
Revised: [Yes] See Section 2.2, 2.1.3, and 2.4.

Original:  Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [No] They are all under CC0 for the English [Table]
Revised: [No] They are all under CC0 for the En-

Original:  and French training audio data, the Perceptimatic Benchmark, The Little Prince datasetand the Narratives dataset.
Revised: glish and French training audio data, the Perceptimatic Benchmark, The Little Prince dataset and the Narratives dataset.

Original: Magic data is under CC BY-NC-ND 4.0, Audioset is underCC BY 4.0 for the audio recordings, and CC BY-SA 4.0 for the ontology, Narratives’preprocessing code is under GNU GENERAL PUBLIC LICENSE and fairseq code is 
Revised: Magic data is under CC BY-NC-ND 4.0, Audioset is under CC BY 4.0 for the audio recordings and CC BY-SA 4.0 for the ontology, Narratives’ preprocessing code is under GNU GENERAL PUBLIC LICENSE andfairseq code is under MIT License.

Original: [Table] under MIT License. [Table] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: The code will be given upon request
Revised: The code will be given upon request

Original: [Table] (d) Did you discuss whether and how consent was obtained from people whose datayou’re [Table]
Revised: (d) Did you discuss whether and how consent was obtained from people whose data

Original:  using/curating?
Revised: you’re using/curating?

Original: [Yes] See Section 2.2 and 2.4.
Revised: [Yes] See Section 2.2 and 2.4.

Original: More details on the way theygave
Revised: More details on the way they

Original:  their consent can be seen in the original papers/websites that provided the experimental
Revised: gave their consent can be seen in the original papers/websites that provided the exper-

Original:  results.
Revised: imental results.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifi-

Original:  information or offensive content?
Revised: able information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: The data were anonymized before we usedthem.
Revised: The data were anonymized before we used them.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: We did not conduct the experiments ourselves
Revised: We did not conduct the experiments ourselves

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: We did not conduct the experiments
Revised: We did not conduct the experiments

Original: ourselves
Revised: ourselves

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: We did not conduct the experiments our-
Revised: We did not conduct the experiments our-

Original: selves
Revised: selves

Revised: A AppendixA.1 Self-supervised loss

Revised: formula

Revised: Wav2Vec 2.0, when trained in a self-supervised way, uses a loss ( L ) which is the weighted combination of two losses: one diversity loss ( L d ), which pushes the quantization module to containrepresentations that are as diverse as possible, and one Contrastive Predictive Coding loss ( L m ),which pushes the model to choose, from the context network output c , the right quantized representation ( q ) of some masked input, among other possible representations.

Revised: L m has the followingformula, for some masked time step t :

Revised: [Equation]

Revised: with sim( a , b ) = a T b / ∥ a ∥∥ b ∥ , κ the temperature, which is constant during training, Q t the set of

Revised: K + 1 quantized candidate the model has to choose from, including the right one, i.e. q t .

Revised: L d is included to encourage the equal use of the V possible entries of each of the G codebooks ofthe quantization module.

Revised: The goal is to maximize the entropy of the averaged softmax distributionover the codebook entries for each codebook ¯ p g , across a set of utterances:

Revised: [Equation]

Revised: A.2 Supervised loss formula

Revised: When trained in a supervised way, Wav2Vec 2.0 is trained to optimise a Connectionist Temporal

Revised: Classification loss parameterized over θ :

Revised: [Equation]

Revised: where m θ ( U ) ∈ R d τ × d v are the probabilistic predictions of the model at each τ time sample giventhe input raw waveform

Revised: U ∈ R d τ × d u , V ∈ R d t × d v are the true transcriptions of U , and a U,V is theset of all possible alignments between U and V .

Revised: A.3 Preprocessing of the model’s activations

Revised: The activations of the network X ∈ R d ˆ t × d x are first normalized to be between [0 , 1] for eachlistening session.

Revised: Then, we use nistats Abraham et al.

Revised: [2014] compute_regressor function withthe ‘glover’ model to temporally convolve ( h ∈ R d ˆ t ) and temporally down-sample ( using g : R d ˆ t → 568 R d t ) each artificial neuron j :

Revised: [Equation]

Revised: A.4 Penalized linear model - Ridge regression

Revised: For each split s , we fit an ℓ 2 -penalized linear model V ∈ R d x × d z trained to predict the transformed

Revised: BOLD time series from the model activations for each dimension independently.

Revised: The formula of theoptimization is the following:

Revised: [Equation]

Revised: A.5 Probing the linguistic features encoded in Wav2Vec2 activations

Revised: To investigate what features the different layers of Wav2Vec 2.0 encode, we perform a ridge regression on the Timit dataset 4 to predict five auditory and linguistic features from the activation functions of each layer and model of the present paper.

Revised: We explore the following features: the MELspectrogram of the audio, the transcripted phonemes, the word embedding and part-of-speech targs,and the sentence embedding of each 10 s sample.

Revised: The results show that the layers of Wav2Vec 2.0partially follows the hierarchy predicted from neuro-linguistics Hickok and Poeppel [2007]

Revised: (Table

Revised: S4): the first layers of the transformer best account for the spectro-temporal information, whereasdeeper layers best account for the phonetic, word-level and sentence level information.

Revised: While allof these features emerge with training (Figure S1), only the highest-level features (phone, word andsentence-level) appear to be specific to speech and to the language with which wav2vec 2.0 wastrained (Figure S1).

Revised: To investigate what features the different layers of Wav2Vec 2.0 encode, we perform a ridge regression on the Timit dataset 5 to predict five auditory and linguistic features from the activation functionsof each layer and model of the present paper.

Revised: We study the following features:

Revised: • the MEL spectrogram of the audio, computed using librosa (d=128)

Revised: • the phonemes (categorical features).

Revised: We use the transcripts and alignments provided inTimit.• the word embedding and part-of-speech of the words.

Revised: The time alignments for words areprovided by Timit.

Revised: We use spaCy to compute the word embedding (medium model, d=300),and their part-of-speech (categorical feature, d=19).

Revised: • the sentence embedding of each sample, provided by Laser.

Revised: We use a subset of 1680 samples from Timit, each sample being an audio recording of a short sentence (<10 seconds) from 24 speakers.

Revised: The model’s activations were mean-pooled to the samplingrate of each feature.

Revised: The results show that the layers of Wav2Vec 2.0 partially follows the hierarchy predicted from neurolinguistics Hickok and Poeppel [2007]

Revised: (Table S4): the first layers of the transformer best account forthe spectro-temporal information, whereas deeper layers best account for the phonetic, word-leveland sentence level information.

Revised: While all of these features emerge with training (Figure S1), onlythe highest-level features (phone, word and sentence-level) appear to be specific to speech and to thelanguage with which wav2vec 2.0 was trained (Figure S1).

Revised: A.6 Noise ceiling analysis

Revised: The noise in fMRI recordings is inevitable.

Revised: To estimate the maximum explainable signal given thislevel of noise, we follow previous studies and employ a shared-response model, or "noise ceiling"

Revised: Huth et al.

Revised: [2016], Caucheteux and King [2022], Caucheteux et al.

Revised: [2021c].

Revised: Precisely, we predictthe brain signals of one subject given the brain activity of the other subjects, in response to the sameaudio recording.

Revised: In practice, we apply the same evaluation as Equation (2.3), for one subject s andone voxel v , but we use the average brain signals of other subjects’ brains Y ( s ) = 1 |S| (cid:80) s ′ ̸

Revised: = s Y ( s ′ )instead of the activations X .

Revised: As a result, the "noise ceiling" of one subject ( s ) and one voxel ( v ) iscomputed as follows:

Revised: [Equation]

Revised: (7) where W is an ℓ 2 -penalized linear regression fitted on separate train data, using a cross validationsetting with five test folds.

Revised: We compute such noise ceiling on 290 subjects of the Narrative dataset listening to the same stories(Figure S2).

Revised: We report the noise ceiling across voxels in Figure S2, and, in Table S1, the neuralpredictivity of the networks studied in the main paper normalised by the noise ceiling.

Revised: Precisely,for each voxel, we divide the average neural predictivity by the noise ceiling for this particularvoxel.

Revised: While low on average, the unsupervised Wav2Vec2 model reaches 74% of the noise ceilingin Heschel, and more than 20% in STS, STS and IFG.

Revised: Below, we report the neural predictivity of our models, normalised by such noise ceiling.

Revised: Precisely,we compute the neural predictivity for each subject and voxels

Revised: [Table]

Revised: Random Wav2Vec

Revised: [Table]

Revised: Non-Speech

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Non-Native 17.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Native, Supervised 18.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Native, Unsupervised 18.8% 37.9% 74.4% 40.3% 31.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Wav2Vec

Revised: [Table]

Revised: Acoustic Wav2Vec

Revised: [Table]

Revised: Mandarin Wav2Vec

Revised: [Table]

Revised: French Wav2Vec

Revised: [Table]

Revised: Dutch Wav2Vec

Revised: [Table]

Revised: English Wav2Vec

Revised: [Table]

Revised: English Wav2Vec

Revised: [Table]

Revised: supervised)

Revised: [Table]