Paper: The Temporal Structure of Language Processing in the Human Brain Corresponds to The Layered Hierarchy of Deep Language Models (ICLR-2024)

Reviewer comment (Experiments):
While the authors state they used 10 patients, details are lacking on the specific demographic and clinical characteristics of the patient sample. This could impact the generalizability of the findings.

Author response:
We thank the reviewer for this comment and agree that not including this result obscures the generalizability of our findings. We have added a table to the supplementary material (Supp Table T1), which includes patient age, ethnicity, gender and language. We included a table in the supplementary materials with clinical characteristics of the patients and will make sure we add a reference to it in the main body of the paper (Supp Table T2).

Reviewer comment (Reproducibility):
Preprocessing steps for the ECoG data should be described in more detail (e.g. filtering, re-referencing, artifact rejection).

Author response:
We thank the reviewer for this comment. We do have a section on the preprocessing of the ECoG data in the Supplementary, which we have copied below.  Given the opportunity we will add a reference to this section in the main body of the paper. 

A.2 PREPROCESSING
66 electrodes from all patients were removed due to faulty recordings. Large spikes in the electrode signals, exceeding four quartiles above and below the median, were removed, and replacement samples were imputed using cubic interpolation. We then re-referenced the data to account for shared signals across all electrodes using the Common Average Referencing (CAR) method or an ICA based method (based on the participant’s noise profile). High-frequency broadband (HFBB) power provided evidence for a high positive correlation between local neural firing rates and high gamma activity. Broadband power was estimated using 6-cycle wavelets to compute the power of the 70- 200 Hz band (high-gamma band), excluding 60, 120, and 180 Hz line noise. Power was further smoothed with a Hamming window with a kernel size of 50 ms.

Reviewer comment (Experiments):
The encoding model parameters, such as context length, smoothing windows, regularization, could be optimized more thoroughly. Ablations could be performed to test the effect of these modeling choices.

Author response:
We thank the reviewer for their comment. We followed the encoding procedure of Goldstein et. al 2022, where they predict the brain from GPT2 last layer embeddings. We did this so that our results would build upon what they had observed for the last layer. We also wanted to avoid overfitting our model as we felt this might make our results less generalizable. However, we did verify robustness against different context lengths in the input to GPT2-XL (see Supplementary Figure S2)  and window sizes of neural signals centered around word onset (see Supplementary Figure S12). 

Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A., Aubrey, B., ... & Hasson, U. (2022). Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3), 369-380.

Reviewer comment (Experiments):
The linear interpolation analysis addresses one type of baseline model, but comparisons to other neural language models (BERT, ELMo etc) would be informative.

Author response:
We thank the reviewer for their comment. We acknowledge that the linear interpolation analysis measures our results against a linear baseline model. We therefore took the reviewer’s advice and ran our analysis on BERT embeddings. However, it is important to note that there are several considerations that arise when generating an embedding from BERT, considerations that are not present with GPT2.  Firstly, BERT is designed to train on complete sentences. Given that spoken language doesn't always present in full sentences, we were required to manually intervene in punctuation (aided by the nltk package). Nevertheless, this implies that the word embeddings carry information from subsequent words. From a cognitive viewpoint, this constitutes an entirely different model. 

We extracted embedding from Bert-Large’s 24  layers for predictable words and repeated the analyses. Using BERT,  the results for mSTG,  aSTG and IFG were not significant (p>0.5). Surprisingly, TP demonstrated negative correlation (r=-.6,p<.003). The plots can be seen in Supp file S1.

Reviewer comment (Experiments):
Can you provide summary demographic and clinical details for the patient sample?

Author response:
We thank the reviewer for this comment and agree that not including this result obscures the generalizability of our findings. We have added a table to the supplementary material (Supp Table T1), which includes patient age, ethnicity, gender and language. We included a table in the supplementary materials with clinical characteristics of the patients and will make sure we add a reference to it in the main body of the paper (Supp Table T2).

Reviewer comment (Experiments):
Did you perform any optimization of encoding model parameters and architectures?

Author response:
We thank the reviewer for their comment. We followed the encoding procedure of Goldstein et. al 2022, where they predict the brain from GPT2 last layer embeddings. We did this so that our results would build upon what they had observed for the last layer. We also wanted to avoid overfitting our model as we felt this might make our results less generalizable. However, we did verify robustness against different context lengths in the input to GPT2-XL (see Supplementary Figure S2)  and window sizes of neural signals centered around word onset (see Supplementary Figure S12). 

Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A., Aubrey, B., ... & Hasson, U. (2022). Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3), 369-380.

Reviewer comment (Experiments):
Have you compared model performance to any other neural language models besides GPT-2?

Author response:
Relating to your question about other neural language models, just now we replicated our results for a recently published, larger language model (LLaMA 7 billion parameters) we generated embeddings from its 33 layers using the podcast words. We get significant correlations for the lag layer result for the aSTG, IFG and TP (r=0.587, r=0.825, r=0.639; p< 10e-3, p<10e-8, p<10e-4). We get a similar range of times of peak encoding correlation for these three ROIs as well (0-300ms for IFG and TP, -50ms to 100ms for aSTG). The layers all peak around 0ms in the mSTG which also reproduces our results. Thus, our results were replicated for LLaMA 7b, but not for BERT, further strengthening our claim that autoregressive deep language models are powerful cognitive models of language comprehension in the human brain. The results can be seen in Supp Fig. S13.

Reviewer comment (Evaluation):
Maybe I've missed it, but did you measure the decoding accuracy (Decoder : ECoG -> Words). Before we we ask if we can reconstruct those representations it would be nice to know that they actually carry information about the words and are not just unrelated neural processes. Without this, from Reconstruction : Embeddings -> Signal we know that we can reconstruct and capture the temporal dynamics in the brain signal, but this does not necessarily mean that this dynamics is relevant to language processing (I understand that the area where the signal is from is a language area, but who knows what else it might be doing that is contributing to the signal we are so diligently are trying to reconstruct).

Author response:
We appreciate the reviewer's inquiry. Indeed, Goldstein et al. (2022) undertook decoding on this data set, demonstrating the feasibility of classifying the identity of words from the neural signal. In fact, they revealed that the contextual embedding induced by the final layer could be predicted from the neural signal and by comparing the predicted embedding it is possible to classify the identity of the words. Therefore, the reconstruction of the embedding indeed holds word-specific information. Given the opportunity we will cite these results and explain their importance in the interpretation of our results.

Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A., Aubrey, B., ... & Hasson, U. (2022). Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3), 369-380.

Reviewer comment (Evaluation):
What happens with not-so-much-predictable words? Why?

Author response:
We thank the reviewer for this question. We realized that we did not refer to this result in the manuscript. These analyses  can be found in Supp S4&S5).If given the opportunity we will refer to it in the revision. Interestingly, the temporal ordering of the layers is robust to the choice of predictable or unpredictable words, however it occurs later for unpredictable words (between 0 and 250ms for predictable and between 250ms and 500ms for unpredictable). Though we have not explored this effect in depth, it may indicate a neural response that occurs in response to prediction error.

Reviewer comment (Experiments):
How well would you be able to classify (using as deep and powerful model as needed) individual words from ECoG signal alone? In my mind (in)ability to do this would be informative in terms of whether the recorded activity actually contains strong enough signal to claim that it carries any bits of language representation in the brain.

Author response:
We thank the reviewer for the question.  We believe our answer to A1 applies here as well but please let us know if there is something we missed and we will happily address it.

Reviewer comment (Presentation):
What is the canonic path of a word through the brain? Which areas are being activated in which order according to the modern day knowledge on this? The reader would benefit of a figure explaining this, same as we've seen in all the vision papers.

Author response:
To the best of our knowledge, a definitive neural architecture for word processing does not currently exist. Furthermore, contextual embeddings are not limited solely to representing the identity of a word, but also consider its meaning in relation to its context. However, in a study conducted by Hasson, Chen, and Honey (2015), they propose a model of language comprehension and provide a diagram in Figure 2B (right) that illustrates the flow of information. Interestingly, this diagram demonstrates a similar pathway to the one we describe in our paper (mSTG->aSTG->TP, IFG). Additionally, further supporting our claims is the observation that the order of regions of interest (ROIs) in this pathway correlates with the temporal receptive field, which we measure in our manuscript through the spread of the peaks.
Hasson, U., Chen, J., & Honey, C. J. (2015). Hierarchical process memory: memory as an integral component of information processing. Trends in cognitive sciences, 19(6), 304-313.

Reviewer comment (Experiments):
Are there differences between spoken words and written words? Do their brain-paths converge at some point? Where? Do we believe that representations are different depending on whether a word was perceived auditorily and visually? If semantic representation is what we are after, should we only compare representations "after" this point of convergence once the word took its abstract semantic representation form that is divorced the "mechanics" of delivery?

Author response:
A thorough literature review on the relationship between comprehension of written and spoken language is beyond the scope of our paper, which specifically focuses on speech comprehension. However, Regev, Honey, Simony, & Hasson (2013) did conduct a study comparing the brain dynamics of written and spoken language comprehension. Their findings revealed that certain areas, such as the IFG, were consistently activated regardless of the medium of language. In contrast, other areas, such as the STG, were specifically activated for auditory language, and the visual cortex exhibited unique activation for written words. In alignment with these findings, we commence our paper by emphasizing the significance of the IFG, as it remains invariant across different language modalities and later look at areas that are specific for auditory comprehension (such as STG).
Regev, M., Honey, C. J., Simony, E., & Hasson, U. (2013). Selective and invariant neural responses to spoken and written narratives. Journal of Neuroscience, 33(40), 15978-15988.

Reviewer comment (Evaluation):
Figure 2: Also, if we, once again, remove the signal from the data via some sort of permutation test, how all these plots will look like? Some other shape - which one? Just flat - why? The answer to this question will, in turn, lead to a question about why those shapes look like they do in the absence of the true signal.

Author response:
We thank the reviewer for these questions. We performed an analysis where we phase shuffled the neural signal and re-ran our encoding procedure. The lag layer correlation effect disappears. We also see maximum correlations in the coding plots < 0.05 (much less than for true signal) and none of them turned significant (p>0.1).  The plots can be seen at Supp S7. Given the opportunity we will include this new analysis in the paper.

Reviewer comment (Evaluation):
Figure 2: And one more here: how high the correlation would rise for the data we know should have no correlation? Would it be actually 0 all the way from -4000 to 4000, or because of some general dynamics and the way how to the linear readout models are trained some positive correlation will still be observed? How high?

Author response:
We thank the reviewer for these questions. We performed an analysis where we phase shuffled the neural signal and re-ran our encoding procedure. The lag layer correlation effect disappears. We also see maximum correlations in the coding plots < 0.05 (much less than for true signal) and none of them turned significant (p>0.1).  The plots can be seen at Supp S7. Given the opportunity we will include this new analysis in the paper.

Reviewer comment (Evaluation):
Could the correlation be explained by, say, non-sparsity of the representation in DLM in the mid layers, or some other technical reason, and not by the actual match between representations?

Author response:
We thank the reviewer for this comment. It is always the case that there are confounders we could not control (because we did not think about them). However, the control analysis we introduce in the original paper, as well as the controls suggested by the current reviewers give us confidence in the results.

Reviewer comment (Experiments):
Current work focuses more on layer-wise transformations learned by GPT2-XL map onto the temporal sequence of transformations of natural language. However, the current paper lacks in providing fine-grained details as follows: (i) Why are intermediate layers well aligned with the Brain? (ii) It could be interesting if authors could report more fine-grained details like the following study: Subba Reddy Oota, Mariya Toneva. Joint processing of linguistic properties in brains and language models, NeurIPS-2023, https://arxiv.org/pdf/2212.08094.pdf (iii) Supraword meaning analysis and layer-wise hierarchy details.

Author response:
We thank the reviewer for this comment. Previous work noted  the higher alignment between intermediate layers and the brain during language processing (Toneva and Wehbe 2019 and Caucheteux and King 2022, for example). We also replicated this result as can be shown in Figures 2 and 5, however, the focus of our paper is different - our primary finding revolves around the correspondence between temporal dynamics in the brain and the layered structure (or sequence of embeddings) of GPT2. We demonstrate that the deeper the layer is, the later the embedding it induces correlates with the brain. This suggests a shared dynamic between the brain and GPT-2 computations or representaions. For greater clarity, we have scaled the encodings to 1 in plots 2 and 3, thereby emphasizing the importance of temporal dynamics rather than focusing on determining which layer induces the optimal correlation with the brain. 

Caucheteux, C., King, JR. Brains and algorithms partially converge in natural language processing. Commun Biol 5, 134 (2022). https://doi.org/10.1038/s42003-022-03036-1
Toneva, Mariya, Leila, Wehbe. "Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)." Advances in Neural Information Processing Systems. 2019.

Reviewer comment (Novelty):
Caucheteux et al. 2022 "Brains and algorithms partially converge in natural language processing" shown that both MLM and CLM models yield best alignment with fMRI & MEG in middle layers and reported better performance at predicting next word -> better prediction of fMRI & MEG. How does the current work differ in terms of new findings except for the use of ECoG recordings?

Author response:
We thank the reviewer for the question. Indeed, Caucheteux et al. (2022), also demonstrates correlation with embedding induced by deep language models and the brain. However, here we focus on the shared dynamic between the brain and the flow of information through the GPT-2 layers (see A1 for more details).

Reviewer comment (Evaluation):
It could be interesting if authors could report syntax and low-level surface feature analysis across language regions.

Author response:
We fully acknowledge the intriguing nature of assessing the predictive ability of LLM-induced embeddings in comparison to symbolic language representations such as syntax and part-of-speech (PoS). In fact, we are actively engaged in the writing of a manuscript focused specifically on this topic. However, in the context of the present manuscript, we believe that diverting the discussion towards a comparison with symbolic representations would deviate from its primary scope and purpose.

Reviewer comment (Experiments):
Did authors try the encoding with different context lengths?

Author response:
We thank the reviewer for this question. In response to this request we generated embeddings by running GPT2-XL on the podcast data with input sizes of 500 tokens and 100 tokens. We then re-ran our analysis for correct words in the inferior frontal gyrus (IFG). The global trends are robust to different context lengths during embedding generation (see in Supp S2). If given the opportunity we will report this analysis in the manuscript.

Reviewer comment (Experiments):
Did authors try the encoding performance with previous word representations (i.e. using w{t-1} to predict w_{t} ECoG data, similarly w_{t-2}, w_{t-3} ..)?

Author response:
We thank the reviewer for this question. In our paper, we followed Goldstein et al. (2022) and chose the representation of the w_{t-1} word. Following the reviewer question we repeated the main analysis for embeddings extracted for w{t} w{t-2}, w{t-3}. The results can be seen at Supp S3. If given the opportunity we will include them in the paper. 
Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A., Aubrey, B., ... & Hasson, U. (2022). Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3), 369-380.

Reviewer comment (Experiments):
Did authors try the encoding performance with low-level features, including the number of letters, phonemes, word length, word frequency, etc., to syntax-level features?

Author response:
We fully acknowledge the intriguing nature of assessing the predictive ability of LLM-induced embeddings in comparison to symbolic language representations (e.g., number of letters, phonemes, word length, word frequency, Part of Speech etc.). In fact, we are actively engaged in the writing of a manuscript focused specifically on this topic. However, in the context of the present manuscript, we believe that diverting the discussion towards a comparison with symbolic representations would deviate from its primary scope and purpose. Our primary finding revolves around the correspondence between temporal dynamics in the brain and the layered structure (or sequence of embeddings) of GPT2. We demonstrate that the deeper the layer is, the later the embedding it induces correlates with the brain. This suggests a shared dynamic between the brain and GPT-2 computations or representaions.  For greater clarity, we have scaled the encodings to 1 in plots 2 and 3, thereby emphasizing the importance of temporal dynamics rather than focusing on determining which layer induces the optimal correlation with the brain.

Reviewer comment (Reproducibility):
The authors state that they "are willing to provide our data to those interested in reproducing our experiments". Why not make the data entirely available for future work to build on these results? Is that related to some required privacy clause in the data collection?

Author response:
We sincerely value the reviewer's comment. Given the sensitive nature of the clinical data collected, the institutions responsible for the data collection have opted to share the data through collaborative efforts. Adhering to this policy, they have so far shared the data with numerous institutions.

Reviewer comment (Experiments):
Lack of comparative baselines: The paper only uses GPT2-XL, and other language models such as BERT, as well as recently released LLMs (e.g., LLaMA, Vicuna) should be considered for comparison.

Author response:
We appreciate the comment offered by the reviewer. Three foundational studies establish a connection between language comprehension in the brain and GPT-2 (Goldstein et. al 2022, Caucheteux et al.  2022, Schrimpf et al 2021). In particular, two of these papers (Goldstein et. al 2022, Schrimpf et al 2021) explicitly discuss the selection of autoregressive language models (such as GPT-2) as particularly suitable for modeling brain responses to natural language stimuli as the brain is known to make predictions about subsequent words before they are spoken On the contrary, the BERT model is bidirectional and less suitable for modeling the cognitive system, which does not have access to the future words at prediction time. However, LLAMA is autoregressive, fitting the required model criteria. We are currently generating word embeddings from LLAMA and will re-run our analysis. This process takes some time as we need to get familiar with a new tokenizer and other technical details. If given the opportunity we will add the LLAMA based results. In addition, we will include the rationale we have outlined here in the manuscript for choosing GPT-2 for modeling the process of language comprehension in the brain.
Following the reviewer's request, we also performed an analysis for BERT embeddings. However, it is important to note that there are several considerations that arise when generating an embedding from BERT, considerations that are not present with GPT2.  Firstly, BERT is designed to train on complete sentences. Given that spoken language doesn't always present in full sentences, we were required to manually intervene in punctuation (aided by the nltk package). Nevertheless, this implies that the word embeddings carry information from future words. From a cognitive viewpoint, this constitutes an entirely different model. 
We extracted embeddings from BERT-Large’s 24  layers for predictable words. The mSTG, IFG and aSTG did not have significant results (p>0.5, p> 0.1, p>0.7). Neither the IFG nor the aSTG show the effect of increasing peak encoding correlation time with the layer index that we see with GPT2-XL embeddings. In the TP we get a significant effect (p<0.03) of a negative correlation between layer index and lag, which is unexpected. As pointed out before, there are multiple aspects that need to be considered when using BERT to model neural signal. Thus a further investigation is needed to understand these results. The BERT based plots can be seen in Supp table S1.

Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A., Aubrey, B., ... & Hasson, U. (2022). Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3), 369-380.