Paper: Discovering Divergences between Language Models and Human Brains (ICLR-2024)

Reviewer comment (Novelty):
The idea of task-based modeling and their alignment with Brain is not new. Recently, several research works utilized task-based representations and shown better brain alignment [1], [2], [3], [4].

Author response:
We acknowledge that the idea of aligning task-based models with the brain is not novel in itself. As we pointed out in the paper (“Drawing inspiration from ...”), our experiments in the second part of the paper were inspired by prior work. However, the focus of our paper is to address the systematic differences between LMs' and human brains' responses to written language, and we use the “improving LM-brain alignment via fine-tuning on corresponding tasks” approach to validate the hypotheses we identify in the first part of the paper. Unlike previous work which pre-defines a task, our work focuses on discovering what aspects LMs and the brain differ on through a data driven approach, and based on these findings, we propose related tasks for targeted fine-tuning.

Reviewer comment (Novelty):
The novelty in the paper is limited as it primarily delves into a comparison between three task-based model representations and their brain alignment. Notably, the authors have not discussed or compared with 3 previous works [1], [2] and [3].

Author response:
As noted above and in the manuscript, the main aim of this work is to identify and study differences between brain activity recordings and LM activations and suggest hypotheses for what makes LMs different from the brain. We use the task fine-tuning approach to validate our hypotheses. Our aim was not to claim novelty with respect to the fine-tuning. Acknowledging this oversight, we will incorporate references to the omitted works.

Reviewer comment (Experiments):
There are a lot of questions left in the methodology: Why do authors consider the last layer representations of GPT-2? All the previous linguistic brain encoding studies reveal that intermediate layer representations are well aligned with brain [5], [6], [7].

Author response:
Thank you for your valuable comments. Due to limited time, we cannot run all the analyses but we will include these analyses in our paper and validate whether different layers and context window sizes would show similar patterns.

Reviewer comment (Experiments):
The authors present these task-based findings specifically related to narrative story reading. It's essential to ascertain whether these findings hold true for the listening modality, for instance, by considering datasets like MEG-MASC story listening.

Author response:
This is an interesting direction for future work, and we thank you for suggesting this dataset, but based on recent work in fMRI [1], we don’t expect that semantic representation varies much between the reading and the listening modalities. Thus, we expect that the results have a good chance of being similar.

[1] Deniz, Fatma, et al. "The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality." Journal of Neuroscience 39.39 (2019): 7722-7736.

Reviewer comment (Experiments):
The focus of the authors is primarily on contextual word representations from the last layer of Language Models (LLMs). It would be valuable to explore the representation similarity between pre-trained and fine-tuned layer representations. Moreover, investigating which layers are predominantly affected during the fine-tuning process would add depth to the analysis.

Author response:
Thank you for the suggestion, we will incorporate it in our work.

Reviewer comment (Presentation):
The clarity can be improved as Figure 4 is hard to follow and more details are needed.

Author response:
Thank you for the comments on the writing, we have added details regarding the two points you mentioned.

Reviewer comment (Novelty):
Several citations are missing [1], [2], [4], and [8].

Author response:
Thank you for providing citations that we missed, we have included them in the new version.

Reviewer comment (Experiments):
Why do authors consider the last layer representations of GPT-2? What about the performance of other layer representations?

Author response:
Thank you for your valuable comments. Due to limited time, we cannot run all the analyses but we will include these analyses in our paper and validate whether different layers and context window sizes would show similar patterns.

Reviewer comment (Experiments):
Why do authors consider longer context length i.e. 100?

Author response:
Thank you for your valuable comments. Due to limited time, we cannot run all the analyses but we will include these analyses in our paper and validate whether different layers and context window sizes would show similar patterns.

Reviewer comment (Experiments):
What happens if authors can tune on 3 tasks at the same time?

Author response:
This is a very good idea. We hypothesize that this would improve LM-brain alignment and it would be interesting to compare the model finetuned on all three tasks with the models finetuned on each task. We will incorporate this suggestion into the next version of our draft, though we do not have enough time now to run the experiments.

Reviewer comment (Experiments):
Whether the paper findings hold true for the listening modality, for instance, by considering datasets like MEG-MASC story listening?

Author response:
This is an interesting direction for future work, and we thank you for suggesting this dataset, but based on recent work in fMRI [1], we don’t expect that semantic representation varies much between the reading and the listening modalities. Thus, we expect that the results have a good chance of being similar.

[1] Deniz, Fatma, et al. "The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality." Journal of Neuroscience 39.39 (2019): 7722-7736.

Reviewer comment (Experiments):
Which layers are predominantly affected during the fine-tuning process would add depth to the analysis?

Author response:
Thank you for the suggestion, we will incorporate it in our work.

Reviewer comment (Evaluation):
Why is the parietal region, particularly the angular gyrus, not exhibiting similar enhancement between 250-375ms, considering its association with high-level semantic processing?

Author response:
It’s hard to pinpoint the angular gyrus since we have not done source localization. Most of the sensors do exhibit improvement due to fine-tuning. It’s also unclear that 250-375ms is when the angular gyrus gets involved.