Paper: Instruction-tuned LLMs with World Knowledge are More Aligned to the Human Brain (ICLR-2024)

Reviewer comment (Reproducibility):
It remains unclear in the whole passage that which “internal representations” from LLMs are used, which makes it difficult to reproduce the results.

Author response:
We thank the reviewer for pointing out areas that require additional clarifications. We have improved our paper’s clarity by adding details on which layers were used to evaluate brain alignment, and the types of world knowledge in BBH and MMLU.

Reviewer comment (Writing):
The “world knowledge” part in the BBH dataset is different from the knowledge required in MMLU. The former consists of subsets such as Sports Understanding, Movie Recommendation, and Causal Judgement; while the latter is mainly about disciplinary knowledge such as Anatomy and College Physics. This makes the key term “world knowledge” ambiguous. What knowledge is considered “world knowledge”? Are there any differences between factual, general, and disciplinary knowledge?

Author response:
We agree with the reviewer. Hence, we restricted our selection of LLMs to only two model families. In each model family, the models are trained on a similar format of instruction data, so they have similar ability to follow the instruction format of MMLU and BBH. Thus, any differences in their MMLU or BBH performance would point to differences in the quantity of knowledge they possess, rather than their ability to follow the instruction format of MMLU or BBH.

Reviewer comment (Evaluation):
The authors use the performance on MMLU and BBH to represent the models’ capability of “world knowledge”. However, performance on these benchmarks is affected not only by the quantity of knowledge that the models possess, but also by their ability to follow instructions. Thus, a higher performance on MMLU and BBH doesn’t necessarily mean that the model has more world knowledge, and the correlation between benchmark scores and brain scores does not necessarily show a link between world knowledge and the fit to human neural data.

Author response:
We agree with the reviewer. Hence, we restricted our selection of LLMs to only two model families. In each model family, the models are trained on a similar format of instruction data, so they have similar ability to follow the instruction format of MMLU and BBH. Thus, any differences in their MMLU or BBH performance would point to differences in the quantity of knowledge they possess, rather than their ability to follow the instruction format of MMLU or BBH.

Reviewer comment (Experiments):
Lack of comparative analysis with other tuning techniques such as reinforcement learning from human feedback (RLHF).

Author response:
RLHF is often performed *after* the instruction-tuning phase. Hence, we did not consider it as a comparable alternative to instruction-tuning. In order to perform a comparative analysis between RLHF and instruction-tuning, as suggested by the reviewer, we would need to find LLMs trained with RLHF *instead of* instruction-tuning. Ideally, both LLMs should be trained on the same amount of training data and compute, to allow for a fair comparison. These LLMs also need to be open-source, in order to evaluate their brain alignment. Unfortunately, we could not find such LLMs.

We believe that investigating how RLHF affects the brain alignment of instruction-tuned LLMs constitutes future work. RLHF explicitly aims to align LLMs using human feedback. It would be an interesting direction to see if it also aligns LLMs to human brain activity (brain alignment) and behavior (behavioral alignment).

Reviewer comment (Evaluation):
The investigation of behavioral alignment is limited. A more comprehensive exploration could offer insights into the discrepancy between brain and behavioral alignments and its implications for LLM development and application.

Author response:
We agree with the reviewer’s point. Hence, we mentioned this in our original submission (and keep it in our rebuttal version too). We included a paragraph in the discussion section 6.2, where we argue for the need to examine more dimensions of behavior. We mention that our work and many prior works compare LM and human next-word surprisal on reading tasks as a measure of behavioral alignment. However, this evaluates only a single dimension of LM and human behavior (per-word perplexity and reading times). We highlight the need to create more benchmarks to expand the dimensions of behavior examined for both LLMs and humans, in order to holistically evaluate LLM behavior, as well as LLM-human behavioral alignment.

Hence, we focus instead on the brain alignment of LLMs.  Our paper focuses on investigating how instruction-tuning affects LLMs from a neuroscientific perspective and the key factors underlying brain-LLM alignment (e.g., world knowledge).

Reviewer comment (Reproducibility):
What is the computational cost of this study? 33B model is quite large, are there any quantization techniques used like LoRA?

Author response:
For the 33B models, we only performed inference, not training. Hence, LoRA was not used. All our evaluations involve only model inference. This includes the brain alignment evaluations (Pereira2018, Blank2014, Wehbe2014), behavioral alignment evaluations (Futrell2018), and the evaluation of world knowledge (MMLU) and problem-solving abilities (BBH). These evaluations do not involve model training.

Reviewer comment (Experiments):
The authors use models from just two families, T5 and LLaMA. Looking at Figure 2, it seems that LLaMA models do not show a significant correlation between brain alignment and MMLU score, BBH world knowledge, and model size. The results would be more convincing if the authors could use a few more families such as GPT.

Author response:
In our work, we evaluate the correlation of brain alignment to other LLM properties, such as world knowledge (MMLU) and problem-solving abilities (BBH). We agree with the reviewer that adding a greater variety of models would strengthen the generalizability of our claims. 

However, we realized that the performance on these benchmarks is affected not only by the quantity of knowledge that the models possess, but also by their ability to follow the instruction format of MMLU and BBH. (This point was also mentioned by reviewer WxVH.)

Hence, we restricted our selection of LLMs to only two model families. In each model family, the models are trained on a similar format of instruction data, so they have similar ability to follow the instruction format of MMLU and BBH. Thus, any differences in their MMLU or BBH performance would point to differences in the quantity of knowledge they possess, rather than their ability to follow the instruction format of MMLU or BBH.

If we selected LLMs from many different model families, it would be difficult to control for their ability to follow the instruction format of MMLU and BBH, as they are trained on vastly different instruction formats. This would make it challenging to use their MMLU and BBH results to suggest the quantity of knowledge that the models possess. 

However, we overall agree with the reviewer that greater model variety would strengthen generalizability, so we added it to our paper’s section on limitations and future work.

Reviewer comment (Experiments):
Concerning the tasks related to world knowledge, it appears that these tasks may simply exhibit greater linguistic diversity compared to the other tasks examined. The concept of world knowledge seems somewhat ambiguous, and any clarification could be insightful. For instance, would similar results be observed if more language understanding tasks were added? I was unable to determine how the BBH tasks are categorized into "language understanding" and "world knowledge."

Author response:
For both MMLU and BBH, we follow their respective papers for how they categorized the world knowledge tasks.

The MMLU paper describes that they design the benchmark to measure knowledge from many domains. It contains 57 tasks, categorized by the subject domain of world knowledge tested: STEM, Humanities, Social Sciences, and Others. The STEM category includes questions on computer science, physics, mathematics, etc. The Humanities category includes questions on philosophy, law, history, etc. The Social Sciences category includes questions on politics, sociology, economics, geography, etc. The Others category includes questions on business topics such as finance, accounting, as well as general knowledge of global facts.

The world knowledge category of BBH contains tasks that test for factual and general knowledge. Tasks requiring factual knowledge include: “Sports Understanding” and “Movie Recommendation”. Tasks requiring general knowledge include: “Causal Judgement”, which tests knowledge about causal-reasoning suppositions, and “Ruin Names”, which requires knowledge about human perception and usage of humor in the English language.

Reviewer comment (Experiments):
How did the authors determine the category classification for the BBH tasks?

Author response:
We use the same category classification as the original BBH paper. The BBH paper categorizes tasks into four categories: (1) Algorithmic and Multi-Step Arithmetic Reasoning, (2) Natural Language Understanding, (3) Use of World Knowledge, and (4) Multilingual Knowledge and Reasoning. We add this clarification to our paper.

BBH paper:
M. Suzgun et al., “Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them.”

Reviewer comment (Evaluation):
Is it possible to provide a more detailed analysis regarding world knowledge? Any discussion and additional analysis would be appreciated.

Author response:
For both MMLU and BBH, we follow their respective papers for how they categorized the world knowledge tasks.

The MMLU paper describes that they design the benchmark to measure knowledge from many domains. It contains 57 tasks, categorized by the subject domain of world knowledge tested: STEM, Humanities, Social Sciences, and Others. The STEM category includes questions on computer science, physics, mathematics, etc. The Humanities category includes questions on philosophy, law, history, etc. The Social Sciences category includes questions on politics, sociology, economics, geography, etc. The Others category includes questions on business topics such as finance, accounting, as well as general knowledge of global facts.

The world knowledge category of BBH contains tasks that test for factual and general knowledge. Tasks requiring factual knowledge include: “Sports Understanding” and “Movie Recommendation”. Tasks requiring general knowledge include: “Causal Judgement”, which tests knowledge about causal-reasoning suppositions, and “Ruin Names”, which requires knowledge about human perception and usage of humor in the English language.

Reviewer comment (Novelty):
This paper appears to be a replication of [1,2], specifically focusing on instruction-tuned models. It lacks novelty and originality.

Author response:
**Q1: This paper appears to be a replication of [1,2], specifically focusing on instruction-tuned models. It lacks novelty and originality.** 

Our work presents many new insights for researchers studying the parallels between LLMs and human brains: (a) Instruction-tuning aligns LLM representations to human brain activity, and (b) World knowledge is a key factor underlying this alignment. These are both new directions of study that produced new insights. Hence, our work is different from both [1] and [2], which focus on studying how the next-word prediction surprisal of language models relates to human brain activity and behavior.

[1] Schrimpf, Martin, et al. "The neural architecture of language: Integrative modeling …”

[2] Oh, Byung-Doh, and William Schuler. "Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times?."

Reviewer comment (Novelty):
Increasing the model size and integrating world knowledge (using a larger training dataset) are not surprising new discoveries for improving language modeling.

Author response:
**Q2: Increasing the model size and integrating world knowledge (using a larger training dataset) are not surprising new discoveries for improving language modeling.** 

Our paper is not focused on discovering new methods for improving language modeling. Rather, our paper focuses on evaluating the representational and behavioral similarity of LLMs to the human language system. In particular, we investigate how instruction-tuning affects LLMs from a neuroscientific perspective and the key factors underlying brain-LLM alignment (e.g., world knowledge and model size).

Reviewer comment (Experiments):
This paper should demonstrate the effects of contributing factors separately (world knowledge and model size). The plots seem to be dependent on model size.

Author response:
**Q3: The plots of world knowledge (against brain alignment)  seem to be dependent on model size.**

Our work shows that larger LLMs are more aligned to the brain. However, we wish to find out why. What aspects of larger LLMs make them more brain-aligned? Some possibilities: (1) greater world knowledge, (2) better problem-solving abilities, (3) better next-word prediction ability. Our work conducts further experiments to identify the underlying properties of LLMs (beyond model size) that contribute towards greater brain alignment. We show that world knowledge is a key determinant of brain alignment, more than other tested factors.

To show that the plots of world knowledge are not dependent on model size, we wish to highlight that smaller models instruction-tuned to gain greater capabilities (e.g., Vicuna-13B) can achieve greater brain alignment scores than larger models before instruction-tuning (e.g., LLaMA-33B). For reference, here are the average brain alignment values of the models: LLaMA-13B = 0.220, Vicuna-13B = 0.229, LLaMA-33B = 0.227, and Vicuna-33B = 0.232. The Vicuna-13B model has greater brain alignment than LLaMA-33B although it is less than 40% the size. We observe a similar trend when looking at another set of four models: T5-base, Flan-T5-base, T5-large, Flan-T5-large.

To further demonstrate that model size alone does not determine the brain alignment of an LLM, we ran new experiments to evaluate the brain alignment of a randomly-initialized LLaMA-7B. It is a large model with 7B parameters. The randomly-initialized LLaMA was not trained on any data and hence does not contain world knowledge. It also achieves roughly zero accuracy on the BBH and MMLU benchmarks. We observe that the randomly-initialized model achieves lower brain alignment than the trained LLaMA and instruction-tuned Alpaca-7B version. Prior works have also shown that large random embedding models do not achieve high brain alignment scores [1]. These results demonstrate that there are LLM properties aside from model size that contribute significantly to brain alignment.

[1] Schrimpf, Martin, et al. "The neural architecture of language: Integrative modeling …”

Reviewer comment (Experiments):
How is the 'No Instruction' model trained in Figure 1D? The Alpaca instruction dataset is formed with both non-empty input fields (instruction, input, output) and empty input fields (instruction, output). Did you only use non-empty input fields and remove the instruction in those cases?

Author response:
**Q4: How is the 'No Instruction' model trained in Figure 1D? Did we use all the training data from the Alpaca dataset?**

We used all provided training samples from the Alpaca dataset. This is to ensure that the ablation model is trained on the same data as the instruction-tuned Alpaca-7B, with the only difference that the ablation model does not receive the “instruction” portion during training. We added this clarification (that we use all the provided training samples) to our paper.