Paper: Turning large language models into cognitive models (ICLR-2024)

Reviewer comment (Presentation):
While the authors do a good job in couching the generality of their method and limitations in the Discussion, this could be set up further in the Introduction. At times, the motivation feels a bit confused. There is a difference between seeking to bridge the gap between out-of-the-box language models and fits to behavioral data, versus actually using LLMs as cognitive models to predict and study human behavior. This work supports and expands on the former and teases the possibility of the latter, but the results do not yet fully convince that these models are ready to serve as stimuli-computable models of human behavior.

Author response:
We agree with the reviewer’s interpretation of results and thank them for their feedback. We have updated the end of our introduction to better reflect this sentiment:

**Taken together, our work demonstrates that it is possible to align large language models with human behavior by finetuning them on data from psychological experiments. We believe that this could open up new opportunities to harvest the power of large language models to inform our theories of human cognition in the future.**

Reviewer comment (Evaluation):
It is not clear that CENTaUR is better than BEAST (Fig 1c). While the NLL is objectively lower, there is no information about error bars or whether the difference is significant. Although qualitative strengths of CENTaUR are highlighted, a deeper dive into the gaps between CENTaUR and human performance would be helpful. For example, constructing the inverse of Fig 5 to show cases where CENTaUR did not match humans (but perhaps BEAST or a hybrid did), or addressing the note that 8 out of 60 individual participants are not well-captured.

Author response:
We thank the reviewer for bringing up this question. There are two main ways of conducting significance tests in our setting: (a) treating outcomes of individual participants as samples from a random variable or (b) treating outcomes for runs with different seeds as samples from a random variable.

(a) can only be done for the horizon task data as data from individual participants is not available for the choices13k data set. That being said, we have added an additional analysis regarding (b) to our paper. We have repeated our model fitting procedure 5 times and performed significance tests on the resulting data. In general, we found very little variation across seeds. We have added the following information to our updated PDF:

**To ensure that our fitting procedure is robust to random noise, we repeated this analysis with five different seeds. CENTaUR's results were almost identical across seeds with an average negative log-likelihood of 48002.3 (SE = 0.02) for the choices13k data set and an average negative log-likelihood of 25968.6 (SE = 0.02) for the horizon task. In both cases, these values were significantly lower than those of the domain-specific models (p<0.001).**

Reviewer comment (Experiments):
The choices13k dataset appears to have been released in 2019, raising the possibility that the models were trained on this data. Although LLaMA performance is poor initially and this may reduce concern, it would be good for the authors to comment as it could impact performance and is a nuance when leveraging these models as cognitive models. One idea to explore this would be to repeat the prompt variation in Appendix A.4 in the training set as well.

Author response:
We thank the reviewer for pointing this out. It is, in principle, possible that the data sets are part of CommonCrawl which is used to train LLaMA. However, we believe that this is unlikely given that the raw data is stored as .csv and .mat files. Further evidence for this conjecture comes from the observation that non-finetuned LLaMA does not predict human choices well.

Reviewer comment (Evaluation):
It would be interesting to know if there is any visible structure in the embeddings learned. Have the authors run visualizations (e.g., tSNE) to look for possible structures that could explain why some participants are not captured by CENTaUR? While not necessary and not affecting the score, this could strengthen the work.

Author response:
We have indeed conducted a few basic embedding visualizations. We have now included them in our updated PDF under Appendix A.7. In particular, we write:

**We performed a simple t-SNE (Van der Maaten & Hinton, 2008) analysis to investigate the information contained in the LLaMA embeddings. Figure 7 shows the results of this analysis. We found that the different tasks can be separated in the t-SNE space and that the embeddings for the experiential-symbolic task lie in between those of the two individual tasks. While the t-SNE space also displays some structure with regard to the difference in values between both options, the overall picture is less clear in this case.**

Reviewer comment (Experiments):
My primary substantive concern is that the model is only evaluated on publicly available datasets. Do the authors know whether this data is included in LLaMa's pretraining data? I'm not entirely certain, but given the open-source nature of the model I think it should be possible to determine this. This seems like an important factor for considering how generalizable the approach will be to new datasets.

Author response:
We thank the reviewer for pointing this out. It is, in principle, possible that the data sets are part of CommonCrawl which is used to train LLaMA. However, we believe that this is unlikely given that the raw data is stored as .csv and .mat files. Further evidence for this conjecture comes from the observation that non-finetuned LLaMA does not predict human choices well.

Reviewer comment (Experiments):
For the holdout task, is there an appropriate domain-specific model with which to compare CENTaUR?

Author response:
The original paper that introduced this experimental paradigm did not suggest a domain-specific model. While we could potentially come up with one, we think that this is out-of-scope for the current project.

Reviewer comment (Writing):
Is CENTaUR an acronym, and if so what does it stand for?

Author response:
It is not an acronym. The capitalization was chosen in a way that matches that of LLaMA.

Reviewer comment (Experiments):
Although using open-sourced models (e.g., llama) is a good choice, the most powerful models to date (including instruction fine-tuned ones) are not tested as a baseline method (e.g., few-shot evaluations on GPT-3.5/GPT-4/PaLM-2/Claude/instruction-finetuned llama 2 variants), it is suggested that some of those models can also demonstrate human-like behaviors in some human decision making tasks, through prompting or few-shot evaluations [A]. The few-shot method might be done by prompting some of the training examples in the CENTaUR method. So, it might be helpful to prove that CENTaUR is better than other prompting-based methods on more powerful LLMs.

Author response:
We thank the reviewer for raising this point. We have included GPT-4 as a baseline in our analysis based on this feedback. In general, we find that GPT-4 predicts human behavior better than LLaMA but not to the level of CENTaUR. Note that we had to perform this analysis using accuracy as a metric since GPT-4 does not provide us with log probabilities. We have added the following text to our paper:

**Figure 6 contains a validation of our main analysis using accuracy as a measurement of goodness-of-fit. We again found that CENTaUR was the best performing model, achieving accuracies of 65.18% for the choices13k data set and 83.5% for the horizon task. We also included GPT-4 (OpenAI, 2023) as a baseline in this analysis. GPT-4 performed worse when compared to CENTaUR for both the choices13k data set (59.72%) and the horizon task (80.3%) but better than LLaMA without any finetuning.**

The resulting plot can be found in Appendix A.3 in our updated PDF.

Reviewer comment (Novelty):
Missing some discussions to previous works using LLMs to simulate and replicate behavior study results: [A-E]. Some similar aspects for using LLM in decision-making evaluations have already been proposed by Aher et al. [A].

Author response:
Thanks a lot for pointing us to these references. We have included all of them at the appropriate places:

Recently, researchers even started to wonder whether large language models can replace human participants in experimental studies (Dillion et al., 2023), **i.e., whether they can be used as proxies for human behavior (Aher et al., 2023; Jiang et al., 2022).**

To summarize, large language models are an immensely powerful tool for studying human behavior **(Demszky et al., 2023; Frank, 2023b).**

Previous work has shown that these models can even successfully navigate when they are placed into classic psychological experiments (Binz & Schulz, 2023; **Shiffrin & Mitchell, 2023;** Coda-Forno et al., 2023; Dasgupta et al., 2022; Hagendorff et al., 2022).

Reviewer comment (Presentation):
Minor problems (only suggestions, not affecting the score): please use vectorized graphics (Figure 6 is good, but Figures 1-5 look blurry).

Author response:
We have updated all figures with their vectorized equivalents (see updated PDF).

Reviewer comment (Evaluation):
Is there any statistical analysis (e.g., significance tests) for comparing the correlation between human and CENTaUR predictions?

Author response:
We thank the reviewer for bringing up this question. There are two main ways of conducting significance tests in our setting: (a) treating outcomes of individual participants as samples from a random variable or (b) treating outcomes for runs with different seeds as samples from a random variable.

(a) can only be done for the horizon task data as data from individual participants is not available for the choices13k data set. In fact, we have already done an analysis like this for the horizon task data: the random-effects model selection procedure from Section 3.3 can be considered as a fancy Bayesian version of a significance test (it gives the probability that a model is better than the considered alternatives).

Furthermore, we have added an additional analysis regarding (b) to our paper. We have repeated our model fitting procedure 5 times and performed significance tests on the resulting data. In general, we found very little variation across seeds. We have added the following information to our updated PDF:

**To ensure that our fitting procedure is robust to random noise, we repeated this analysis with five different seeds. CENTaUR's results were almost identical across seeds with an average negative log-likelihood of 48002.3 (SE = 0.02) for the choices13k data set and an average negative log-likelihood of 25968.6 (SE = 0.02) for the horizon task. In both cases, these values were significantly lower than those of the domain-specific models (p<0.001).**

Reviewer comment (Experiments):
BEAST does not appear to be the best or only relevant baseline, and it is usually evaluated using mean squared error, which differs from the approach in the paper.

Author response:
We have opted to measure goodness-of-fit on the choices13k data set using the negative log-likelihood (instead of the MSE which is commonly used for this data set) because this allows us to use consistent metrics across tasks. We agree with the reviewer that there are other plausible baselines. To highlight this, we have added the following footnote:

**There are a number of other domain-specific models that could be used as baselines instead (Bourgin et al., 2019; He et al., 2022; Zhang & Yu, 2013; Wilson et al., 2014). While the ones we selected are good representatives of the literature, future work should strive to conduct a more exhaustive comparison.**

The footnote also refers to other work on the choices13k data set. We are happy to add additional references here if there are any specific suggestions.

Reviewer comment (Novelty):
There is a history of work involving the choices13k dataset with machine learning methods that the authors do not review.

Author response:
We have opted to measure goodness-of-fit on the choices13k data set using the negative log-likelihood (instead of the MSE which is commonly used for this data set) because this allows us to use consistent metrics across tasks. We agree with the reviewer that there are other plausible baselines. To highlight this, we have added the following footnote:

**There are a number of other domain-specific models that could be used as baselines instead (Bourgin et al., 2019; He et al., 2022; Zhang & Yu, 2013; Wilson et al., 2014). While the ones we selected are good representatives of the literature, future work should strive to conduct a more exhaustive comparison.**

The footnote also refers to other work on the choices13k data set. We are happy to add additional references here if there are any specific suggestions.

Reviewer comment (Evaluation):
The authors do not provide more interpretable metrics such as accuracy.

Author response:
Thanks a lot for this suggestion. We have replicated our main analysis using accuracy as a metric. The results mirror those obtained using negative log-likelihoods. We have added the following text to our paper:

**Figure 6 in Appendix A.3 further corroborates these results using accuracy as a measurement of goodness-of-fit.**

and the following to the Appendix:

**Figure 6 contains a validation of our main analysis using accuracy as a measurement of goodness-of-fit. We again found that CENTaUR was the best performing model, achieving accuracies of 65.18% for the choices13k data set and 83.5% for the horizon task.**

The resulting plot can be found in Appendix A.3 in our updated PDF.

Reviewer comment (Evaluation):
The claims of the paper could be toned down, as the space of human behavior is massive, and success in fine-tuning to narrow ranges of behavior does not guarantee success with other kinds of behaviors.

Author response:
We agree with the reviewer’s assessment and have been careful to not make such a claim. For example, we write that:

We have presented preliminary results in this direction, showing that a model finetuned on two tasks can predict human behavior on a third. However, we believe that our current results only hint at the potential of this approach.

and:

However, it is important to mention that the present paper still falls short of this goal.

Furthermore, we have modified the ending of our introduction to tone down some of its claims a bit:

We show that this approach can be used to create models that describe human behavior better than traditional cognitive models. We verify our result through extensive model simulations, which confirm that finetuned language models indeed show human-like behavioral characteristics. Furthermore, we find that embeddings obtained from such models contain the information necessary to capture individual differences. Finally, we highlight that a model finetuned on two tasks is capable of predicting human behavior on a third, hold-out task. **Taken together, our work demonstrates that it is possible to align large language models with human behavior by finetuning them on data from psychological experiments. We believe that this could open up new opportunities to harvest the power of large language models to inform our theories of human cognition in the future.**