Large Language Models for Psycholinguistic Plausibility Pretesting (2024)

Abstract: In psycholinguistics, the creation of controlled materials is crucial to ensure that research outcomes are solely attributed to the intended manipulations and not influenced by extraneous factors. To achieve this, psycholinguists typically pretest linguistic materials, where a common pretest is to solicit plausibility judgments from human evaluators on specific sentences. In this work, we investigate whether Language Models (LMs) can be used to generate these plausibility judgements. We investigate a wide range of LMs across multiple linguistic structures and evaluate whether their plausibility judgements correlate with human judgements. We find that GPT-4 plausibility judgements highly correlate with human judgements across the structures we examine, whereas other LMs correlate well with humans on commonly used syntactic structures. We then test whether this correlation implies that LMs can be used instead of humans for pretesting. We find that when coarse-grained plausibility judgements are needed, this works well, but when fine-grained judgements are necessary, even GPT-4 does not provide satisfactory discriminative power.

## introduction psycholinguistic research explores humans' exceptional language comprehension abilities, aiming to uncover underlying mechanisms through experiments and cognitive modelling @xcite @xcite @xcite @xcite . researchers use measures such as reading times and comprehension accuracy to compare sentences with distinct processing demands. as an example, @xcite investigated reading times to determine if sentences with two animate nouns (e.g., (1a), (2a)) pose greater processing challenges than those with one animate and one inanimate noun (e.g., (1b), ( 2b )). longer reading times in the (a) sentences would indicate that similarity between the noun phrases interferes with processing. controlling sentence plausibility is therefore essential in sentence processing experiments, and is typically accomplished through pretests, where participants rate sentence plausibility on a scale, guiding the selection of materials for the main experiment. however, plausibility pretesting is a time-and resource-consuming process, involving multiple iterations and prolonged data collection with different participant groups. recently, large language models (llms) @xcite @xcite @xcite have shown human-like performance on various language understanding tasks without task-specific training @xcite . previous studies have established a strong correlation between lms' predicted probabilities and human reading time @xcite @xcite @xcite . thus, it is natural to ask -can lms provide plausibility judgements that are similar to human judgments and consequently be used to reduce the cost of psycholinguistic pretesting? in this study, we investigate the correlation between lms and human plausibility judgments. to accomplish this, we examine four sets of sentences that represent a variety of syntactic structures and plausibility levels, for which human judgements have been collected in prior work in the course of pretesting @xcite @xcite . we then gather multiple lm judgements for these sets from a wide range of lms, and compare average human plausibility ratings and average llm plausibility ratings. our findings indicate that while several llms exhibit high correlation with human judgments on common syntactic structures, only gpt-4 shows strong correlation on the rarer syntactic structures. figure 1 displays the average plausibility ratings of the llm with the highest correlation against average human ratings, along with a linear regression model. the pearson correlation between llm and human judgments is consistently high across all the datasets. interestingly, the fitted linear regressions are quite similar across three of the datasets, indicating robustness in the translation of llm judgements into human judgements. based on these findings, we examine if using llms instead of humans can lead to similar outcomes when filtering materials in the course of pretesting. we find that when pretesting requires coarse-grained plausibility judgements, i.e., when it is used to filter out implausible sentences, llms perform well. however, when fine-grained plausibility judgements are needed, e.g., to ensure that a pair of sentences has similar plausibilty ratings, even gpt-4's performance is not satisfactory yet. to summarize, in this work we thoroughly investigate the correlation between human and lm plausibility judgements across a wide range of lms and syntactic structures. we find that many llms perform well on simple syntactic structures, and gpt-4 performs well across-the-board. we translate this finding into a method for using llms to provide plausibility judgements, and find that performance is high when coarse-grained judgements are needed, but still lagging behind when fine-grained judgements are necessary. ## experimental setup an experiment is defined by instantiating three parameters: (a) the lm used for eliciting plausibility judgements, (b) the prompt provided as input to the lm, and (c) the linguistic dataset used. we leverage data from existing pretests for which human plausibility ratings were already collected @xcite @xcite , and also create our own pretest materials and collect human plausibility judgements for them. in all experiments, we generate 20 plausibility ratings per sentence per lm, using a scale from 1 to 7. we now describe the datasets ( §2.1), lms ( §2.2), and prompts ( §2.3). ## methodology in §3, we saw significant correlation between plausibility judgments of humans and gpt-4. we now evaluate directly the performance of llm judgments when replacing human judgements. plausibility judgements can be used in different ways for constructing experimental materials. 5 three common uses are: (a) filtering out implausible sentences by requiring a minimum average plausibility rating, (b) filtering out plausible sentences by requiring a maximum average plausibility rating, and (c) filtering out sentence pairs that have dissimilar average plausibility ratings. we evaluate the performance of llms across these operations. ## variance of humans vs. lms thus far, we saw that the average plausibility ratings of humans and llms correlate well. it is important to note that this is not the case w.r.t variance. explicitly, human variance is much higher than the variance of lms, despite the high temperature used for sampling, which is 1.5. figure 8 shows the standard deviation for gpt-4 and gpt-3.5 on all the datasets when using the specific prompt, as well as the standard deviation for human judgements. standard deviation for these lms is dramatically lower than humans, i.e., we obtain relatively similar plausibility judgements when sampling multiple times from the model. a possible theoretical explanation for this phenomenon is that the outputs of lms can be viewed as an average over multiple samples, since pretraining is done on texts from many authors. thus, when sampling plausibility ratings from a lm, we are sampling from an average of plausibility ratings. let each human rating r i be a sample from a distribution with mean µ and variance σ 2 . we can view each sample from a lm as an average of n human ratings: 1 n n @xmath0. this is a random variable with mean µ and variance σ 2 n . this observation can be used to estimate for a particular sentence what is the number n of humans that the lm is averaging over, by computing the ratio between the observed variance of humans and the observed variance of the lm for that sentence. ## conclusion we investigate the correlation between plausibility judgements of humans and language models and find high correlation for simple syntactic structures overall, and high correlation throughout for gpt-4. we show language models can be used to provide coarse-grained plausibility judgements, which can reduce the cost of and accelerate psycholinguistic research. we view this work as a first step in this direction, where future work can improve the correlation through finetuning and prompt engineering and further investigate the utility of language models for conducting psycholinguistic research.