Can Transformer Language Models Predict Psychometric Properties? (2021)

Abstract: Transformer-based language models (LMs) continue to advance state-of-the-art performance on NLP benchmark tasks, including tasks designed to mimic human-inspired “commonsense” competencies. To better understand the degree to which LMs can be said to have certain linguistic reasoning skills, researchers are beginning to adapt the tools and concepts of the field of psychometrics. But to what extent can the benefits flow in the other direction? I.e., can LMs be of use in predicting what the psychometric properties of test items will be when those items are given to human participants? We gather responses from numerous human participants and LMs (transformer- and non-transformer-based) on a broad diagnostic test of linguistic competencies. We then use the responses to calculate standard psychometric properties of the items in the diagnostic test, using the human responses and the LM responses separately. We then determine how well these two sets of predictions match. We find cases in which transformer-based LMs predict psychometric properties consistently well in certain categories but consistently poorly in others, thus providing new insights into fundamental similarities and differences between human and LM reasoning.

## introduction the current generation of transformer-based language models (tlms) @xcite continues to surpass expectations, consistently achieving state-of-the-art results on many natural language processing (nlp) benchmark tasks. especially surprising is their remarkable performance on benchmark tasks designed to assess "commonsense" reasoning (e.g., @xcite , possibly owing to their ability to encode and retrieve a surprising amount of structural knowledge @xcite @xcite @xcite , despite initial worries that all connectionist language models in general would suffer the same limitations as previous generations @xcite @xcite @xcite . understanding how tlms reason is a complex task made more difficult by the fact that the sizes of contemporary tlms are so large as to effectively render them black boxes. as such, researchers are continually searching for new methods to understand the strengths and limitations of tlms. one promising approach is to draw from psychometrics, a sub-field of psychology particularly suited to dealing with perhaps the most mysterious black box of them all: the human mind. psychometrics is concerned with psychological measurement-i.e., how to measure latent attributes like reasoning skills, attitudes, and personality traits. psychometricians have developed tools to measure such properties even when the mechanisms that give rise to them are not fully understood, thus suggesting a possible fruitful application of those tools to complex artificial black boxes like tlms. although some have called for bridging the gap between psychometrics and ai @xcite @xcite @xcite , the amount of work attempting to do so is limited: although some existing work attempts to use advances in psychometrics to benefit the study of tlms, none to our knowledge have used sota tlms (or even lms in general) to benefit psychometrics. to illustrate, assume that someone wishes to design a test to assess the degree to which a person possesses mastery of some cognitive skill s. a good place to start is for a panel of experts to de-sign a set of test items (questions) i, such that they believe solving i requires s. however, although many nlp benchmarks tend to consider this sufficient, the items in i only have face validity, in that they only have been demonstrated to superficially test for s. to go beyond face validity, one must assess i's psychometric properties by establishing their validity (how well the items actually measure the phenomenon s they purport to measure), reliability (how stable the items are as measurements), and fairness (how well the items are free from biases against certain sub-populations of subjects). 2 but establishing these psychometric properties can be prohibitively costly, requiring large numbers of human participants to answer the items in i and iteratively refine them. this drawback motivates the central research question of our paper: can tlms be used to predict psychometric properties of test items? if so, the benefit for psychometric practitioners foot_2 is enormous, as it can reduce the need for multiple rounds of costly empirical testing. but the benefits for nlp are significant as well: knowing how the psychometric properties of items differ when applied to artificial versus human populations will give us unique insight into how they solve such problems, and how they can be improved. ## related work what reason do we have to suspect that tlms can predict the psychometric properties of test items? although tlms were not primarily designed to compute in a human-like way, there are some rea-sons to suspect that they may have the ability to effectively model at least some aspects of human linguistic reasoning: they consistently demonstrate superior performance (at least compared to other lms) on human-inspired linguistic benchmarks @xcite , and they are typically pre-trained using a lengthy process designed to embed deep semantic knowledge, resulting in efficient encoding of semantic relationships @xcite @xcite . common optimization tasks for pretraining transformers, such as the masked lm task @xcite are quite similar to the word prediction tasks that are known to predict children's performance on other linguistic skills @xcite @xcite . finally, tlms tend to outperform other lms in recent work modeling human reading times, eye-tracking data, and other psychological and psycholinguistic phenomena @xcite @xcite @xcite . there are many studies probing tlms in various ways, a body of work sometimes called "bertology" @xcite . however, work explicitly bridging psychometrics with ai is less common. @xcite augmented the dina @xcite and @xcite cognitive diagnostic models @xcite with a feedforward neural network that used a semi-supervised learning objective. the architecture achieved superior results to multiple baselines. @xcite created a deep learning architecture for extracting psychometric dimensions related to healthcare, specifically numeracy, literacy, trust, anxiety, and drug experiences. their architecture did not use transformers, and relied instead on a sophisticated combination of convolutional and recurrent layers in order to extract representations of emotions, demographics, and syntactic patterns, among others. @xcite examined the correlation between human and lm next-word predictions and proposed a procedure for achieving more human-like cloze probabilities. in nlp, methods from item response theory (irt) (reckase, 2009) have been particularly popular. @xcite used irt models to study the impact of question difficulty on the performance of deep models on several nlp tasks. in a follow-up study, lalor and yu (2020) used irt models to estimate the competence of lstm @xcite and bert models during training. this allowed them to create a dynamic curriculum learning @xcite algorithm, which achieved superior performance to the same models trained using a static scheduler for several tasks. sedoc and ungar (2020) used irt to efficiently assess chat-bots. @xcite used irt to analyze the performance of machine learning classifiers in a supervised learning task. irt has also been used to evaluate machine translation systems @xcite and speech synthesizers @xcite , and also in computer vision @xcite . this literature clearly indicates that there has been a lot of interest in applying psychometrics to ai. so far, most of this effort has focused on specific use cases, and has not attempted to broadly assess commonalities between machine and human reasoning. most similar to our current work is @xcite , who showed that deep models could achieve a strong correlation with irt parameters fitted using human data on several nlp datasets. however, they compared the human responses to lstms and neural semantic encoders @xcite , and did not consider tlms. furthermore, they focused on the snli dataset, which is less challenging than the glue diagnostic and does not group questions based on fine-grained linguistic competencies. besides the glue diagnostic, other taxonomies have been proposed, such as taxinli @xcite . although taxinli includes some types of reasoning which have no clear analogue in glue, many of their categories are quite similar. 4 since the taxinli questions were also taken from the mnli dataset, we were concerned they would be too easy for some of the larger tlms we planned to evaluate. we, therefore, chose to focus specifically on the challenging glue diagnostic set and leave taxinli for future work. ## gathering language model data the glue and superglue benchmarks @xcite are suites of nlp tasks designed to test the general linguistic capabilities of lms. included as part of the glue benchmark is a set of diagnostic questions, called the broad coverage diagnostic, which are all formatted as natural language inference (nli) problems. nli problems consist of two sentences: a premise (p) and hypothesis (h), and solving such a problem involves assessing whether p textually entails h. there are typically three choices: either p does textually entail h (entailment), p entails that h is impossible (contradiction), or h's truth can not be determined from p alone (neutral). the nli task is therefore quite general and can encompass a wide variety of other "commonsense" reasoning tasks. the broad coverage diagnostic was manually curated by linguistics and nlp experts and is meant to assess broad psycholinguistic competencies of lms across multiple categories. for instance, the propositional structure category contains questions that exploit propositional logic operators; e.g., @xmath0." and @xmath1." the diagnostic covers four main categories of linguistic competencies: lexical semantics, predicateargument structure, logic, and knowledge and common sense. these categories are further divided into multiple sub-categories, each of which covers a specific and interesting phenomenon in language. the glue diagnostic thus aims to be a comprehensive test of linguistic reasoning skills, making it suitable for our present study. to evaluate our models, we selected a subset of the glue diagnostic questions that were a member of only one sub-category, to better isolate factors. in most cases, there were enough questions in a single sub-category that we could just drop all questions that belonged to multiple sub-categories, further details on this preprocessing can be found in appendix a. after performing preprocessing, we had 811 remaining diagnostic questions encompassing 20 sub-categories. each sub-category had at least 15 questions, and we selected 7 of the subcategories to use in our experiments: we selected these 7 sub-categories based on how much the average performance of the lms improved after pre-training and finetuning. a substantial performance improvement indicated the category was solvable by the models, and would therefore provide a meaningful comparsion to the human data. we gathered responses to the diagnostic from a wide array of tlms, including bert @xcite , roberta @xcite , t5 @xcite , albert @xcite , xlnet @xcite , electra @xcite , longformer @xcite , span-bert @xcite , deberta @xcite , and convbert @xcite . each of these models differs from the others along one or more factors, including underlying architecture, pre-training objective and data, or the general category the model belongs to (autoregressive, autoencoding, or sequence-to-sequence). for most of these models we used the transformers @xcite implementation, the exception being t5, which was implemented in pytorch lightning @xcite . we use lstm-based lms @xcite as a baseline, further details on the lms can be found in appendix a. we used the snli @xcite , mnli @xcite , and anli @xcite training and dev sets to finetune our models. we found that the amount of finetuning data had a significant impact on final diagnostic performance. therefore, to increase the variance in our results as much as possible we used the following training set partitions for all model configurations: both the train and dev sets were shuffled before every trial. we finetuned our models for between 5 to 10 epochs. we used the reported matthews correlation @xcite on the dev set during training to determine when the performance had saturated; when this correlation stopped consistently increasing for at least a few dev set evaluations we stopped training. we evaluated on the dev set every 15,000 steps. all the transformer's key hyperparameters were selected in a similar way to the study by @xcite . for all models, we used a learning rate of 1 * 10 - foot_4 and a max sequence length of 175. since running even a small grid search to optimize the hyperparameters of each model would have dramatically increased the number of trials, we instead chose to fix these hyperparameters to be similar to what was used in prior work (e.g. @xcite . we also found that nearly all models consistently achieved a matthews correlation of about 0.5 or higher on the dev set, and thus concluded that our hyperparameters were suitable. it is important to note that our goal in finetuning was not to completely optimize the model's performance on these nli datasets. rather, since the diagnostic is formatted as an nli task, we hoped that finetuning would help the models to learn what the output labels should be. 5 to evaluate these models, we experimented with four different training regimes: for bert, we experimented with both devlin et al.'s pre-trained models, and a bert model we trained from scratch. our bert model had an identical architecture to bert-base and was pre-trained on google's one billion words corpus @xcite . we used the same hyperparameters from the bert paper @xcite , using a learning rate of 4 * 10 -5 , a max sequence length of 128, a warmup ratio of 0.01, and a weight decay of 0.01. we used the transformers library to pre-train this model, and saved every end-of-epoch checkpoint. we pre-trained for 52 epochs and used every 10th checkpoint to gather diagnostic data separately. this allowed us to study the effect pre-training had on diagnostic performance. in summary, this process allowed us to vary the underlying architecture, the number of trainable parameters, and the amount of finetuning data used in each trial. this allowed us to treat each trained model as effectively being a different "individual" (and we will refer to them as such), which might have a radically different cognitive profile from its counterparts. for example, a roberta-base model that was pre-trained and finetuned on all 3 nli datasets might produce very different response patterns than a roberta-large model evaluated zeroshot. we used three tesla v100 gpus with 32gb of video memory each, as well as preemptable gpus on google colab, 6 to train all models. wherever possible, we used apex foot_6 to speed up training. ## human studies as our purpose in gathering this lm data was to evaluate it against human performance, we additionally ran a human study. to do this, we recruited workers on amazon mechanical turk (mturk foot_7 ) to complete our subset of glue diagnostic questions. while mturk makes conducting large-scale human studies convenient, there are also well-documented problems with participants not completing tasks in good faith @xcite @xcite . there are multiple techniques for filtering out bad-faith participants, such as the use of "attention check" questions, sometimes called "instructional manipulation checks" @xcite , which are designed so that a good-faith participant would be unlikely to get them incorrect. but this alone would not suffice for our purposes here, as we want a certain amount of low-scoring participants on some sub-categories, so that the population variances on sub-category items would better reflect their actual variances. 9 therefore, we designed a procedure for distinguishing bad-faith from low-performing participants. we first obtained attention checks from the chaosnli dataset @xcite , which gathered over 450,000 human annotations on questions from snli and mnli. since each question in chaosnli was annotated by 100 different workers, if inter-annotator agreement for a given question is high, we conclude that question is likely extremely easy to solve. these questions were also in the same format as the diagnostic questions, which made it less likely that workers would realize they were being given an attention check. we gathered 36 questions from chaosnli where the agreement for the correct label was at least 90%. the labels for this subset were perfectly balanced. these were enough questions to ensure that each phase of our trials used a unique set of attention check questions. the human studies were split up into 5 phases, and workers who did sufficiently well in a given phase were given a qualification to continue to the next phase: in each phase, questions were randomly ordered, except for attention checks which were spread evenly throughout the survey. we used qualtrics 10 to create the surveys for each hit and collect the responses. participants were first presented with instructions for the task and some examples, which were based on the instructions originally given to annotators for the mnli dataset. 11 the questions from each category were a randomly chosen subset of 15 questions tested on the lms for that category, balanced for each label. for each question, workers also had to provide a short justification statement on why they believed their answer was correct, which was used to help filter out bad faith participants. to validate the responses to our surveys, we developed the following authentication procedure: stage 1: look for duplicate ips or worker ids, indicating that the worker took the hit more than once. if there are any, reject the second and future hits, but keep the first submission. stage 2: if the worker's overall score was less than 40%, reject the hit. if their overall score was greater than 60%, accept the hit. for workers who scored between 40% and 60%, we still rejected the hit if they got less than 75% of the attention checks correct. stage 3: finally, examine the justifications of all workers not previously rejected. here we were looking for simple, but clear, reasons for why workers chose their answer. we included this step because we found in a pilot study that workers sometimes provided nonsensical justifications for their answers even when they did well on the survey, making it unclear whether they were truly paying attention. we checked that the justifications appeared relevant to the question (some workers seemed to paste random text from other websites into the justification), that they did not paste part of the question for their justification, that they did not use the same justification for every question, and that they did not use short nonsensical phrases for their justification (some workers simply wrote "good" or "nice" as their justification). this allowed us to keep some low-scoring participants who had put genuine effort into the task. manual inspection of the resulting responses suggested that workers who passed stage 3 consistently gave higher quality responses than those who did not. these workers gave more detailed justifications that clearly articulated their thought process, often citing specific details from the question. on the other hand, workers who failed to give good justifications also tended to perform quite poorly, gen-erally scoring at or below random chance, which further indicated that they were not actually paying attention. we, therefore, believe the use of justifications helped us gather higher-quality responses. further details on the human study can be found in appendix b. ## experimental results using the procedures described in §3 and §4, we gathered results from 27 human participants and 240 neural lms (183 transformer-based and 57 lstm-based). in addition to the lstms, we also include a true random baseline which simply guesses randomly on every question. in the following experiments, we use the human performance on each category as the basis for analyzing the performance of the artificial populations, specifically in terms of how well each artificial population's responses correlate with the human data. ## discussion our analysis has revealed some interesting patterns that would have been difficult to discern using traditional evaluation metrics. overall, tlms perform consistently better than either of our baselines in modeling human psychometric properties. however, this improvement is also not uniform across all psycholinguistic categories. in fact, we have found some regularities in this regard. for instance, tlms failed to achieve a strong correlation on morphological negation in all cases. this might be explained by two facts: there is little relative variance in the human responses in this sub-category, and the average accuracy of human participants was above 90%, as opposed to lm accuracy of 55%. this sub-category also tests for reasoning over negation, which prior studies found that transformers struggle with @xcite . this ability to analyze the specific kinds of reasoning transformers have become proficient in is a clear advantage psychometrics have over typical nlp evaluations. the nlp community is becoming increasingly aware of the need to construct more fine-grained evaluation benchmarks @xcite , and we believe our work complements these efforts nicely. of course, this study also has limitations. the number of human participants in our study was somewhat small compared to typical psychometrics studies, which makes it difficult to draw stronger conclusions. one of the main criticisms irt models draw is that they can require thousands of responses to get good estimates of the latent parameters @xcite . as stated earlier, practical limitations on population size is a common problem in psychometrics research, one which our present work hopes to alleviate somewhat. future work will need to repeat our experiments with much larger population sizes, and also take measures to ensure sufficient diversity in the study population (e.g., age, income, education level, english fluency, etc.). improvements in the computational efficiency of tlms is likely also necessary for our approach to be practical, as it is unlikely most pyschometricians have access to extensive gpu resources. one possible solution would be to identify a subset of tlms that preserves the psychometric properties of the entire population, which might allow us to achieve similar results with fewer models. furthermore, although we reported in detail on certain psychometrics measures where our method demonstrated promising results for tlms, it is worth reporting that certain other measures we examined did not appear to align well. for example, item-total correlations using human data did not appear to correlate with any lm data better than with the random baseline. likewise, our lms failed to predict average inter-item correlations between either random subsets of items or our diagnostic sub-categories. more work is needed to better understand why. finally, while our experiments have given us some insights into the validity and reliability of the diagnostic items, it is unclear whether our approach can allow us to measure their fairness. although it is an important property, fairness is somewhat more controversial than other psychometric properties, in part because there are multiple interpretations of what constitutes test bias @xcite . being able to probe the fairness of items would have interesting downstream applications. for instance, it might indicate whether a diagnostic gives an unfair advantage to certain types of classifiers. ## conclusion we believe our work offers a clear path forward for bridging psychometrics and ai. the use of psychometrics measures gives us a more nuanced understanding of the latent abilities of lms than singlevalued measures like accuracy or f 1 can provide. furthermore, the increasingly powerful ability of tlms to model human "commonsense" reasoning and knowledge suggests new ways to predict psychometrics properties of test items, reducing the need for costly human empirical data.