Generating Better Items for Cognitive Assessments Using Large Language Models (2023)

Abstract: Writing high-quality test questions (items) is critical to building educational measures but has traditionally also been a time-consuming process. One promising avenue for alleviating this is automated item generation, whereby methods from artificial intelligence (AI) are used to generate new items with minimal human intervention. Researchers have explored using large language models (LLMs) to generate new items with equivalent psychometric properties to human-written ones. But can LLMs generate items with improved psychometric properties, even when existing items have poor validity evidence? We investigate this using items from a natural language inference (NLI) dataset. We develop a novel prompting strategy based on selecting items with both the best and worst properties to use in the prompt and use GPT-3 to generate new NLI items. We find that the GPT-3 items show improved psychometric properties in many cases, whilst also possessing good content, convergent and discriminant validity evidence. Collectively, our results demonstrate the potential of employing LLMs to ease the item development process and suggest that the careful use of prompting may allow for iterative improvement of item quality.

## introduction ai is having increasingly profound impacts on educational and psychological measurement @xcite . technologies built on ai and machine learning, including educational data mining @xcite , intelligent tutoring systems @xcite , deep item response theory @xcite , and deep knowledge tracing @xcite , among others (asfahani, 2022, inter-alia) are transforming educational and psychological measurement, and this trend seems likely to continue. one promising educational application of large language models (llms) is for the automatic gen-eration of test items (aig). writing high-quality test items is critical to building effective educational assessments, but has also traditionally been a time-consuming process, as items must be developed by experts and undergo numerous rounds of review @xcite . there has been significant research interest in using aig to create highquality items with minimal intervention to speed up the test development process @xcite . prior work has demonstrated that llms can generate items with at least face validity (i.e, they appear valid based on item content) for both noncognitive @xcite and cognitive @xcite constructs. careful psychometric analysis of items generated from such models has also revealed that they are just as valid and reliable as their human written counterparts @xcite . although promising, this research has largely focused on generating items for constructs that have been well-studied, using items already known to have strong validity evidence. suppose an educator wishes to develop a test for a new construct where existing items may have only undergone pretesting. or suppose the educator wishes to use a new type of item for a well-established domain (e.g, a test of algebraic reasoning that uses a novel item format). in either case, the items will likely have limited validity evidence, and much time would need to be spent revising the items to improve their psychometric properties before they can be used. in this work, we ask: can llms be used to generate valid and reliable items even in these scenarios where existing items have only limited validity evidence? if so, llm-based aig could be used to iteratively improve the psychometric properties of items, explore the underlying construct space, and shed light on what makes a good item. we explore this using gpt-3 @xcite ) and focus on generating items that test for natural language inference (nli) @xcite . nli is an important cognitive construct in nlp research which, to our knowledge, has only undergone limited psychometric analysis in human participants @xcite . we develop a novel prompting strategy that uses the psychometric properties of items, calculated using prior human responses, to select the most informative examples to send to the model to maximize the quality of the generated examples. our main contributions are as follows: ## generation of test items the general language understanding evaluation (glue) @xcite is a benchmark designed to measure broad linguistic constructs in llms. included in glue is a diagnostic set, ax, 1 which is meant to be a challenge set for diagnosing faults in llms. items on ax are framed as nli: given a premise (p) and hypothesis (h), a model must determine whether p entails, contradicts, or is neutral with respect to h @xcite . items were written by nlp experts, inspired by categories taken from the fra-cas suite @xcite , and are based on sentences from a variety of artificial and naturalistic contexts. @xcite reported strong interrater reliability when labeling a random sample of ax items, and ax has been used successfully to evaluate many new llms @xcite @xcite , which suggests the diagnostic has good predictive validity. furthermore, laverghetta jr. et al. (2021) previously ran human studies on a subset of items from ax, targeting those testing for propositional structure (ps), quantifiers (q), morphological negation (mn), and lexical entailment (le). our goal is to use llms to generate new items for ax, such that the psychometric properties of both the items and the test as a whole are improved. formally, given an llm m and a prompt p that contains one or more items that have a psychometric property θ, we seek to sample new items i from m that lead to an improvement in θ: where i and p are assumed to test for the same construct (e.g., nli). prior work has demonstrated that when llms are given existing items as prompts, they can generate new items that match the construct measured by those items @xcite . we build on this approach by designing prompts to instruct llms to generate new items for a particular construct, that possess a desired psychometric property. figure 1 shows one of the prompts we developed. the model is instructed to generate only items that match the target property, and we use items from only one category at a time. we use item discrimination as the target property in our experiments. discrimination refers to the ability of an item to separate high from low-ability test takers @xcite and is computed using the item-to-total correlation (the correlation between the responses to a single item and total scores across all items). an item that is i need to generate new nli items for a given trait. here are some examples: ### trait: high discrimination items (3): [items] ### trait: low discrimination items (3): [items] ### trait: high discrimination new items (5): highly discriminating will predict total scores and thus should be maximized. our use of discrimination was based on preliminary analysis of the data @xcite , which indicated that at least one item in every category had negative discrimination. in general, items with negative discrimination are regarded as problematic and possibly erroneous, and should not be included in cognitive assessments @xcite , which makes improving the discrimination of the ax items a natural optimization target. we use existing human written items as examples of the desired property in the prompt, selecting the top k items with the highest discrimination as "high discrimination" and the bottom k items with the lowest discrimination as "low discrimination". 3 we set @xmath0, as we found larger values caused the difference in discrimination to become negligible. by providing examples of both good and bad items, we hope to teach the model general characteristics of high-quality items. 4 we use gpt-3 @xcite as our item generator, given its strong performance across many nlp tasks, the presence of an easy-to-use and inexpensive api, and the success prior work has had in using gpt-3 to generate non-cognitive @xcite and nli @xcite items. we set temperature to 1 for all experiments, to encourage diversity in the generated items, and use a maximum token limit of 300. we explore the effect of varying other key hyperparameters: following best practices in scale development @xcite we conduct a content review on the generated items. four ph.d. students with prior publications in nlp, nli, or psychometric ai were asked to rate the quality of the gpt-3 items. we ask our annotators to rate the relevance of the items for measuring the category, the clarity of the items (in terms of whether they have spelling or grammatical errors), whether the items have potentially harmful content, and their certainty in their annotations. before beginning the study, we gave annotators detailed instructions they were asked to review in advance, including information about the ax categories, how to answer each of the ratings, and example ratings. we instructed annotators to rate items as "completely irrelevant" if either the label was incorrect or the item did not match the target category. we followed standard practices in nli research for determining what the correct label should be @xcite , which all our annotators were informed of. in particular, annotators always assumed p and h referred to the same event or situation @xcite . for determining category membership, we follow the definitions of each ax category provided by @xcite , and developed a simple code book for determining this. the majority of the annotations were done synchronously in a four-hour annotation session. per recommended practices for content analysis, each item was rated by every annotator @xcite . annotators were encouraged to discuss items with each other and come to an agreement on what ratings should be used. further details on the content review, including an example of the annotation interface, can be found in appendix a. for a generated item to pass the content review, we determined that all annotators must rate the item as very clear, either relevant or very relevant, that the item contained no harmful content, and that annotators were either sure or very sure of their predictions. of the 400 items, 92 met these criteria across all categories, with at least 15 in every category passing. we sampled 15 at random from each category, balanced for the label, to obtain the gpt-3 generated items. in total, 60 items were sampled. ## experiments we determined in section 3 that gpt-3 can generate ax items that possess at least face validity evidence. but are these items really more valid and reliable measures of basic linguistic reasoning, given that we designed our prompts to induce this? to study this, we recruited human participants on amazon mechanical turk foot_5 to complete both the gpt-3 items and the original human-written items. we ultimately gathered data from 18 participants and base the following analysis on this sample. broadly, our goal is to compare the psychometric properties of the gpt-3 written items to the human-written items, focusing specifically on item difficulty, item discrimination, reliability (assessed using internal consistency), and convergent and discriminant validity. these are all important properties to analyze when establishing the validity and reliability of a new assessment @xcite , and when assessed using a measurement framework known as classical test theory (ctt), can be computed using small sample sizes. ctt essentially posits that an individual's true proficiency on a cognitive task (their true score) can be decomposed into an observed (actual) score they obtain and an error term that represents the measurement error @xcite . note that this error is assumed to be random, and not systematic. methods from ctt for assessing both validity and reliability are hence based on analysis of observed scores, and correlations between observed scores, where the observed scores are simply accuracy on the task: observed @xmath1, 2013), they typically rely on latent variable modeling and require much larger sample sizes. furthermore, in practice, establishing validity and reliability under ctt is often a first step in validating new assessments @xcite , which we believe justifies our focus on ctt in the present study. we begin by comparing mean item difficulties (figure 2 ) and mean item discriminations (figure 3 ) for both human and gpt-3 written items. difficulty is based on the participants' observed scores, and is equivalent to accuracy. classical psychometrics dictates that items should have difficulties at approximately the midpoint between chance and perfect scores @xcite , which in our case is roughly 70%. we again use item-to-total correlation to measure discrimination, and recall that item discrimination should be positive, with high values indicating better discrimination. we find that gpt-3 items are consistently closer to the optimal difficulty level than human-written items. gpt-3 items are also more discriminating than human-written ones, though a notable exception is for le, where the gpt-3 items are noticeably less discriminating. as le tests for all forms of lexical entailment, and is a much more broadly scoped construct than the others, lower discrimination is expected @xcite , though this does not fully explain the rather sizeable drop. ## discussion and conclusion collectively, our results demonstrate that llms can generate items with superior validity evidence, even for constructs that have undergone limited psychometric analysis. gpt-3 items were found to have better discrimination and reliability, while maintaining strong convergent, discriminant, and content validity. lid, while confirmed to be present in both item types, appeared no worse and perhaps slightly better in gpt-3 items. these positive results, while clearly present for ps and q, were less clear for mn and le, and validity evidence as a whole appeared strongest for the categories testing the most narrowly scoped constructs. figure 7 : distribution of annotator relevance scores (checking that the item both has a correct label and matches the category) for both gpt-3 and gpt-4 items, on items from the ps category. a lower percentage of items marked as "completely irrelevant" indicates stronger evidence of the content validity of items generated using that model. though promising, our results come with limitations that should be addressed in future work. the small sample size we collected makes it difficult to assess the generalizability of our findings. this also prevented us from running any analysis of internal structure or differential item functioning (dif) using methods from factor analysis or item response theory, as these models require large sample sizes @xcite . as items generated by gpt-3 should contain no dif and have similar factor structures as items written by humans, these are important analyses to explore in future work. we also did not examine the diversity of the generated items, in other words, how thoroughly the model explored the construct space. it is a well-known problem in psychometrics that having too many similarly worded items can inflate the reliability and reduce the validity of a measure @xcite , and our results may have been susceptible to this. a related problem is ensuring that the distribution of labels in the generated items remains balanced, and while we took steps to account for this, we did find that the distribution of gpt-3 items was somewhat unbalanced. for example, there were far fewer neutral items than either entailment or contradiction. improving the prompt design to account for diversity and other psychometric properties simultaneously is a fruitful direction for future work. our experiment with gpt-4, while disappointing, was also quite limited and should be expanded upon. we deliberately kept the prompt design as similar as possible between the two models, to avoid possible confounds. making effective use of the system query and changing the structure of the prompts to suit a conversational style could lead to much better results, however. finally, although we believe nli is a good task to use for initial experimentation, we also acknowledge that it is significantly different from the tasks of interest in education (e.g., question answering), and future work should explore our approach on tasks with stronger educational applications. llms have the potential to greatly ease the burden of scale development, and transform educational and psychological measurement. our results contribute to the growing field of llm-based automated item generation, and demonstrate the potential these methods have for generating valid and reliable items at a scale that would have previously been impossible. further research, combining our approach with more advanced prompting strategies, or zero-shot parameter estimation, could conceivably lead to a system that generates high-quality items in a fully autonomous fashion, which would transform the practice of writing and validating test items.