Generating and Evaluating Tests for K-12 Students with Language Model Simulations: A Case Study on Sentence Reading Efficiency (2023)

Abstract: Developing an educational test can be expensive and time-consuming, as each item must be written by experts and then evaluated by collecting hundreds of student responses. Moreover, many tests require multiple distinct sets of questions administered throughout the school year to closely monitor students’ progress, known as parallel tests. In this study, we focus on tests of silent sentence reading efficiency, used to assess students’ reading ability over time. To generate high-quality parallel tests, we propose to fine-tune large language models (LLMs) to simulate how previous students would have responded to unseen items. With these simulated responses, we can estimate each item’s difficulty and ambiguity. We first use GPT-4 to generate new test items following a list of expert-developed rules and then apply a fine-tuned LLM to filter the items based on criteria from psychological measurements. We also propose an optimal-transport-inspired technique for generating parallel tests and show the generated tests closely correspond to the original test’s difficulty and reliability based on crowdworker responses. Our evaluation of a generated test with 234 students from grades 2 to 8 produces test scores highly correlated (r=0.93) to those of a standard test form written by human experts and evaluated across thousands of K-12 students.

## introduction developing an educational test can be resourceintensive and time-consuming, as each item must be written by experts and then evaluated by collecting hundreds of student responses. this process of evaluating items in terms of properties like difficulty and their ability to discriminate between student abilities, known in psychometrics as item calibration, is fundamental to test development. furthermore, schools often require the creation of multiple, distinct test forms (a collection of comparison of student scores on a test form of item-response-simulator-filtered language-model-generated items and a test form of human-generated items for a sentence reading efficiency test. the human-generated test form was designed by experts and calibrated across thousands of k-12 students; the ai test form was generated by gpt-4 and filtered by our proposed item-response simulator. in response to the challenge of item development, many works have proposed leveraging language models to generate items for educational assessments and instruction @xcite @xcite @xcite ; we first train the simulator to predict a student's response to a question conditioned on their previous responses. then, we generate new items and insert them into past tests for simulation. finally, we perform simulation and match the new items to an existing test. note the simulation and generation models differ. @xcite . however, estimating the relevance, quality, and difficulty of these generated items and test forms is an open challenge that must be carefully addressed for both psychometric and nlp communities as well as stakeholders in our education system. we propose an itemresponse simulator, training llms with student response data and simulating the responses of past participants on new items in order to calibrate them. as a case study, we apply this method to develop and calibrate new items for a silent sentence reading efficiency (sre) task @xcite , which we elaborate on in section 2. we then propose an optimal-transport-inspired technique for generating parallel test forms with the new items by referring to a well-calibrated human-written test form. in doing so, we make the following contributions: overall, our work advances the measurement of reading efficiency by introducing scalable methods to generate test items. we address the novel task of parallel test generation without collected responses and approach parallel test construction with simulated responses as relaxed optimal transport. children can be sad . true ( response time : slow ) you sleep on a log . false ( response time : slow ) [...] sweaters can be made of coal . false ( response time : very slow ) you can feed a hamster . false ( response time : very slow ) you can fill a balloon with air . ## silent sentence reading efficiency how can we measure students' reading abilities? oral reading fluency (orf) is a widely used measure in research and practice, measuring words read correctly per minute @xcite . recently, researchers have examined the relationship between oral and silent reading fluency @xcite and shifted focus to silent reading, as it is the most common form of reading for k-12 students. however, given the lack of observable verbal responses, it is a more challenging measurement task. our task, silent sentence reading efficiency (sre), is an online, selfadministrated measure assessing the speed with which a student can read simple english sentences @xcite . modeled after other standardized silent reading fluency measures such as test of silent reading efficiency and comprehension (tosrec) @xcite , sre requires students to read sentences and respond whether each is true or false, which we refer to as the item truth value (see fig. 3 ). a student responds to as many sentences as they can within three minutes, and the final score is the number of correctly answered sentences minus incorrectly answered ones. unlike tosrec, sre targets reading efficiency with less emphasis on reading comprehension. this complicates item development, requiring syntactically and semantically simple sentences with only vocabulary that all schoolage students should be able to read and understand. we focus on sre for three reasons. first, from an educational perspective, there is a high demand from schools to closely monitor students' reading development. thus, it is important to develop diverse parallel test forms with identical difficulty and reliability. manually authoring thousands of items and collecting the data to match test forms is extremely time-consuming and resource-intensive. second, from a psychological measurement standpoint, sre is a task where both accuracy and response time are crucial in determining ability. this characteristic does not align well with classic psychometric models, such as item response theory @xcite , which focus on accuracy. third, of particular relevance to the nlp community, measuring sentence-level reading fluency rather than comprehension poses a novel challenge, as traditional readability metrics and linguistic features fail to predict fluency accurately. ## training dataset we collect student data from 1st grade to high school by working closely with more than 30 diverse school partners in the united states for two school years (see fig. 5 for breakdown by grade level). all data is collected under irb guidelines. as part of the sre validation study @xcite , each student completed two 3-minute ## conclusion our study presents a new framework to generate test items and select them according to their effectiveness to assess students by leveraging large language models and previous student responses. we use silent sentence reading efficiency assessment, a task with implications for the millions of students who take such tests annually, to illustrate the utility of this framework by generating new test items, predicting the difficulty and ambiguity of unseen items, and creating multiple parallel test forms that have comparable quality and appropriateness to human-generated test forms as well as remain appropriately difficult for students. we ultimately validate our approach with a wide range of evaluations, including more traditional machine learning metrics, crowdworker evaluations, as well as a real-world deployment in k-12 classrooms.