Evaluating Reading Comprehension Exercises Generated by LLM s: A Showcase of C hat GPT in Education Applications (2023)

Abstract: The recent advancement of pre-trained Large Language Models (LLMs), such as OpenAI’s ChatGPT, has led to transformative changes across fields. For example, developing intelligent systems in the educational sector that leverage the linguistic capabilities of LLMs demonstrates a visible potential. Though researchers have recently explored how ChatGPT could possibly assist in student learning, few studies have applied these techniques to real-world classroom settings involving teachers and students. In this study, we implement a reading comprehension exercise generation system that provides high-quality and personalized reading materials for middle school English learners in China. Extensive evaluations of the generated reading passages and corresponding exercise questions, conducted both automatically and manually, demonstrate that the system-generated materials are suitable for students and even surpass the quality of existing human-written ones. By incorporating first-hand feedback and suggestions from experienced educators, this study serves as a meaningful pioneering application of ChatGPT, shedding light on the future design and implementation of LLM-based systems in the educational context.

## introduction reading comprehension is a vital skill that english learners need to develop and master. chinese middle school students, for instance, are required to do numerous english practices, including reading at least 150,000 words of supplemental materials to enhance their reading abilities, as mandated by the english curriculum standards. through interviews with experienced english teachers in beijing, we discovered a challenge faced by both educators and students: the repeated use of outdated reading materials, with only minor modifications made, if any. for instance, grade 8 students are likely to practice the same exercises used by their predecessors in the previous academic year (currently grade 9 students). english teachers believe that offering up-to-date, engaging reading exercises tailored to each student's capabilities and interests can spark their enthusiasm for learning and ultimately boost their english proficiency. however, obtaining a large collection of diverse, customized, high-quality english reading exercises proves to be a non-trivial task. there is an abundance of articles in newspapers, magazines, textbooks, and children's books from english-speaking countries that could serve as potential sources of reading materials for middle school students. nonetheless, adjustments and rewrites are typically necessary due to variations in topic, length, and difficulty level. moreover, even for veteran teachers, crafting appropriate exercise questions based on textual materials is still not easy. pre-trained large language models (llms) have been proposed by researchers as a means to address this labor-intensive and unscalable issue @xcite . reading comprehension exercises typically consist of two components: a lengthy, coherent passage and several multiple-choice questions that align with its content. to generate such exercises, it is essential for llms to possess an advanced understanding and inference ability of human language. while the generation of long texts (such as stories, news articles, and poems) @xcite and question-andanswer (q&a) pairs @xcite have been extensively studied, existing task-specific models fall short of meeting our needs. for instance, the generated content still remains distinguishable from human-written text, and the level of personalization for different learners is inadequate @xcite , making these models unsuitable for direct application in educational settings. recently, openai released chatgpt 1 , a versatile and interactive chatbot that outperforms state-of-the-art models in various nlp tasks, even in zero-shot or few-shot scenarios. this powerful llm presents numerous opportunities for education, including the creation of reading materials and customized practice questions. in this study, we attempt to develop a system for middle school teachers and students that leverages chatgpt to generate reading comprehension exercises. guided by carefully crafted prompts, chatgpt can produce personalized reading passages and multiple-choice questions of high quality. to assess the generated exercises and the overall system, human evaluators (comprising students, teachers, and native speakers) conducted an extensive analysis, determining that the system holds promise for implementation in middle schools and has the potential to make a significant educational impact. in summary, this study makes threefold contributions: llm and controllable text generation with the emergence of transformers (vaswani et al., 2017), llms have been performing remarkably well and showing considerable progress across a 2 the codes for our system is available at https://github.com/xiaochr/ reading-exercise-generation-system . chatgpt is developed on the foundation of gpt-3.5 or gpt-4 architectures, with the inclusion of additional human-directed instructions for enhanced performance. it possesses robust in-context learning capabilities, enabling it to interpret requirements specified in input prompts without the need for additional information (zero-shot learning), or by utilizing a minimal number of provided examples (few-shot learning). even without massive domain knowledge, chatgpt is able to follow human instructions and generate text of higher quality. for instance, to generate a 200-word reading passage on the topic of school life, one simply needs to specify the subject and length requirements in the prompt to chatgpt. ## evaluation in this section, we conduct extensive evaluations of our reading exercise generation system, which are visually depicted in figure 3 . for reading passage quality evaluation, we randomly select 30 human-written reading passages from dataset 2 (the reading exercises from textbooks), which are paired with an additional 60 passages: 30 produced by chatgpt and 30 by the baseline model. this mixture of passages is shuffled and compiled into what we refer to as the reading passages example set 1. we utilize both automatic evaluation metrics and human assessments (section 4.1) in order to comprehensively evaluate these passages. to further verify the high quality of chatgpt passages, a series of one-to-one comparisons is conducted between passages produced by language models and their human-written counterparts. we select 10 human-written reading comprehension passages, distinct from the passages in the reading passages example set 1, and summarize the topic of each one. we then use these topics as guiding constraints to direct conditional text generation with both the gpt-2 + pplm baseline and chat-gpt (zero-shot), resulting in passages mirroring the topics of the original human-written examples. additionally, a one-shot variant of chatgpt, using the human-written passage as a reference, is utilized to generate a third group of passages. to sum up, the reading passages example set 2 encompasses 10 original human-written passages, augmented with 30 generated passages that align with the same topics. moving to the evaluation of exercise question quality, we select 10 exercises containing reading passages and their associated questions from the textbook to serve as benchmarks. a new set of multiple-choice questions is generated based on the human-written passages using our system. thus, these 10 reading passages and their corresponding 20 sets of questions form the exercise questions example set, which is thoroughly evaluated in section 4.2. for the overall evaluation of our system (section 4.3), we invite middle school educators, the intended users of our system, to utilize it first-hand. we request their insightful feedback and suggestions, furthering our goal of consistent improvement and customization to user needs. ## conclusion in this study, we attempted to develop an educational system for teachers and english learners in chinese middle schools that leverages the capabilities of llms to generate reading comprehension exercises. extensive evaluations were conducted among various groups of representative human evaluators, and the high quality of the generated reading exercises was widely acknowledged. experienced english teachers also provided extremely positive feedback on the system, indicating its potential for widespread use in real-world education. our system is among the first applications of chat-gpt in educational contexts, and the valuable feedback and findings are likely to inspire future researchers and educators in integrating ai technology into education.