How Good are M odern LLM s in Generating Relevant and High-Quality Questions at Different Bloom’s Skill Levels for I ndian High School Social Science Curriculum? (2024)

Abstract: The creation of pedagogically effective questions is a challenge for teachers and requires significant time and meticulous planning, especially in resource-constrained economies. For example, in India, assessments for social science in high schools are characterized by rote memorization without regard to higher-order skill levels. Automated educational question generation (AEQG) using large language models (LLMs) has the potential to help teachers develop assessments at scale. However, it is important to evaluate the quality and relevance of these questions. In this study, we examine the ability of different LLMs (Falcon 40B, Llama2 70B, Palm 2, GPT 3.5, and GPT 4) to generate relevant and high-quality questions of different cognitive levels, as defined by Bloom’s taxonomy. We prompt each model with the same instructions and different contexts to generate 510 questions in the social science curriculum of a state educational board in India. Two human experts used a nine-item rubric to assess linguistic correctness, pedagogical relevance and quality, and adherence to Bloom’s skill levels. Our results showed that 91.56% of the LLM-generated questions were relevant and of high quality. This suggests that LLMs can generate relevant and high-quality questions at different cognitive levels, making them useful for creating assessments for scaling education in resource-constrained economies.

## introduction in recent years, large language models (llms) have seen significant advances. they undergo training on extensive text datasets sourced from the internet and are utilized for a variety of natural language processing tasks. the introduction of ope-nai's chatgpt and google's bard has made llms more accessible to a wider audience, enabling individuals without expertise in natural language processing (nlp) to leverage them for their everyday needs. these models are characterized by their substantial size and their ability to comprehend and produce intricate text. through instruction finetuning, language models are calibrated to adhere to user directives @xcite . in contrast to conventional language models, these llms possess zero-shot capabilities, allowing them to handle various tasks without specific training by simply interpreting the given instructions @xcite . the educational applications of llms are varied and promising, covering personalized content generation, assessments, and feedback @xcite . according to world bank data, the teacher-pupil ratio in india's high schools is 1:29 foot_0 , compared to middle and high-income countries with an average of 1:18 and 1:13, respectively. this increases the workload on teachers and the quality of the instruction and assessment decreases. in india, subjects such as history are taught and evaluated, focusing on rote memorization @xcite with minimal emphasis on higher-order thinking skills or inquiry. inquiry-based learning with high-quality questions fosters deep engagement and real-world connections for learners @xcite . assessments aligned with bloom's taxonomy levels @xcite , as detailed in although llms are capable of natural language generation (nlg) tasks, their output can have errors and inconsistencies for specific contexts. these models are also prone to hallucinations @xcite . these issues directly impact the quality of educational questions generated, which can vary significantly across llms. for this reason, evaluating the quality of these questions is important. despite the existence of automated techniques focusing on readability and linguistic aspects, these methods do not address pedagogical aspects and question appropriateness for the given context @xcite . therefore, expert evaluation remains essential to guarantee the quality of llm-generated questions. in this study, we followed a zero-shot prompting approach for question generation. we prompted llms to generate questions at different cognitive levels, as defined in bloom's taxonomy, on topics covering events of the indian independence struggle from 1857 to 1947. using five different llms, we generated 510 questions in total. two subject matter experts evaluated the generated questions based on a nine-item rubric to consider both the linguistic and pedagogical aspects of the questions @xcite . this work investigates the following research questions. (i) can modern llms generate relevant and high-quality educational questions of different cognitive levels and follow the instructions provided in the prompt?; (ii) which llm performs the best in question generation? our experiments and evaluations demonstrate that the questions generated by llms are relevant and of good quality. these llms can be used for aeqg with minimal effort of the educator. our dataset 'historyq' foot_1 containing 510 questions eval-uated by two experts and annotated with bloom's taxonomy levels will be made available for research in the development and evaluation of aeqg systems. ## related work traditional automated question generation (aqg) systems mainly relied on question-answering datasets before the widespread adoption of llms. the primary reading comprehension datasets used for question generation tasks included squad @xcite , squad 2.0 @xcite and nq @xcite . one of the crowd-sourced educational datasets used for question generation tasks is sciq @xcite . learningq @xcite and @xcite are the other two popular datasets available for aeqg. the lack of availability of these datasets for all subjects and the human expert labor associated with creating highquality datasets restricted the ability to develop effective aqg systems @xcite . with the advent of large transformer-based pre-trained large language models, nlg tasks in recent years have improved rapidly @xcite . pretrained and fine-tuned models such as the text-to-text transfer transformer (t5) and gpt3 were used for question generation @xcite . leaf @xcite is a question generation developed using a pre-trained t5 model. a pretrained t5 model (eduqg) was developed in educational text to improve the quality of the generated question @xcite . most aeqg systems are generic with a focus on reading comprehension or science and mathematics. aeqg research for social sciences is minimal @xcite . subjects like science and mathematics tend to seek precise, quantifiable, and objective answers. but for subjects like social sciences, the questions can be more subjective, often do not have a single correct answer, and can be interpreted differently by different people. many aqg systems, built by fine-tuning llms on specific datasets such as the ones mentioned above, often generate questions that focus on lowerorder cognitive skills or simply retrieve answers directly from the context information provided @xcite . most of the questions in eduqg @xcite are within the first three levels of bloom's taxonomy. these questions do not assess students' higher-order thinking abilities. bloom's taxonomy guides educators in generating learning objectives and questions to teach and test different cognitive skills. a recent work @xcite uses gpt4 to create course content based on bloom's taxonomy. although automated metrics exist to evaluate machine-generated questions, they primarily analyze linguistic aspects. in the case of educational question generation, pedagogical elements play a crucial role. expert evaluation is necessary to understand the pedagogical aspects of machinegenerated questions @xcite . such evaluations are also used in student-generated questions @xcite . ## results and analysis the percentage agreements and cohen's κ values obtained between the two human evaluators for the nine-item rubric are given in ## conclusion we found that 91.56% of the questions generated by different llms are relevant and of high quality. this indicates that llms can be used for aeqg with minimal effort of the educator. however, the performance varies between different llms. gpt 3.5 and gpt 4 generated the highest proportion of relevant and high-quality questions. in the metric of adherence to bloom's level, gpt 4 outperformed the other models, followed by palm 2. in contrast, the open source llms, falcon 40b and llama 2 70b, performed poorly on all metrics, except adherence to cefr levels. this could be due to the large size of these proprietary models, which results in their ability to capture and represent complex patterns in the text data. another interesting observation in the study was the inability of most models to generate high-quality questions at the 'apply' and 'create' levels of bloom's taxonomy. gpt 3.5 and gpt 4 showed comparable performance in all criteria. surprisingly, gpt 4 and gpt 3.5 had poor alignment with the cefr level requested in the prompt. these models produced complex texts compared to other models. our research suggests that educators can lever-age palm 2, gpt 3.5, and gpt 4 to create relevant, high-quality questions of different cognitive levels defined by bloom's taxonomy for scaling social science research in india. the llms must be prompted with the context in english obtained from the relevant curriculum. this approach considerably reduces the workload on teachers, especially in an under-resourced school setting where the teacher-pupil ratio is low. in addition, students can create practice tests for themselves and identify learning gaps. expert-evaluated 'historyq' could serve as a training and validation dataset for research involving the development and evaluation of aeqg models with a focus on higher-order cognitive skills.