Planning First, Question Second: An LLM -Guided Method for Controllable Question Generation (2024)

Abstract: In the field of education, for better assessment of students’ abilities, generated questions often need to meet experts’ requirements, indicating the need for controllable question generation (CQG). However, current CQG methods mainly focus on difficulty control, neglecting the control of question content and assessed abilities, which are also crucial in educational QG. In this paper, we propose an LLM-guided method PFQS (for Planning First, Question Second), which utilizes Llama 2 to generate an answer plan and then generates questions based on it. The plan not only includes candidate answers but also integrates LLM’s understanding and multiple requirements, which make question generation simple and controllable. We evaluate our approach on the FairytaleQA dataset, a well-structured QA dataset derived from child-friendly storybooks. In the dataset, the attribute label represents content control, while the local_or_sum and ex_or_im labels denote difficulty control. Experimental results demonstrate that our approach outperforms previous state-of-the-art results and achieves better consistency with requirements compared to prompt-based method. Further application of our method to Llama 2 and Mistral also leads to improved requirement consistency in a zero-shot setting.

## introduction educational studies over the years have demonstrated that asking questions foster reading comprehension skills, critical thinking skills and writing skills @xcite @xcite . in order to maximize their educational value, questions posed in the teaching and learning activity should be reasonably designed in terms of content and diversified by the levels of questioning @xcite . recent advancements in natural language processing have in certain educational settings, such as exams, it is crucial that automatically generated questions derived from a given context are not only pertinent to the context, but also capable of assessing students' diverse abilities @xcite and can be controlled in terms of difficulty @xcite . however, meeting these practical requirements presents challenges for educational qg. for instance, given the sample in to alleviate the above limitations, we propose an llm-guided method pfqs (planning first, question second) that enhances the controllability of generated questions in terms of content and difficulty simultaneously. our approach consists of two steps. firstly, we utilize llama 2 @xcite to generate an answer plan, a concept proposed in this paper, containing various information on candidate answers. in addition to candidate answers and answer-containing sentences extracted by llama 2, the plan in our method contains various control information to ensure controllable question generation. the inclusion of answer-containing sentences is inspired by @xcite . in terms of control information, we prioritize content control followed by difficulty control and express control information in various forms, as content control is typically linked to candidate answer selection, while difficulty control is based on these answers. secondly, we combine designed prompt, generated plan and given context together and feed them into qg models to generate qa pairs. furthermore, we design several prompts to enable large language models (llms) to assist in evaluating whether the content and difficulty of the generated questions meet predefined requirements. as the example shown in ## method our question generation method comprises two main steps: plan generation and question generation. in the first step, we use an instructiontuned llm, specifically llama 2 @xcite , to generate an answer plan based on the given context and predefined requirements (i.e. expert-annotated labels in the dataset, where the attribute label represents content control, and the local_or_sum and ex_or_im labels denote difficulty control). the plan typically includes several points that serve as candidate answers or answer-containing sentences extracted by the llm, incorporating content and difficulty control information. in the second step, the generated plan, along with the context and the designed prompt, is fed into qg models to generate question-answer pairs. the overall architecture of our method is illustrated in figure 1 . ## conclusion in this paper, we propose pfqs, a novel qg method with llm-guided answer planning at its core. unlike existing qg methods, pfqs utilizes llm to generate an answer plan for candidate answer selection, rather than training another model. the generated plan extends the standard prompt, typically including candidate answers, llm's understanding, and control information, all of which contribute to question generation. experimental results indicate that pfqs achieves outstanding performance and integrating the plan enhances the diversity and label consistency of the qg model, in comparison to using a standard prompt. our approach opens up new possibilities for incorporating more information into prompts with the assistance of llm, instead of training another model when faced with detailed or complex requirements in controllable question generation tasks. in the future, with the emergence of more and more educational question generation datasets with expert-annotated labels, the planning first, question second idea in our qg method may gradually show its application value.