SGSH : Stimulate Large Language Models with Skeleton Heuristics for Knowledge Base Question Generation (2024)

Abstract: Knowledge base question generation (KBQG) aims to generate natural language questions from a set of triplet facts extracted from KB. Existing methods have significantly boosted the performance of KBQG via pre-trained language models (PLMs) thanks to the richly endowed semantic knowledge. With the advance of pre-training techniques, large language models (LLMs) (e.g., GPT-3.5) undoubtedly possess much more semantic knowledge. Therefore, how to effectively organize and exploit the abundant knowledge for KBQG becomes the focus of our study. In this work, we propose SGSH — a simple and effective framework to Stimulate GPT-3.5 with Skeleton Heuristics to enhance KBQG. The framework incorporates “skeleton heuristics”, which provides more fine-grained guidance associated with each input to stimulate LLMs to generate optimal questions, encompassing essential elements like the question phrase and the auxiliary verb.More specifically, we devise an automatic data construction strategy leveraging ChatGPT to construct a skeleton training dataset, based on which we employ a soft prompting approach to train a BART model dedicated to generating the skeleton associated with each input.Subsequently, skeleton heuristics are encoded into the prompt to incentivize GPT-3.5 to generate desired questions. Extensive experiments demonstrate that SGSH derives the new state-of-the-art performance on the KBQG tasks.

## introduction knowledge base question generation (kbqg) has attracted a lot of attention owing to its wide range of applications in academia and industry @xcite . on the one hand, kbqg can augment training data for question answering (qa) to improve the performance of qa models @xcite ; the methods include the state-of-the-art plm-based method dsm (yellow), text-davinci-003 (green), and text-davinci-003 with skeleton heuristics (blue). @xcite . on the other hand, kbqg empowers machines to actively ask questions in conversations with humans @xcite . the quality of kbqg has been significantly improved, largely attributable to the success of pretrained language models (plms) like bart @xcite and t5 @xcite . a noteworthy example is dsm @xcite , which introduces a meta-learner based on the bart for kbqg, effectively capturing diverse semantic information within a kb. moreover, auto-qgs @xcite designs an auto-prompt approach upon the bart, achieving the lowresource kbqg. current plms, pre-trained on comprehensive corpora, come equipped with rich semantic knowledge, facilitating significant performance improvements in the downstream kbqg task upon fine-tuning. recently, large language models (llms) such as instructgpt @xcite and chatgpt 2 , have exhibited impressive capabilities in a variety of tasks @xcite . however, the vast amount of generalized knowledge poses a challenge in extracting pertinent information for the kbqg task, making llms fall short of the expected performance on kbqg. as demonstrated in figure 1 , the plm-based cutting-edge approach dsm outperforms the direct application of the llm, davinci003. in view of this, our focus is on how to trigger llms to effectively utilize their knowledge to improve the quality of kbqg, which is an under-explored problem in the community of natural language processing. inspired by the way humans learn a language, which typically involves acquiring grammatical knowledge before progressing to reading and writing, we summarize the grammatical elements to guide the desired question generation, instead of directly applying an llm. in this work, the grammatical elements include the question word phrase and the auxiliary verb, which we call "skeleton". through a pilot study, we observe that prompts coupled with skeleton heuristics can boost the performance of davinci003 on the kbqg task (cf. the comparison between davinci003 and davinci003+sh in figure 1 ). in effect, the skeleton heuristics can be viewed as the fine-grained guidance to excavate task-specific knowledge from the llms, thereby stimulating the llms to generate more accurate questions. motivated by the above insights, we propose sgsh -a simple and effective framework to stimulate gpt-3.5 foot_1 with skeleton heuristics for kbqg, which contains two modules, i.e., a skeleton generator and a black-box llm (e.g., gpt-3.5). figure 2 illustrates the overview of sgsh. specifically, a skeleton generator implemented by a small plm (e.g., bart) generates the skeleton for each input, where the skeleton is a series of discrete tokens that act as a particular signal to guide the llm toward the ground-truth question. to train the skeleton generator, we propose an automatic strategy to construct a high-quality training dataset, which leverages a rule-based method to initially extract skeletons and then utilizes the power of chatgpt to refine these skeletons. based on the training set, we learn the skeleton generator with a soft prompting strategy to generate the skeleton for each input. subsequently, the black-box llm utilizes the skeleton heuristics via skeleton injection and skeleton-aware in-context learning. concretely, given a test input consisting of triples along with the corresponding answer, the skeleton injection step integrates the generated skeleton into the test input. afterward, the skeleton-aware in-context learning step incorporates in-context examples with skeletons to effectively enhance the incontext learning capability for the test input, where each example shares a similar target question with the test input. ## pilot study to evaluate the effectiveness of the skeleton heuristics in enhancing the performance of kbqg, we undertake a preliminary investigation to analyze. ## conclusion we explore how to steer gpt-3.5 toward the gold question on kbqg. in this paper, we pro-pose a simple but effective framework sgsh to stimulate gpt-3.5 with skeleton heuristics for kbqg, which provides fine-grained guidance for gpt-3.5 to generate high-quality questions. specifically, we employ a bart-based skeleton generator that is trained on our constructed training dataset using a learnable prompting strategy to obtain skeleton heuristics. toward these skeleton heuristics, we then devise a skeleton heuristics-enhanced prompt to trigger gpt-3.5 to align with the ground-truth question. extensive experiments on widely used datasets demonstrate the advanced performance of our proposed sgsh. in addition, optimizing the skeleton generator with only a small amount of training data (i.e., 10%) can outperform existing sota (i.e., dsm). this fine-grained guiding framework could be inspiring for other nlp tasks.