Prompting Large Language Models with Chain-of-Thought for Few-Shot Knowledge Base Question Generation (2023)

Abstract: The task of Question Generation over Knowledge Bases (KBQG) aims to convert a logical form into a natural language question. For the sake of expensive cost of large-scale question annotation, the methods of KBQG under low-resource scenarios urgently need to be developed. However, current methods heavily rely on annotated data for fine-tuning, which is not well-suited for few-shot question generation. The emergence of Large Language Models (LLMs) has shown their impressive generalization ability in few-shot tasks. Inspired by Chain-of-Thought (CoT) prompting, which is an in-context learning strategy for reasoning, we formulate KBQG task as a reasoning problem, where the generation of a complete question is splitted into a series of sub-question generation. Our proposed prompting method KQG-CoT first retrieves supportive logical forms from the unlabeled data pool taking account of the characteristics of the logical form. Then, we write a prompt to explicit the reasoning chain of generating complicated questions based on the selected demonstrations. To further ensure prompt quality, we extend KQG-CoT into KQG-CoT+ via sorting the logical forms by their complexity. We conduct extensive experiments over three public KBQG datasets. The results demonstrate that our prompting method consistently outperforms other prompting baselines on the evaluated datasets. Remarkably, our KQG-CoT+ method could surpass existing few-shot SoTA results of the PathQuestions dataset by 18.25, 10.72, and 10.18 absolute points on BLEU-4, METEOR, and ROUGE-L, respectively.

## introduction question generation task requires a system to produce natural language questions based on the given context. kbqg @xcite is one of the imperative question generation tasks when the given * *corresponding author context derived from knowledge bases (kbs) is in the form of logical. kbqg has attracted increasing interests from both the industry and academia due to its potential for data augmentation in qa systems @xcite and its ability to assist dialogue systems in creating coherent questions @xcite . existing studies @xcite @xcite @xcite for kbqg tasks have predominantly utilized neural network-based approaches and demonstrated impressive performance by conducting finetuning on extensive training datasets. however, as the collection of kbqg data is labor-intensive, researchers start paying attention to the few-shot kbqg tasks @xcite , where a great challenge is posed for suppliers with limited resources: 1) a great deal of annotated data is demanded to allow the existing fine-tuned models to generalize well over different logical forms. however, due to the limitations of low-resource availability, training conventional models by fine-tuning on the full data becomes unrealistic. 2) a logical form is composed of entities, relations, and query grammar. having logical forms with various combinations of these basic components is crucial to uphold the model's capability for compositional generalization. the lack of data leads to a compositional challenge to the kbqg tasks @xcite . 3) certain logical forms can become complex when operations such as aggregation, superlatives, and comparisons are involved. representing these logical forms presents additional challenges. moreover, developing a kbqg method that incorporates diverse and elaborate expressions becomes particularly difficult in such low-resource scenarios @xcite . recently, llms such as gpt-3 and codex @xcite @xcite have proven their strong generalizability on a wide range of few-shot and zeroshot tasks with cot, including text interpretation, computer vision, planning and reasoning. meanwhile, a line of work @xcite @xcite @xcite validates that llms have the strong capability to accurately capture the semantics of relations between values in the data, enabling to transform the structured instructions to narrative text. the above studies inspire us to explore few-shot kbqg tasks by prompting llms with cot. however, how to apply llms to kbqg with cot is still unclear. on one hand, kbqg differs from tasks like code generation or question answering, as it involves incorporating kb-specific items into the input instead of self-contained narratives. therefore, formatting the input in an easily understandable manner while considering the kb schema is crucial. on the other hand, the challenge lies in designing effective cot prompts @xcite that can enhance the performance of llms in the context of few-shot kbqg. in this work, we propose kqg-cot framework, which is the first attempt for training-free few-shot kbqg with llms. as shown in figure 1 , our framework consists of two main steps, the objects of which are supportive logical forms selection from an unlabeled data pool and prompt construction. to acquire coherent logical forms, we employ a clustering technique to carefully choose multiple logical forms that serve as representatives, considering both their syntactic and semantic characteristics. to construct prompt, inspired by the principle of cot @xcite , we take the selected logical forms as exemplars and write rationales to split the generation of a complete question into mul-tiple steps. we concatenate the above rationales with the queried logical form to form a prompt, which guides a llm to outcome a reasoning process of generating a complex question aligning with the logical form. we further improve kqg-cot to kqg-cot+ via sorting the supportive logical forms by complexity. as previous methods rely heavily on the training instances to fine-tune a kbqg model. kqg-cot does not need numerous logical form question pairs to train the models. we test the performance of our prompting methods under few-shot setting on three public datasets, namely webquestions @xcite , pathquestions @xcite , and grailqa @xcite . we conduct a comprehensive comparison with a range of commonly used cot baseline methods including auto-cot @xcite , active-cot @xcite , random-cot @xcite and so on. the experimental results show that we can outperform all of them with an observable margin. besides, we also compare with a set of sota systems trained with full data or few data. our few-shot method could achieve competitive results to the full training methods. remarkably, our fewshot method could surpass existing few-shot sota results of pathquestions dataset by 18.25, 10.72 and 10.18 absolute points on bleu-4, meteor and rouge-l, respectively. kqg-cot provides a simple but effective solution to few-shot kbqg problem, we expect it could serve as an important baseline for future investigation to kbqg tasks under low-resource scenarios. our main contributions are summarized as follows: few-shot learning for text generation. in recent years, significant progress has been made in the field of few-shot learning for text generation. one line of work develops meta-learning frameworks for text generation @xcite @xcite , which aims to acquire an optimal initialization that enables accurate and rapid adaptation to a new task, even when limited data is available. other line of work proposes different augmentation algorithms to synthesize the data for training @xcite , so that conventional text generation models can be applied to the augmented data. most recently, llms are leveraged to solve few-shot text generation tasks such as text summarization @xcite @xcite , machine translation @xcite , dialogue generation @xcite @xcite ) and so on. there is no existing study applying llms to few-shot kbqg tasks. in-context learning with llms. without gradient updates, in-context learning (icl) effectively tackles a wide range of nlp tasks by incorporating a small number of prompted examples as part of the input @xcite to help llms understand the tasks. multiple studies @xcite explored the selection of examples that are similar to the query during prompt construction. recent researches @xcite @xcite highlight that the order of these examples in the prompt has a substantial influence. cot is a prompting strategy decomposing complex tasks into sub-tasks, helping the model to derive the correct answers progressively @xcite . it has been widely used in mathematical word problem solving, common-sense reasoning, and symbolic reasoning. our work incorporates cot strategy into kbqg tasks, where iterative process enables llms to ultimately obtain a complex question aligning with the logical form. ## experiment in this section, we first introduce the kbqg datasets used to evaluate the performance of our proposed method and the comparable baseline methods. next, we present the implementation details and demonstrate the experimental results. ## conclusion in this paper, we presented the kqg-cot approach to tackle few-shot kbqg tasks. kqg-cot retrieves relevant logical forms from unlabeled data and incorporates their characteristics. it then generates explicit prompt to showcase the reasoning process for complex question generation based on the selected examples. experimental results demonstrate that our approach achieves state-of-the-art performance compared to baselines and even shows competitive results to full-training methods.