SPARQL -to-Text Question Generation for Knowledge-Based Conversational Applications (2022)

Abstract: This paper focuses on the generation of natural language questions based on SPARQL queries, with an emphasis on conversational use cases (follow-up question-answering). It studies what can be achieved so far based on current deep learning models (namely pretrained T5 and BART models). To do so, 4 knowledge-based QA corpora have been homogenized for the task and a new challenge set is introduced. A first series of experiments analyzes the impact of different training setups, while a second series seeks to understand what is still difficult for these models. The results from automatic metrics and human evaluation show that simple questions and frequent templates of SPARQL queries are usually well processed whereas complex questions and conversational dimensions (coreferences and ellipses) are still difficult to handle. The experimental material is publicly available on https://github.com/Orange-OpenSource/sparql-to-text .

## introduction knowledge-based approaches have recently become popular in the field of question answering (qa) and dialogue, raising the task of semantic parsing that seeks to map a user's input questions to a formal representation that can be queried in a knowledge graph (kg). alternatively, techniques have been proposed to verbalize small kgs, for instance to summarize information to a user. still, the task which consists in verbalizing formal queries has been less studied. yet, interesting applications could be derived from sparql-to-text question generation: for instance, the generation of tutoring systems where users can exercise on a topic, or the simulation of users for qa or dialogue systems. this is why this paper studies sparql-to-text question generation, with a particular consideration attached to the generation of questions in a conversational context. the objective of the paper is to study what can be achieved so far on sparql-to-text question generation using datasets and pretrained models available in the literature. in this regard, the contributions are the following: after a literature review in section 2, section 3 and 4 present the datasets and models, respectively. then, prototyping experiments using different training setups are described in section 5, while a detailed analysis of the models' performance is given in section 6. ## related work question generation frequently refers to the task of generating a natural language questions based on a text @xcite . the generation can be conditioned on the manually spotted expected answer in the text @xcite , whereas generating them in a free way @xcite , even potentially generating possible answers @xcite . in the field of knowledge-based approaches, several propositions have been made for the verbalization of formal queries (in sql, sparql, owl, etc.) through rules or templates @xcite @xcite , or intermediate representations @xcite , leading to verbalizations with a variable naturalness. using neural approaches, several contributions have been made to generate questions from rdf triples @xcite or small kgs depicting multi-hop questions @xcite . in @xcite , this principle is improved by enriching the entities from the triples with information from a broader kg. a limit of these approaches is that they cannot cover several features offered by query language like sparql (e.g., union of triples, filters, aggregation functions, etc.). hence, to the best of our knowledge, our work is the first attempt to study the verbalization of sparql seeking to generate a large diversity of questions types. among other related work, knowledge-based qa (kbqa) tasks are interesting to study since they provide data with paired natural language question and formal representation (usually triples or sparql queries) @xcite @xcite @xcite . it is important to note that some of these corpora overlap because they are extensions or refinements of common ancestors. less datasets exist when considering the conversational kbqa: convquestions @xcite and csqa @xcite . while the former does not provide the formal representations associated to the natural language questions, the latter is relevant for our task. finally, in the field of dialogue, propositions have also raised to enable interoperability with kgs through a formal language @xcite . however, annotated datasets are usually private or small. hence, the conversational dimension in our sparql-to-text task is original. ## datasets in this paper, 4 kbqa corpora from the literature are used: simplequestions @xcite , lc-quad 2.0 @xcite , paraqa @xcite , and csqa @xcite . they have different characteristics, and they do not overlap. additionnaly, a new corpus is introduced to serve as a challenge set, i.e. no training data is available for it. this corpus has been generated based on the webnlg v.3.0 corpus @xcite , and is referred to as webnlg-qa. this section presents an overview of the 4 corpora from the literature, the generation process and resulting content of webnlg-qa, and how all these datasets were homogenized. general statistics and examples for the 5 resulting sparql-to-text datasets are given in ## models this paper investigates the difficulty of the task for pretrained transformer models. this section first provides information about the fine-tuning process of these models, and then introduces several naive models used as baselines in the experiments. transformer models. the proposed models are encoder-decoder (i.e., autoregressive) transformers, namely bart @xcite and t5 (kale and rastogi, 2020), fine-tuned on the sparql-totext task. for both architectures, the models are the "base" version, as provided by huggingface foot_5 . this appeared as a reasonable size since csqa is a very large corpus and many experimental settings are considered. hence, the impact of the size is not considered here. tokenizers are the default ones. input sequences longer than the length limit of 512 tokens were truncated from the beginning, and no padding was used. the t5 prefix is "sparql to nl: ". the fine-tuning is performed for 2 epochs with a batch size of 4 samples, which appeared to be the best setting on the development set. the optimizer is adamw with a static learning rate of 5 × 10 -5 and no weight decay. finally, note that webnlg data was not part of bart's or t5's training data for their pre-training. naive models. several naive approaches are experimented to intuit the difficulty of the task and provide reasonable baselines. the simplest approach is to concatenate all terms of all triples in the query, except variables which are ignored. the order of the triples is the same as in the query-i.e., randomized, no micro-planning (reiter and dale, 1997, chap. 5), hence the name blind concatenation. alternatively, a rule-based micro-planning was implemented to spot the main triple in the query, that is the one on which the beginning of the question will focus foot_6 . then, the main triple is placed first when concatenating. this approach is denoted as smart concatenation. to complete the approach, templates of questions were introduced to instanciate the triples. the most naive solution is to prefix all questions with "what" since this is the most frequent prefix in the training datasets. another solution relies on a set of more sophisticated patterns, each being adapted to specific query configurations (query verb, target variable, shape of the main triple, etc.). this technique is called smart concatenation + pattern. the next sections provide global results used to prototype a unique model for all the datasets this section studies the design of a sparql-totext model and provides global results. first, it studies the impact of adding input information along with the single sparql query. then, the different training datasets are merged in order to investigate the generalization capacity of the models and to come up with a unique model for all the datasets. all results are presented in terms of me-teor @xcite and bertscore (f1 score) @xcite on the test set of each corpus foot_7 , using huggingface metrics. ## detailed analysis this section first analyses how the t5 model behaves on different query types. then, a human evaluation on a real application is presented to evaluate the intelligibility and effectiveness of the generated questions. the focus is given on the challenge set webnlg-qa but complementary results for the other datasets are reported in different appendices. ## conclusion and future work in this paper, we have studied in depth the problem of generating questions from sparql queries, in particular in order to be able to integrate these questions in a conversational knowledge-based application such as a qa system or a task-oriented dialogue. contributions stand in the proposed corpora, including a new challenge set (webnlg-qa), and in the multiple experiments conducted to highlight the limits of the popular pretrained models bart and t5 for the sparql-to-text task. these experiments show that, although the linguistic quality of the generated questions is good, the task only really works well for unambiguous and frequent situations, generally conforming to what has been seen in training. in the future, it would be interesting to evaluate the questions generated with a qa system. although the varying performance of these systems may bring uncertainty in the interpretation of the results, this would complement the human evaluation results and provide another basis for other researchers to compare their own question generation models. then, several limitations remain to be overcome. first of all, a better generation of coreferences and ellipses should be investigated, as well as a better transfer capacity from one corpus to another. then, apart from the use of other kbqa corpora than those used in this paper, it is likely that the use of unsupervised approaches, i.e. not requiring aligned questions and queries, is a challenging avenue to explore. in particular, this could favor help mixing knowledge-based and text-based approaches, as called by our last results. tianyi zhang, varsha kishore, felix wu, kilian q. weinberger, and yoav artzi. 2020. bertscore: evaluating text generation with bert. in proceedings of the international conference on learning representations (iclr). a appendices a.1 examples of generated sparql queries this sections presents sequences of sparql queries generated as exposed in section 3.2 and algorithm 1 based on 2 sample kgs, depicted in figures and 3. using the graph of figure a.1, the resulting sequence of sparql queries is the following: 1. select distinct ?d where { ?d property:birth_date ?k . filter ( contains ( year ( ?k ) , '1942' ) ) . ?d property:known_for resource:no_hair_theorem } 2. select distinct ( count ( ?m ) as ?g ) where { resource:brandon_carter property:known_for ?m } 3. select distinct ?m where { resource:brandon_carter property:known_for ?m . filter ( ?m != resource:no_hair_theorem ) } 4. select distinct ?b where { resource:brandon_carter property:birth_place ?b . filter ( strstarts ( lcase ( ?b ) , 'e' ) ) } 5. select distinct ?t ?g where { resource:brandon_carter property:alma_mater ?g . resource:brandon_carter property:doctoral_advisor ?t } 6. select distinct ?x where { resource:brandon_carter property:sports_offered ?x } using the graph of figure 3, the generated queries are: 1. select distinct ?k where { { { ?k property:stylistic_origin resource:ska } union { ?k property:stylistic_origin resource:rock_music } } } 2. select distinct ?k where { ?k property:stylistic_origin resource:rock_music } 3. ask where { resource:mermaid_(train_song) property:genre resource:pop_rock } a.2 performance of each separate dataset on webnlg-qa this appendix details how the models trained on each dataset separately transfer to the webnlg-qa challenge set. results reported in