Transformer based Natural Language Generation for Question-Answering (2020)

Abstract: This paper explores Natural Language Generation within the context of Question-Answering task. The several works addressing this task only focused on generating a short answer or a long text span that contains the answer, while reasoning over a Web page or processing structured data. Such answers’ length are usually not appropriate as the answer tend to be perceived as too brief or too long to be read out loud by an intelligent assistant. In this work, we aim at generating a concise answer for a given question using an unsupervised approach that does not require annotated data. Tested over English and French datasets, the proposed approach shows very promising results.

## introduction question-answering systems (qas) aim at analyzing and processing user questions in order to provide relevant answers @xcite . the recent popularity of intelligent assistants has increased the interest in qas which have become a key component of "human-machine" exchanges since they allow users to have instant answers to their questions in natural language using their own terminology without having to go through a long list of documents to find the appropriate answers. most of the existing research work focuses on the major complexity of these systems residing in the processing and interpretation of the question that expresses the user's need for information, without considering the representation of the answer itself. usually, the answer is either represented by a short set of terms answering exactly the question (case of qas which extract answers from structured data), or by a text span extracted from a document which, besides the exact answer, can integrate other unnecessary information that are not relevant to the context of the question asked. the following presents two answers for who is the thesis supervisor of albert einstein? possibly generated by two systems : alfred kleiner albert einstein is a german-born theoretical physicist who developed the theory of relativity, one of the two pillars of modern physics. given the specificity of qas which extract answers from structured data, users generally receive only a short and limited answer to their questions as illustrated by the example above. this type of answer representation might not meet the user expectations. indeed, the type of answer given by the first system can be perceived as too brief not recalling the context of the question. the second system returns a passage which contains information that are out of the question's scope and might be deemed by the user as irrelevant. it is within this framework that we propose in this article an approach which allows to generate a concise answer in natural language (e.g. the thesis superviser of albert einstein was alfred kleiner) that shows very promising results tested over french and english questions. this approach is a component of a qas that we proposed in rojas @xcite and that we will briefly present in this article. in what follows, we detail in section 3 the approach we propose for answer generation in natural language and we briefly discuss the qas developed. we present in section 4 the experiments that we have conducted to evaluate this approach. ## related work the huge amount of information available nowadays makes the task of retrieving relevant informa-tion complex and time consuming. this complexity has prompted the development of qas which help spare the user the search and the information filtering tasks, as it is often the case with search engines, and directly return the exact answer to a question asked in natural language. the qas cover mainly three tasks: question analysis, information retrieval and answer extraction @xcite . these tasks have been tackled in different ways, considering the knowledge bases used, the types of questions addressed @xcite @xcite and the way in which the answer is presented. in this article, we particularly focus on the answer generation process. we generally notice two forms of representation addressed in literature. the answer can take the form of a paragraph selected from a set of text passages retrieved from the web @xcite @xcite @xcite , as it can also be the exact answer to the question extracted from a knowledge base @xcite @xcite . despite the abundance of work in the field of qas, the answers generation issue has received little attention. a first approach indirectly addressing this task has been proposed in @xcite . indeed, the authors aimed at diversifying the possible answer patterns by permuting the question's words in order to maximise the number of retrieved documents that may contain the answer to the given question. another answer representation approach based on rephrasing rules has also been proposed in @xcite ; @xcite within the context of query expansion task for document retrieval and not purposely for the question-answering task. the few works that have considered this task within the qas framework have approached it from a text summary generation perspective @xcite @xcite @xcite @xcite @xcite @xcite . these works consist in generating a summary of a single or various text spans that contain the answer to a question. most of these works have only considered causality questions like the ones starting with "why" and whose answers are para-graphs. to make these answers more concise, the extracted paragraphs are summed up. other approaches @xcite @xcite have explored this task as a classification problem that consists in predicting whether a text passage can be considered as an answer to a given question. it should be noted that these approaches only intend to diversify as much as possible the answer representation patterns to a given question in order to increase the probability of extracting the correct answer from the web and do not focus on the answer's representation itself. it should also be noted that these approaches are only applicable for qas which extract answers as a text snippet and cannot be applied to short answers usually extracted from knowledge bases. the work presented in @xcite tried to tackle this issue by proposing a supervised approach that was trained on a small dataset whose questions/answers pairs were extracted from machine comprehension datasets and augmented manually which make generalization and capturing variation very limited. our answer generation approach differs from these works as it is unsupervised, can be adapted to any type of factual question (except for why) and is based only on easily accessible and unannotated data. indeed, we build upon the intuitive hypothesis that a concise answer and easily pronounced by an intelligent assistant can in fact consist of a reformulation of the question asked. this approach is a part of a qas that we have developed in rojas @xcite that extracts the answer to a question from structured data. in what follows, we detail in section 3 the approach we propose for answer generation in natural language and we briefly discuss the qas developed. we present in section 4 the experiments that we have conducted to evaluate this approach. and we conclude in section 5 with the limitations noted and the perspectives considered. ## nlg approach for answer generation the answer generation approach proposed is a component of a system which was developed in rojas @xcite and which consists in a spoken conversational question-answering system which analyses and translates a question in natural language (french or english) in a formal representation that is transformed into a sparql query foot_0 . the sparql query helps extracting the answer to the given question from an rdf knowledge base, in our case wikidata foot_1 . the extracted answer takes the form of a list of uris or values. although the qas that we have developed (rojas @xcite is able to find the correct answer to a question, we have noticed that its short representation is not user-friendly. therefore, we propose an unsupervised approach which integrates the use of transformer models such as bert @xcite and gpt @xcite . the choice of an unsupervised approach arises from the fact that there is no available training dataset associating a question with an exhaustive and concise answer at the same time. such dataset could have helped use an end-to-end learning neural architecture that can generate an elaborated answer to a question. this approach builds upon the fact that we have already extracted the short answer to a given question and assumes that a user-friendly answer can consist in rephrasing the question words along with the short answer. this approach is composed of two fundamental phases: the dependency analysis of the input question and the answer generation using transformer models. ## experiments and evaluation the existing qas test sets are more tailored to systems which generate the exact short answer to a question or more focused on the machine reading comprehension task where the answer consists of a text passage from a document containing the short answer. therefore, we have created a dataset which maps questions extracted from the qald-7 challenge dataset @xcite with natural language answers which were defined by a linguist and which we individually reviewed. this dataset called quereo consists of 150 questions with the short answers extracted by the qas that we described above. we denote an average of three possible gold sanswers in natural language for each question. french and english versions were created for this dataset. to evaluate the proposed approach, we have referred to standard metrics defined for nlg tasks such as automatic translation and summarization, as they allow to assess to what extent a generated sentence is similar to the gold sentence. we con-sider three n-gram metrics (bleu, meteor and rouge) and the bert score metric which exploits the pre-trained embeddings of bert to calculate the similarity between the answer generated and the gold answer. to be able to compare the different configurations of the approach, we refer to friedman's test @xcite which allows to detect the performance variation of different configurations of a model evaluated by several metrics based on the average ranks. figure 3: screenshot of the evaluation tool we also conducted a human evaluation study for the french and the english versions of the dataset, in which we asked 20 native speakers participants to evaluate the relevance of a generated answer (correct or not correct) regarding a given question while indicating the type of errors depicted (grammar, wrong preposition, word order, extra word(s), etc). figure 3 presents the evaluation framework that we have implemented and provided to the participants. the results of each participant are saved in a json-file (figure 4 ). the inter-agreement rate between participants reached 70% which indicates a substantial agreement. through the human evaluation study, we wanted to explore to what extent the standard metrics are reliable to assess nlg approaches within the context of question-answering systems. we note that the highest human accuracy score for french of about 85% was scored with the first architecture coupled with bert as the generation model (gm) and camembert as the language model (lm). we also notice that the architecture a1, which considers the lm assessment of the structure before generating missing words, performs better. surprisingly, as a generative model, the multi- these findings mean that we actually can rely on the use of these standard metrics to evaluate the answer generation task for question-answering. we also tried to analyse the errors indicated by the participants. as we can note from figure 6 , the most common error reported for both english and french datasets is the word order which sheds the light on a problem related to the language model assessment phase. the second most reported error addresses the generation process, whether to indicate that there are one or more missing words within the answer (french) or the presence of some odd words (english). when trying to get an insight on the answers generated by the current intelligent systems such as google assistant and alexa, we noted that these systems are very accurate when extracting the correct answer to a question and can sometimes generate user-friendly answers that help recall the question context, specially with alexa. however, we noticed that most of the answers generated by these systems are more verbose than necessary, we also found out that when addressing yes/no questions, these systems generally settle for just a yes or no without elaborating, or, on the other hand, present a text span extracted from a web page and let the user guess the answer. let us take for example the following question was us president jackson involved in a war? here's something i found on the web. according to constitutioncenter.org: after the war of 1812, jackson led military forces against the indians and was involved in treaties that led to the relocation of indians. the user has to focus on the returned text fragment in order to guess that the answer to his question is actually yes. this issue was particularly noted when addressing french questions. if we also take the example how many grandchildren did jacques cousteau have ? the two systems answer as follows: fabien cousteau, alexandra cousteau, philippe cousteau jr., céline cousteau. jacques cousteau's grandchildren were philippe cousteau jr., alexandra ousteau, céline cousteau, and fabien cousteau however, the user is not asking about the names of cousteau's grand-children and has to guess by himself that the answer for this question is four. a more accurate answer should indicate the exact answer to the question and then elaborate jacques cousteau had four grand-children. but these systems perform better in case when the terms employed in the question are not necessarily relevant to the answer. if we take the example of the question who is the wife of lance bass, the approach that we propose will generate the wife of lance bass is michael turchin. as we can note the answer generated was not adapted to the actual answer, while the other systems are able to detect such nuance: lance bass is married to michael turchin. they have been married since 2014. this issue has still to be addressed. ## conclusion and perspectives we have put forward, in this paper, an approach for natural language generation within the framework of the question-answering task that considers dependency analysis and probability distribution of words sequences. this approach takes part of a question/answering system in order to help generate a user-friendly answer rather than a short one. the results obtained through a human evaluation and standard metrics tested over french and english questions are very promising and shows a good correlation with human judgement. however, we intend to put more emphasis on the language model choice as reported by the human study and consider the generation of more than one missing word within the answer.