Natural Answer Generation: From Factoid Answer to Full-length Answer using Grammar Correction (2024)

Abstract: Question Answering systems these days typically use template-based language generation. Though adequate for a domain-specific task, these systems are too restrictive and predefined for domain-independent systems. This paper proposes a system that outputs a full-length answer given a question and the extracted factoid answer (short spans such as named entities) as the input. Our system uses constituency and dependency parse trees of questions. A transformer-based Grammar Error Correction model GECToR is used as a post-processing step for better fluency. We compare our system with (i) a Modified Pointer Generator (SOTA) and (ii) Fine-tuned DialoGPT for factoid questions. We also tested our approach on existential (yes-no) questions with better results. Our model generates more accurate and fluent answers than the state-of-the-art (SOTA) approaches. The evaluation is done on NewsQA and SqUAD datasets with an increment of 0.4 and 0.9 percentage points in ROUGE-1 score respectively. Also, the inference time is reduced by 85% compared to the SOTA. The improved datasets used for our evaluation will be released as part of the research contribution.

## introduction question answering (qa) is an exercise of finding solutions for a query from a given paragraph. normally small spans of text, inclusive of named entities, dates, etc. are extracted as answers. however, knowledge-base (kb) orientated qa systems extract factoid solutions by using a structured query or neural representation of the question. as a natural extension and post-processing step, the retrieved factoid answer is transformed into a full-length natural sentence. unlike conversational chat-bots designed to mimic human communique without worrying to be factually correct, or assignment-orientated dialogue system which places the retrieved solution in a predefined template, our approach routinely generates correct full-length solutions, thereby, improving its utilization in these situations. question : who was the duke in the battle of hastings ? factoid answer : william the conqueror target : [the duke in the battle of hastings was william the conqueror. , william the conqueror was the duke in the battle of hastings.] ## data there is just one available dataset @xcite for this task created from a reading comprehension dataset having 15000 manually annotated, 300000 automatically annotated from squad @xcite , harvertingqa @xcite , and 420 data points in test dataset taken from the rest of the paper is organized as follows: firstly we discuss some recent works and related literature in section 2, after which we give details about the data used for evaluating our system in section 3. after that we talk about our approach in section 4; rule based in section 4.1 (factoid questions in section 4. in most of the recent works, we have observed a question answering system where the answer is in the form of paragraphs (more than 1 sentence) extracted from online retrieved passages. @xcite . after going through the dataset, we realized the available dataset is not of high quality, having multiple grammatically incorrect questions/answers and also wrong or grammatically incorrect target answers in many cases. due to this, improving the quality of the dataset is the need of the hour. in natural language generation (nlg) systems, there can be more than one correct answer that is not incorporated well in the available dataset. in the existing dataset, we see only target (i) type annotations but target (ii) is also the correct way to answer this question and should be added to the annotation. so we improve the quality of the available dataset to handle the above-mentioned issues. we sampled 7200 data points from 15000 manually annotated squad samples @xcite , 420 data points from newsqa @xcite and made the required changes in target answers; some data points were removed due to incomplete question/answer. as given in the codes and the data sets will be publicly available after the acceptance of the paper. ## experimental setup we have used tesla t4 16gb gpu to carry out the experiments. for factoid questions, we have used two datasets having 380 and 6768 data points as given in as a post processing step of all our rule based approaches, we have used a pre-trained transformer encoder, grammar error correction (gec) given in @xcite . this model was available with 3 cutting edge transformer encoders, namely bert, roberta, and xlnet. experiments were carried out using all 3 above encoder based gec models as post processing steps in our rule based approach. for fine-tuning dialogpt, we took a pretrained dialogpt-small (117m parameters) and fine-tuned with around 13000 manually annotated samples data from @xcite . we trained the model for 8 epochs. the results on 380 data points (crossvalidation) of newsqa dataset by the fine-tuned model are reported in ## results we use standard bleu @xcite (nltk), rouge-1, 2, l @xcite (rouge-score) metrics to evaluate our system and compare our system with other approaches. in there are instances in the above tables were employing a gcm sometimes reduces the bleu or rouge scores, especially in in ## error analysis below we present some qualitative discussion and error analysis of answers generated by existing approaches and our proposed approach.