Transfer Learning and Masked Generation for Answer Verbalization (2022)

Abstract: Structured Knowledge has recently emerged as an essential component to support fine-grained Question Answering (QA). In general, QA systems query a Knowledge Base (KB) to detect and extract the raw answers as final prediction. However, as lacking of context, language generation can offer a much informative and complete response. In this paper, we propose to combine the power of transfer learning and the advantage of entity placeholders to produce high-quality verbalization of extracted answers from a KB. We claim that such approach is especially well-suited for answer generation. Our experiments show 44.25%, 3.26% and 29.10% relative gain in BLEU over the state-of-the-art on the VQuAnDA, ParaQA and VANiLLa datasets, respectively. We additionally provide minor hallucinations corrections in VANiLLa standing for 5% of each of the training and testing set. We witness a median absolute gain of 0.81 SacreBLEU. This strengthens the importance of data quality when using automated evaluation.

## introduction question answering (qa) has witnessed a massive number of stupendous improvements over the past few years which marked a new era of qa. at the core of this significant progress is the huge leap in the use of pretrained language model (plm). on several benchmarks, state-of-the art qa systems perform on par with human according to reported evaluation metrics. however, despite remarkable accuracy in answer detection and extraction, few works have considered returning a verbalized response to the user. indeed, most of qa systems out-puts over knowledge bases (kbs) are utterly bereft of any context. to this extent, more works progressively tackled the answer verbalization task (av) which consists in generating a verbalized form of the answer. as a consequence, the user may benefit from a more contextualized response. recently, there have been few techniques proposed to perform surface realisation of a raw answer. with the lack of paired training data, @xcite investigated an unsupervised method to obtain answer verbalizations for both english and french languages. an initial step was to first check whether the question marker (e.g. who, what) could be straightforwardly substituted with the raw answer or not. for instance, with the question "who is the president of the u.s.?", its raw answer "joe biden" can directly replace the question marker "who" with the question mark substituted with a period. if this is not the case, the question is segmented into chunks based on the syntactic tree parsed with udpipefuture @xcite . after defining the raw answer as a new chunk, all possible permutations of the chunks are collected. the most likely permutation is identified with a plm such as gpt2 @xcite . finally, @xcite use bert @xcite to find any possibly missing function words around the raw answer such as a, an, to, with, in etc. in spite of its appealing unsupervised mechanism, this method is computationally expensive because of the cost of estimating the likelihood of all (distinct) permutations. moreover, the likelihood is computed with potential absent words which may jeopardize the final ranking of permutations. following, multiple datasets were released to spur the community to apply end-to-end learning @xcite @xcite . @xcite introduced vogue, an end-to-end model based on a dual encoder-decoder architecture. more precisely, the input question is encoded with a first transformer encoder @xcite . on top of that, a logical form of the question is also encoded with an additional transformer encoder. the logical form is a simplified representation of the question, similar to a query, inspired from @xcite and @xcite . taking our aforementioned question example, its logical form is find(president, u.s). during the decoding phase, vogue uses entities placeholders foot_0 for both the raw answer and the subject entity to generate an abstract version of the response. following the previous example, the generated verbalization would be "[ans] is the president of [ent]". in our work, we utilize a comparable mechanism fused with large-scaled pretrained models to leverage efficient transfer learning. specifically, our contribution is twofold: ## our approach in this section, we present our method based on transfer learning and masked generation. we consider an input question @xmath0, x 2 , . . . , x n -1 , x n } with x i the i th word and its raw answer @xmath1, a 2 , . . . , a k-1 , a k } with a j the j th word of the answer foot_1 . the goal is to generate a verbalized answer @xmath2, y 2 , . . . , y m -1 , y m } we model the generation of each token as a conditional θ-parameterized probability distribution. more precisely, we estimate θ such that p θ (y i |x, a, y 1 , y 2 , . . . , y i-1 ) is maximized. as mentioned in dai and le (2015), howard and ruder (2018) and @xcite , nlg has significantly benefited from transfer learning and very large plms @xcite . the generalization ability to unseen data has tremendously improved over the last decades due to the use of excessively large training corpora. as a consequence, we consider two recent plms for generation to leverage transfer learning: in order to verbalize the answer, a first step consists in encoding x with the encoder part of t5 or bart model. then, the decoder part takes learned representations to generate y . in our case, a placeholder is generated in y which will be replaced by the raw answer a as explained in next section. ## datasets more and more efforts have been made to construct and annotate new qa datasets. however, most of proposed corpora do not include a well-formed and informative response. in fact, no verbalization of the retrieved answer is usually given. only the raw answer acts as the final prediction which puts a curb on possible downstream generation task. to this end, we explore newly released datasets equipped with a natural language form of the response: ## experiments in our experiments, we provide empirical results on the introduced datasets in section 3. in section 4.2, we compare our transfer learning approach over the existing literature using t5 and bart embedded with a masking strategy. then, we explore the advantage of placeholders in section 4.3. our inputs and outputs with and without our masking approach are depicted in ## conclusion we proposed to verbalize answers usually returned by any question-answering system from a structured knowledge base. we combined the ad-vantages of transfer learning and masked generation. we compared our strategies with and without masks using t5 and bart. we showed that using massively pretrained models with answer placeholders alleviates the learning and led to unprecedented results on vquanda, paraqa and vanilla datasets. furthermore, we revealed multiple redundant entity hallucinations in the vanilla dataset. by automatically correcting 5% of them, we observed shifts in performances. this further demonstrates the limitation of automatic metrics when references are not reliable.