Multilingual Generation and Answering of Questions from Texts and Knowledge Graphs (2023)

Abstract: The ability to bridge Question Generation (QG) and Question Answering (QA) across structured and unstructured modalities has the potential for aiding different NLP applications. One key application is in QA-based methods that have recently been shown to be useful for automatically evaluating Natural Language (NL) texts generated from Knowledge Graphs (KG). While methods have been proposed for QG-QA across these modalities, these efforts have been in English only; in this work, we bring multilinguality (Brazilian Portuguese and Russian) to multimodal (KG and NL) QG-QA. Using synthetic data generation and machine translation to produce QG-QA data that is aligned between graph and text, we are able to train multimodal, multi-task models that can perform multimodal QG and QA in Portuguese and Russian. We show that our approach outperforms a baseline which is derived from previous work on English and adapted to handle these two languages.

## introduction the ability to generate and answer questions from both knowledge graphs (kg) and natural language (nl) text is useful in a number of ways. it permits easing users' access to the knowledge contained in kgs without having to master complex query languages, since an nl query from a user may be directly applied to kg information to derive the answer. it also allows for questions to be generated and answered from both the open domain information contained in nl text and the factual knowledge contained in kgs/knowledge bases, thereby widening the pool of information that is interrogable. and it is useful also for verifying the consistency of information between modalities. in particular, @xcite recently showed that multimodal kg/nl question generation @xcite and question answering (qa) can be used for assessing the semantic consistency between an input kg graph and a generated english text, thereby providing a reference-less metric to measure the quality of kg verbalisers (rdf-to-text models). the intuition is that, for a match between an input and its output, the answers that are extracted for a given set of questions from the input should be consistent with the answers that are extracted from the output (for the same set of questions). one key challenge to building multimodal qg-qa however, is the lack of annotated qg-qa data that is aligned between modalities that is available for training and evaluation of such models. furthermore, to be useful for rdf-to-text output evaluation, a multimodal qg-qa approach has to be robust to surface variations since the answer to a question likely has different surface forms in the input graph and the output text. its qg and qa also have to be cross-lingual so that questions from multilingual texts can be generated and answered from english-centric kgs. additionally, the set of questions generated for a given (graph, text) pair should be sufficiently large and the model(s) should also perform qa with accuracy and consistency, so as to be able to provide a fair assessment of the semantic consistency between graph and text. in this work, we investigate the generation and answering of questions from graph and from text for multiple languages besides english. we present models enabling this for russian and in brazilian portuguese. in addition, we apply our approach to webnlg, a dataset of @xcite pairs that is used to train rdf-to-text models, and we show that our approach extends to @xcite 's reference-less, semantic adequacy metric for rdfto-text models for these two languages. we make the following contributions. first, we create training data where questions are aligned with both their graph and their text answers, allowing the training of models that show cross-modal consistency: i.e. for a given question, the answers obtained from a graph and from its semantically equivalent text are consistent. second, we derive silver training data from english using machine translation (mt) and we demonstrate that the resulting multimodal, portuguese and russian qg-qa models trained on them have high internal consistency: in most cases, applying a russian or portuguese question generated from a graph to the same graph returns an answer that matches the graph entity that the question was originally conditionally-generated from. third, we show that our approach has much larger question coverage than a baseline adapting @xcite 's approach to our target languages. finally, to overcome the lack of any gold qa/qg data that is aligned between text and graph in these two languages, we designed our evaluation suite to include multiple internal, cross-modal, cross-approach and external checks. these checks also demonstrate that our approach significantly improves on the baseline. ## approach since there is no cross-modal qg/qa data for brazilian portuguese and russian, we approach the task by first creating the data necessary for training multimodal qg-qa models for english. we then machine translate this data to create training data for brazilian portuguese and russian. finally, we use this automatically translated data to train multimodal (kg/nl), multi-task (qg-qa) models for these two languages. the following outlines our approach and details are provided in the subsequent sections. we first create a large synthetic dataset (section 3) of qa pairs for graphs and texts in english as leveraging existing resources (datasets and models) already developed for that language allows us to obtain qa pairs at scale. we do this by applying off-the shelf qg and qa models to texts from kelm @xcite , a large dataset of (wikidata graph, english text) pairs. we call this data q-kelm en . we show that our approach brings substantial improvements over the baseline, to question coverage, qa consistency, correlations with human judgments of semantic adequacy as well as answerability by a retrieval-based multilingual qa model. we reserve the test sets of webnlg p tbr and webnlg ru -1,606 (g, t p tbr ) and 1,102 (g, t ru ) pairs respectively -for evaluation as the parallel (g, t) in these test sets means that if a question can be answered by a text (or graph), it can also be answered by the graph (or text) that it is paired with -this allows us to test cross-modal qg and qa for consistency (section 7.3). we use the training portion of webnlg to produce our qa/qg datasets, which is done in two phases and which we describe in the following sections and illustrate in figure 1 . 1 in webnlg, the graphs are from the dbpedia kg. here we use a version where some of the dbpedia graphs have been mapped to wikidata @xcite , or else removed of underscores and camelcase to align with the wikidata format. step 1: synthetic qa data in the first step, we derive (t, a t , q) triplets from kelm texts 2 using off-the-shelf models for textual qg and qa. 3 we first use the qg model to generate questions from texts. next we use the qa model to generate answers from the resulting (text, question) pairs. using a set of heuristics, we then align (i) each text answer a t to an entity in g giving us the graph answer a g and (ii) each question to the subgraph g 1 of g matching its content, which in turn allows us to obtain the question size, nf . the set of generated questions is then filtered to ensure the quality of the silver data; this includes posing each question to a second deepset textual qa model for english and only accepting questions which both qa models found answerable and whose answers are the same or overlap. questions whose text answer could not be matched to a graph answer were also removed. this gives us an initial synthetic qa/qg dataset of pg, a g , t, a t , nf, q, g 1 q tuples, which we call q-kelm en . step 2: english text/kg qg models using q-kelm en , we train two qg models that can controllably generate simple (one kg graph fact) and complex (>1 kg graph fact) questions from either text or graph. the controls we use are: the size of the question in terms of kg triples (nf ), and plausible question types (e.g., who, what, when, where, which, how (to)) in the case of simple questions generation. these controllable qg models enable our generation of wide-coverage in-domain qg-qa data that also allows us to generate multiple questions per input (see next point and section 6). 2 we used a version of kelm filtered for (g, t) pairs where g has between 2 and 5 triples (as larger sizes lead to unnatural questions). we also filtered out the kelm (g, t) pairs (i) whose properties in g were not found in the wikidata sparql endpoint or have a functional nature, i.e. containing terms such as 'image of', 'instance of human', 'list of', 'disambiguation' which gives rise to superfluous t in kelm; and (ii) t that do not have a high fidelity with their g, using a similarity measure (scao and gardent, 2023) trained via contrastive loss on rdf graph-text pairs. answerability, consistency and alignment we add an answerability+consistency filter on the questions generated from either graph or text of webnlg, by posing them to the two deepset textual qa models -we then keep only questions where both qa models return an answer which (i) has a confidence score ě 0.7, (ii) shares at least a token overlap with the other model's answer, and (iii) has a token overlap with the answer used to condition qg (i.e. a text answer for questions from text; a graph answer for questions from graph). in this way, for all questions that are generated from a graph, we ensure that they have a matching graph and text answer. we call the resulting datasetof graph-and text-based (question, answer) pairs, q-webnlg en . in the second phase, we leverage mt and multilingual kg entity labels from wikidata to trans-form q-webnlg en into portuguese and russian versions. we describe the process for portuguese first. we translate both the english texts of webnlg training data and the set of questions from q-webnlg en using a t5 english-portuguese translation model foot_0 . we filter these translations using checks (details in appendix a.2) that include back-translation and keeping only those whose automatic scores are above a cut-off. since we machine translate the questions from english, the semantics of the questions might be affected; as such for each translated question q, we then use a textual qa model trained on portuguese squad data to obtain the answer to that question from the (automatically translated) portuguese text (a p tbr t ) and we use heuristics (appendix a.2.1) to align this text answer to an entity a g in the matching graph foot_1 . we discard any qa pair for which a p tbr g is different from the graph answer a en g associated with the original english question (the question q is a translation of). we do the same for russian, except that we do not translate webnlg training data to russian as it is already available @xcite . to translate the q-webnlg en questions into rus- ## multimodal multi-task qg-qa model we train monolingual models (i.e. one each for portuguese and russian) to limit the effect of crosslingual transfer during training (i.e. only english-portuguese or english-russian when generating and answering with graph). each language-specific model -which is fine-tuned from the public checkpoint of the 300m-parameter mt5-small @xcite for 10 epochs (to be comparable with the baseline) -is trained in a multimodal (graph and text), multi-task (qg and qa) manner, where each training batch contains a mix of all the tasks, i.e. text qg, text qa, kg qg and kg qa. this gives us a single unified model that can generate and answer questions from text and from graph. it takes approximately 20 hours to fine-tune one of our language-specific multimodal multi-task models using a single nvidia a40 gpu. ## related work joint qg & qa and data existing work on joint qg & qa @xcite @xcite have mostly focused on the text modality and in english only. where more than one modality or multiple languages are considered, these have usually been for qa alone. this is partly because available annotated multilingual qa/qg datasets @xcite are limited in size and primarily reserved for testing, as well as built for one modality only. while there exists moderatesized kg qa datasets, including the long-running qald @xcite and lc-quad @xcite kg qa challenges, these are still of a limited size (and coverage). furthermore, none of these qa/qg datasets have alignment of qa pairs across modalities, as well as across languages. notably, since open kgs such as wikidata (though large and with wide and growing fact coverage) remain english-centric, the setting we examine has to involve cross-linguality when carrying out qg and qa in the graph modality. ## conclusion in this work, we examine the task of multimodal qg and qa from text and from graph in multiple languages which has application in qa-based evaluation of text generated from kg. by generating synthetic qa/qg data in english that has questions and answers aligned between text and graph, and using mt, heuristics and quality filters, we obtain "silver" data for brazilian portuguese and russian that enables the training of multimodal multi-task models. our models provide wider qg coverage, is cross-lingual for qg-qa from kg graphs, and achieves greater internal and cross-modal qa consistency over a baseline that is derived from work in english (data-questeval) @xcite recently shown to be useful as a metric for evaluating the semantic consistency of rdf-to-text generations. in fact, using our approach leads @xmath0. we also see strong performance in qa accuracy of up to 15.5 bsc when our approach's questions are posed to a multilingual retrieval-based qa model. holistically, our approach's consistently better performance in coverage and qa consistency, demonstrates the improvements that it brings over the baseline.