You Make me Feel like a Natural Question: Training QA Systems on Transformed Trivia Questions (2024)

Abstract: Training question-answering QA and information retrieval systems for web queries require large, expensive datasets that are difficult to annotate and time-consuming to gather. Moreover, while natural datasets of information-seeking questions are often prone to ambiguity or ill-formed, there are troves of freely available, carefully crafted question datasets for many languages. Thus, we automatically generate shorter, information-seeking questions, resembling web queries in the style of the Natural Questions (NQ) dataset from longer trivia data. Training a QA system on these transformed questions is a viable strategy for alternating to more expensive training setups showing the F1 score difference of less than six points and contrasting the final systems.

## introduction question answering is a central problem in ai research. one way of understanding why people ask questions was explained in @xcite foot_1 : questions come from either an information-seeking paradigm @xcite or a probing, evaluative paradigm @xcite . while it is easy to get questions in the cranfield paradigm because the asker creates questions that they do not know the answer to, additional annotations to find these answers are expensive. for example, natural questions @xcite , a benchmark dataset collected by google from questions people asked online, critically does not include "found" correct answers. instead, annotating these answers could be more expensive than their manchester counterparts, mostly written by qa writing experts (e.g., trivia members). moreover, while large corporations can collect large-scale natural cranfield questions at no cost, these questions sometimes are of poor quality because of ambiguity @xcite or false presuppositions @xcite . due to these pit, @xcite argue that manchester questions are more useful for building and evaluating qa systems. thus, we utilize the quiz bowl (qb) samples, a manchester qa dataset, created by trivia experts (section 2). 3 this paper investigates whether and how we can transform manchester qb samples into questions that resemble natural, cranfield questions. to this end, we propose syntactic transformations (naturalization) that convert qb elicitations into qb-trans questions that resemble nq (section 3). to validate the quality of qb-trans for training qa systems, we consider two experimental settings: zero-shot and supervised. the zero-shot setting examines whether qb-trans is an effective training data for a qa system when compared to nq (section 4). we train qa systems with qb-trans training data and compare the two systems on the nq test set. average f1 scores on nq test set vary by less than 6 points, which implies that qb-trans can replace nq training data. we also combine nq with qb-trans as training data in our supervised setting (section 5), improving f1 (tested on nq test set) by 10 points compared to training on only nq. qb-trans lacks issues that plague nq: presupposition and ambiguity (section 6). moreover, naturalization generalizes to other datasets (section 6.4). our contributions are naturalizing manchester qb questions into cranfield qb-trans while retaining the positive traits of qb samples, thereby improving qa with a more affordable process. the dataset generated from naturalization can be used to answer non-nq data (section 6.5) which proves the generalization of naturalization. section 8 shows how this can ensure a cheaper and more up-to-date alternative to nq data by generating large-scale cranfield dataset that benefits training questionanswering models and generalizes to other non-nq datasets. ## artful but arcane qb dataset this section discusses why we use qb data and how different they are from nq questions. the next section explains naturalization (section 3). ## naturalization this section outlines naturalization: converting the elicitations into multiple nq-like questions (figure 1 ). this city on the bay of bothnia is home to nylund's three smiths and takanen and walter runeberg's statues of alexander ii. original: 3. select lexical answer type (over all elicitations with same answer) 1. parse sentence (simplified for diagram) nylund's three smiths np takanen and walter runeberg's statues of alexander ii np and is home to this capital on the bay of bothnia np city finnish city capital municipality 2. generate variations: alternate independent clauses and remove optional clauses this capital is home to nylund's three smiths this capital on the bay of bothnia is home to nylund's three smiths this capital is home to takanen and walter runeberg's statues of alexander ii 4. convert to question what city is home to nylund's three smiths what city on the bay of bothnia is home to nylund's three smiths what city is home to takanen and walter runeberg's statues of alexander ii 5. run classifier, rank by similarity to natural questions what city is home to nylund's three smiths? length: 8 bigram: home to bigram: what @xmath0.8 ## zero-shot qa with qb-trans training we ensure we use no nq data and evaluate on nq test set which disadvantages our approach as nq has issues such as presupposition and ambiguity (section 6.1). ## supervised qa system with qb-nq training data we combine all of the naturalized datasets with the original nq dataset (nq), with the goal of having the largest nq-like dataset and highest accuracy. ## analysis of transformed questions this section discuss the quality of our dataset compared to nq. we incorporate answer equivalence to the experiment with the goal of improving f1 score. finally we prove the generalization of natural-ization by showing how well our transformations can apply to non-nq data (as evaluated on the nq test set) and how well our transformed data can answer non-nq data. ## related work this section discusses the question generation and transformation from existing datasets, which is an effective alternative to expensive data collection. ## conclusion and future work transformed nq-like questions from the qb data is an alternative to expensive datasets like nq. the transformed data itself is not as good as nq by itself, but is competitive; this is a reasonable option if the resources are not available to curate a dataset like nq. however, the dataset is getting old with obsolute questions and out-of-date answers @xcite . if there is a budget to create a dataset comparable to nq, a small amount of this data augmented with transformed data from a dataset like qb can surpass a model trained on the nq dataset. this can act as a continuous flow of new natural questions. moreover, as no new nq tests sets are published, this can provide an alternative benchmark to the obsolete eval nq data. for future work, we can apply this conversion technique to other languages' probing datasets @xcite where transformation heuristics can be learned using human data. there are methods like reinforcement learning from human feedback (rlhf) that use nq along with other datasets @xcite or create new datasets aligning nq with other datasets for llms @xcite or create adversarial dataset @xcite or rank skill with complex questions @xcite . our work shows that there are additional sources of information that are cheaper and more recent that can feed into these datasets instead of nq.