Evaluation of Question Answer Generation for P ortuguese: Insights and Datasets (2024)

Abstract: Automatic question generation is an increasingly important task that can be applied in different settings, including educational purposes, data augmentation for question-answering (QA), and conversational systems. More specifically, we focus on question answer generation (QAG), which produces question-answer pairs given an input context. We adapt and apply QAG approaches to generate question-answer pairs for different domains and assess their capacity to generate accurate, diverse, and abundant question-answer pairs. Our analyses combine both qualitative and quantitative evaluations that allow insights into the quality and types of errors made by QAG methods. We also look into strategies for error filtering and their effects. Our work concentrates on Portuguese, a widely spoken language that is underrepresented in natural language processing research. To address the pressing need for resources, we generate and make available human-curated extractive QA datasets in three diverse domains.

## introduction over the years, question generation (qg) established itself as an important task in natural language generation. qg has been used in asking clarification questions in dialogue @xcite and for providing follow-up questions after an answer @xcite . also, question generation probabilities can be used to improve information retrieval @xcite . it is especially important in educational settings, where it can be used to generate quizzes automatically @xcite . in addition, it can be used for data augmentation aiming at domain adaptation @xcite or enhancing qg robustness @xcite . questions can be generated in an answer-aware or answer-agnostic fashion. question-answer generation (qag) @xcite is the task that aims at producing a set of questions along with their respective extractive answers, taking a context passage as input. qag is more complex than general qg since it needs to produce not only the questions but also the answers. portuguese is the sixth largest language in terms of the number of native speakers and yet it is underrepresented in terms of linguistic resources. to the best of our knowledge, there is only one fully extractive qa dataset, namely faquad @xcite , which contains 900 question-answer pairs. thus, it is pressing that more resources be created for this language. although much has been done in the field of qg, some gaps remain. current studies paid little attention to the decoding method, which is an important factor regarding the resulting accuracy, diversity, and adequacy of the output. also, some works focus on the downstream performance of generated questions in qa tasks, not assessing whether the generated questions are correct @xcite @xcite . in this work, we adapt and apply two qag approaches to generate question-answer pairs for three domains in portuguese using different decoding methods. our contributions include: the main findings from our analyses can be summarized as follows: ## materials and methods most current solutions for qag either employ plms with fine-tuning @xcite or llm with prompting. both have been applied to different languages and domains. our goal is to compare their performance according to quantitative and qualitative criteria across different domains in portuguese. to serve as training data, we translated squad 1.1 (which is licensed under cc by-sa 4.0) from english @xcite to portuguese using the googletranslate api foot_1 . despite the availability of other portuguese translations of squad v1.1, we opted to create our own version to ensure precise alignment of the answer span offsets between the answers and the contexts. maintaining this alignment is crucial because misaligned offsets in existing translations would introduce mismatches between answers and contexts, leading to the loss of training instances during model training. we will refer to our version as pt-squad and make it available at our repository foot_2 . more details on this translation process are in appendix a. a visual inspection of a sample of the translated contexts and questions showed that the translation quality was fair. ## evaluating question quality to assess the quality of the questions produced by the different qag methods and decoding strategies, we used the error taxonomy proposed by @xcite . annotation. human annotators classified the automatically generated questions according to the error taxonomy. we recruited five native portuguese speakers from our research group (two females and three males aged between mid-20s and early 40s). they were compensated at an hourly rate that was well above the regional minimum wage. the annotators were provided with guidelines and examples and were able to discuss them with the authors. the complete guidelines are in our repository foot_5 and a screenshot of the annotation interface is in appendix c. for the geoscientific domain, which demands specific knowledge, one annotator had a phd in geology, and the other one was a third-year bsc student. the inter-annotator agreement calculated using krippendorff's α was 0.49, which we consider a fair agreement. quality metrics. for each generation model and decoding strategy, we computed three evaluation metrics that capture the desired properties of qag. accuracy (acc) is the percentage of correct questions generated by the setup. productivity (prod) is the mean number of question-answer pairs generated by each context. diversity (div) is 1-b, where b is the self-bleu measure @xcite of the generated questions by the setup, and it is higher when there is little overlap among the generated questions. six different setups were evaluated for qag in each of the three domains. the results for the evaluation metrics are shown in ## evaluating answers looking into the answers provides strong cues about the questions. to identify what questions are being asked by the qag strategies, we devised an answer classification scheme. this taxonomy aims to identify biases in the type of answers and their interdependence with the domain. error category fine-grained examples disfluent wrong tense q: qual é a palavra que freud usou para descrever sua doença? awkward phrasing a: neurose not a question q: what is the word freud used to describe his disease? repetitions a: neurosis off-target unanswerable q: qual é a massa do próton? a: 1 / 1836 da massa do próton other answer span q: what is the mass of proton? a:1 / 1836 off the mass of proton wrong context too specific q: qual é o nome do campo de namorado? reveals answer a: namorado inconsistent q: what is the name of the field namorado? not specific enough a: namorado while there are other question classification taxonomies like bloom's cognitive levels @xcite , @xcite , and graesser's @xcite , we did not find them suitable to our goals. we have verified a low adherence to bloom's and graesser's taxonomy classes toward pt-squad data. consequently, we adopted a classification scheme that simplifies the trec-10 taxonomy. we manually classified answer spans from correct question-answer pairs selected in the previous experiment. our classification scheme for types of answers includes the following four classes. figure 2 shows the results of these analyses. to use as a reference, we also manually annotated 414 ground truth answers from pt-squad. with these questions, we can have a sample of humans' extractive answers to questions in the wikipedia domain. despite the increased focus on the term class, we found that the e2e and pipe answer distributions are compatible with pt-squad. a possible reason for the difference is that our wikipedia-pt subsample contains more salient terms. in the news domain, all generation strategies focused more on names and less on terms. this agrees with the intuition that news articles feature more discussion about named entities than debates about concepts. text from the geoscientific domain presents a fair amount of names of geological formations, basins, and locations. across the three domains, the llm approach, gpt-3.5-turbo and sabiá-medium, is biased towards longer and more descriptive answers. overall, the data suggests that the question-answer distribution is defined by the interaction between the types of question-answer pairs the methods tend to produce and the available salient information in the source passages. ## question filtering filtering methods can be used to reduce noisy and erroneous questions. we focus on assessing the effects of the filtering methods on question distribution and what errors the filtering prevents. in the following analysis we consider only the pipeline method. we implemented filtering methods based on the following criteria. answer score @xcite . this filter selects questions in which the probability output of the answer candidate defined in equation 1 surpasses a given threshold k. generation score (gs) @xcite . this filtering strategy selects questions with generation probability above a threshold k. in other words, the probability of the sequence of tokens that constitutes the question must surpass a threshold. roundtrip @xcite . for each question candidate and respective context pair, we check if the candidate answer is the same as the answer given by the 6-way qa ensemble model with the same question and context. we fine-tuned six different bertimbau qa models using different random seeds on pt-squad data. we select candidates for which the answer agrees with at least one qa answer, case "1/6", and when it agrees with all six, case "6/6". qrelscore lrm @xcite . the reference-free metric qrelscore can also be used to filter questions. originally, this metric used roberta and gpt-2 as components to calculate two sub-metrics: local relevance matching (lrm) and global relevance generation (grg), respectively. however, we switched roberta @xcite to bertimbau since our data is in portuguese. we also tried switching @xcite to xglm @xcite . however, it yielded poor grg results and hurt the qrelscore performance. thus, we report only the lrm values. we filter candidates by selecting instances with the qrelscore lrm above the sample mean. ϕ voting. to combine the strengths of previous filtering methods, we establish two voting schemes using the generation @xmath0.9, roundtrip 1/6, and qrelscore lrm. the first one, relaxed, accepts the candidate if one of the conditions is met. the second one, strict, only accepts a candidate if the three conditions are met. the answer score does not select correct questions across different domains. in fact, it only helps to select the right questions in the wikipedia domain, which is the same domain as the training data (see appendix e). the proportion of description type of questions gets lower by using higher thresholds of answer score. this means there are fewer questions that target answers that require more elaborate descriptions. in figure 3 , we plot the description class proportion by the generation score metric and the answer score. the answer span thresholds select simpler answers with less structured text. this does not necessarily mean shorter answers. as a result, we omit it from the next analyses. the errors filtering helps to prevent can be seen in figure 4 . the generation score thresholds reduce the wrong context error class and slightly improve the off-target class. roundtrip in both versions greatly reduces the off-target and slightly decreases the wrong context class. the voting strategy ϕ relaxed shows performance comparable to roundtrip 1/6 and ϕ strict only presents disfluent errors. qrelscore lrm is a marginal improvement over unfiltered question candidates. the results showed that fluency errors resist current filtering strategies. roundtrip consistency and qrelscore lrm consider the passage-question pair coherence through the answer and semantic similarity, respectively. the generation score thresholds could improve fluency. however, it actually improves the consistency and adequacy of questions. although filtering should focus on precision, recall is also important. applications of qag may also need a volume of questions, and a highly selective filtering method may not work. for this purpose, an ensemble of filtering methods in a relaxed voting scheme may be more appropriate. ## conclusion in this study, we provided an in-depth examination of the quality of the generated question-answer pairs. our goal was to identify the strengths and weaknesses of current qag methods. we investigated the impact of different decoding methods, assessed the accuracy and diversity of generated question-answer pairs, and analyzed the types of errors made by various qag models. additionally, we explored the classification of questions based on the model output and evaluated the effectiveness of different question filtering methods. the preference for beam search decoding in qag systems may need to be reconsidered based on the specific application. while it ensures higher accuracy, the reduction in diversity could limit the range of questions generated, which might be crucial in educational settings or for generating varied training data. in educational technology, the findings suggest that qag tools must be carefully designed to balance accuracy and diversity to create comprehensive assessments and learning materials. for training qa systems, ensuring that the generated questions are both accurate and varied can enhance robustness. the insights from this study could guide the development of more effective training datasets. regarding the influence of the source context in qag, the dependency on the input context suggests that qag systems need to be tailored to handle various types of source material. the study of accuracy in generation can be expanded to include non-extractive questions. despite the importance, qag with generative answers