Unsupervised Candidate Answer Extraction through Differentiable Masker-Reconstructor Model (2023)

Abstract: Question generation is a widely used data augmentation approach with extensive applications, and extracting qualified candidate answers from context passages is a critical step for most question generation systems. However, existing methods for candidate answer extraction are reliant on linguistic rules or annotated data that face the partial annotation issue and challenges in generalization. To overcome these limitations, we propose a novel unsupervised candidate answer extraction approach that leverages the inherent structure of context passages through a Differentiable Masker-Reconstructor (DMR) Model with the enforcement of self-consistency for picking up salient information tokens. We curated two datasets with exhaustively-annotated answers and benchmark a comprehensive set of supervised and unsupervised candidate answer extraction methods. We demonstrate the effectiveness of the DMR model by showing its performance is superior among unsupervised methods and comparable to supervised methods.

## introduction question generation (qg) is a burgeoning field of natural language understanding and generation. the objective of question generation is to produce well-structured, coherent, and valuable questions that correspond to a specific context passage and the intended answer. qg systems play a vital upstream role in enhancing the robustness and generalizability of question answering (qa) and machine reading comprehension (mrc) models @xcite , empowering chatbots and virtual assistants to answer more user needs @xcite , and powering ai-driven tutoring systems for educational purposes @xcite . for most existing qg systems, extracting qualified candidate answers from the context passage is an indispensable prerequisite to ensure that the generated questions are of high quality and relevant to the user's interests of salient information contained in the context passage. traditional methods for answer extraction rely on linguistic rules and models to discover the syntactic structure of the input passage's sentences. constituency tags of noun phrases and named entity recognition (ner) tags of person, time, location, etc., are popular choices of candidate answers. however, this kind of answer extraction method can only extract limited types of answers and usually disregards the importance of different possible answer tokens on the context and domain basis. more recently, with the development of machine learning and large language models, candidate answer extraction has been formalized as a sequence labeling problem and tackled with supervised sequence learning and classification with the use of annotated answers from mrc datasets like squad @xcite . nevertheless, the answer annotations of existing mrc and qa datasets have the partial annotation issue due to the annotation protocols that did not enforce exhaustive answer extraction from the context passage, leading to the overlooking of other essential details that could be beneficial in helping readers grasp the context @xcite . training models with partially annotated data could result in degraded performance as the data provide misleading supervision signal @xcite . furthermore, the supervised learning methods still face the generalization challenge when applying them to new domains where acquiring additional human annotations is time-consuming and expensive. to overcome the aforementioned limitations and challenges, we propose a novel unsupervised candidate answer extraction approach that leverages the inherent structure of context passage. we posit that passage tokens can be categorized into two types -backbone tokens and information tokens. backbone tokens are structural and common across various passages within the same domain, and such tokens are easily recoverable when masked. in contrast, information tokens are difficult to recover when masked, and such tokens are crucial information of a specific context passage, making them excellent candidate answers. in section 3, we introduce the differentiable masker-reconstructor model, in which the masker module learns to mask out tokens that the reconstructor module can then readily recover through the enforcement of selfconsistency. for comprehensive assessments, we exhaustively annotated a total of 100 passages with candidate answers on squad and wikihow cooking texts as detailed in section 4.1. we demonstrate the competitive performance of the dmr model in section 5 by comparing it with a comprehensive list of strong baselines that cover recent advancements of both supervised and unsupervised candidate answer extraction. in summary, we make the following contributions: ## related work candidate answer extraction focuses on the extraction of salient information tokens that usually contain key information and knowledge the readers may seek from the context passage. in previous work, named entities are the most popular type of candidate answers @xcite @xcite . in such cases, tokens of the context passage are first processed by existing ner models that tag tokens of person, organization, location, date/time, and numerical expressions. noun phrase (np) is another popular choice that has been used a lot @xcite @xcite . among them, yang et al. and nie et al. also consider more diverse types of phrases including adjective phrases (ap), verb phrases (vp), and sub-clauses (s) extracted from the consistency parsing models of the context passage. specially, nie et al. further expand recognized named entities into longer constituents for more diverse candidate answers. despite the reliance on ner and parsing models, these linguistic and syntactic rules based candidate answer extraction methods fail to take the importance of the information into account. with the advancements of supervised learning methods for neural networks, multiple studies @xcite @xcite utilize the annotated answer phrases of squad, a mrc dataset, to train neural models that are capable of tagging and classifying candidate answer tokens. however, @xcite out that the answer annotations of existing mrc and qa datasets have the partial annotation issue due to the annotation protocols that did not enforce exhaustive answer extraction from the context passage, leading to the overlooking of other essential details that could be beneficial in helping readers grasp the context. training models with partially annotated data could result in degraded performance as the data provide misleading supervision signal. to address the partial annotation problem, previous work resorts to positive-unlabeled (pu) learning @xcite , which uses modified risk estimators to re-balance the weights of positively labeled answer tokens and the remaining deemedas-unlabeled tokens for unbiased learning of an unbiased binary classifier. nevertheless, as a common problem across different tasks, the supervised learning methods still face the generalization challenge when applying them to new domains where acquiring additional human annotations is timeconsuming and expensive. self-consistency learning has been adopted in the field of natural language processing in recent years for tasks like text encoding @xcite , sentence compression and summarization @xcite , ner @xcite , and data-to-text generation @xcite . models that leverage self-consistency learning usually include two modules that are reversals of each other. specifically, one compressor module takes a sentence, paragraph, or document as the initial input and compresses the text into masked language modeling (mlm) is a fundamental technique for the pretraining of large language models (llm), first popularized by the bert model @xcite . during the pretraining stage, some percentage of the input tokens are masked at random, and the model's objective is to predict the original tokens based on their context tokens. this is different from traditional language models that typically predict the next word in a sequence, enabling models to bidirectionally understand the context and achieve strong language understanding capability. follow-ing bert, roberta @xcite also uses mlm as its pretraining objective. it improves upon bert by using dynamic masking that changes the masking pattern applied to the input text rather than static masking that always masking the same words, which results in a more robust llm. along this vein, mlm becomes one of the most popular pretraining objectives for large language models, including albert @xcite , bart @xcite , longformer @xcite , etc. that optimized bert from different aspects. ## approach our approach is based on the intuition and observation that the text passages within the same domain have inherent structure shaped by the underlying writing style, knowledge space, and expression formulation. we convey these concepts by showing a typical example we sourced from the wikihow foot_0 cooking articles. wikihow is a popular online resource that offers user-curated, step-bystep guides on how to perform certain tasks. as shown in ## results and analysis for the partially-annotated data, as the precision of each method might be underestimated due to the partial annotation of answers, we mainly look at the recall metric to assess their coverage of the existing answers. methods that intend to extract a more comprehensive set of candidate answers including scope, dmr, and diverseqa have achieved high recall as expected. as for the evaluation on exhaustively-annotated data, the aforementioned methods were able to maintain high recall. notably, scope consistently achieved the best performance, as measured by f1, with the help of a mass amount of annotated answers from the original squad dataset. when such annotated data is not available, our dmr model achieved the best performance among the unsupervised methods. compared to diverseqa, another competitive unsupervised method, the performance gain of dmr is mainly accomplished by balancing the precision, which emphasizes the importance of potential candidate answers, and recall, which emphasizes the extensiveness and diversity of potential candidate answers. we also would like to point out that, compared to unsupervised methods with second highest f1, our dmr model has the performance gain of +0.05 and +1.13 on exhaustively-annotated squad and wh-c respectively. the gain on the former dataset is quite insignificant while substantial on the later dataset. we attribute this difference to the fact that wh-c is more focused on a specialized cooking domain while squad consists of open-domain data. therefore, wh-c should have a more prominent underlying structure that can be captured by the dmr model through self-consistency learning to discriminate backbone tokens and information tokens. this finding could guide the future development and application of dmr model. lastly, it is also interesting to see that chatgpt achieved the second highest precision across all the datasets among unsupervised methods. upon checking chatgpt extracted answers, we find chatgpt significantly prefers to extract noun phrases and named entities as candidate answers. this preference results in the omission of more diverse answer forms and a lower recall. ## conclusions in this work, we target the underexplored task of unsupervised candidate answer extraction and introduce a novel differentiable masker-reconstructor model inspired by recent progress in self-consistency learning and masked language models. we have created two exhaustively annotated datasets that mitigate the partial annotation issue of existing mrc/qa datasets, thus allowing for a fairer evaluation of candidate answer extraction methods. we demonstrate that the dmr model, without the reliance on annotated data and external tools, achieves superior performance compared to other unsupervised methods, and its performance is also comparable to that of supervised methods.