Learning on Structured Documents for Conditional Question Answering (2023)

Abstract: “Conditional question answering (CQA) is an important task in natural language processing that involves answering questions that depend on specific conditions. CQA is crucial for domainsthat require the provision of personalized advice or making context-dependent analyses, such aslegal consulting and medical diagnosis. However, existing CQA models struggle with generatingmultiple conditional answers due to two main challenges: (1) the lack of supervised training datawith diverse conditions and corresponding answers, and (2) the difficulty to output in a complexformat that involves multiple conditions and answers. To address the challenge of limited super-vision, we propose LSD (Learning on Structured Documents), a self-supervised learning methodon structured documents for CQA. LSD involves a conditional problem generation method anda contrastive learning objective. The model is trained with LSD on massive unlabeled structureddocuments and is fine-tuned on labeled CQA dataset afterwards. To overcome the limitation ofoutputting answers with complex formats in CQA, we propose a pipeline that enables the gen-eration of multiple answers and conditions. Experimental results on the ConditionalQA datasetdemonstrate that LSD outperforms previous CQA models in terms of accuracy both in providinganswers and conditions.”

## introduction recently, question answering (qa) has gained increasing interest in the field of natural language processing. various types of question answering tasks, such as knowledge-based qa @xcite , open domain qa @xcite , and multi-hop qa @xcite , have been extensively studied. among them, conditional question answering (cqa) @xcite is becoming increasingly important in various contexts, such as medical diagnosis, legal consultation, financial analysis, and more. unlike the traditional question answering problem that only accepts a question and returns an answer, cqa involves understanding a complex and lengthy document, finding all possible answers under different conditions, and determining under what condition the answer is applicable. figure 1 shows an example for cqa, where the answer could be different when the questioner is under different conditions. the cqa task includes providing potential answers "yes" and "no" and their corresponding conditions based on the given question and scenario. previous studies on cqa can be broadly categorized into two groups: extractive and generative methods. extractive methods @xcite extract the most relevant span from a document as answers and conditions. in contrast, generative methods @xcite ) use a generative model to generate answers along with their corresponding conditions directly. however, current cqa models face two common challenges. firstly, the supervised data for cqa is limited and expensive to obtain. unlike traditional qa datasets, cqa requires specific annotations that include scenarios, answers, and conditions, making the data collection process more extensive and time-consuming. secondly, current cqa models are unable to provide multiple conditional answers in a coherent and controlled format. extractive methods for cqa are mostly only able to provide a single answer or condition for a question, limiting their ability to produce multiple conditional answers. figure 1 : an example for cqa. the right side is a snapshot of a document discussing the policy of claiming child benefits. the green text span is the condition that has been satisfied. the yellow and blue text spans are the conditions for "yes" and "no" respectively. conversely, generative methods may generate inconsistent and incoherent answers and conditions due to their inherent randomness, especially when generating multiple conditional answers. these challenges underscore the need for improved approaches to effectively handle the generation of multiple conditional answers in cqa. in order to solve the problem of limited supervision, we propose a self-supervised learning method called lsd (learning on structured documents). lsd consists of two main components: conditional question generation and contrastive learning. for conditional question generation, our intuition is that if a more precise context that contains sufficient information to answer a conditional question can be passed to the qa model, then the conditional answers given through this context will have high accuracy and can be used for subsequent training. to achieve this goal, we propose a selective extraction process that extracts parts of a structured document that are likely to be able to answer a conditional question. for a certain selected part of the document, we use a state generator to generate a conditional question and user scenario, and use a label generator to generate highly believed answers. for contrastive learning, we use four methods of document perturbation to perturb the structure of the document, including node reordering, repetition, masking, and deletion. these methods will change the content of the document but have little impact on its semantics. we design a contrastive learning objective that encourages the model to give similar representations of document corresponding sentences before and after perturbation, enabling the model to learn effective semantic representations from complex documents and helping with conditional question answering. to solve the problem of complex output formats, we propose a pipeline that can generate multiple answers and their corresponding conditions. our pipeline extracts answer spans from sentences, generating query vectors for each answer and key vectors for each candidate condition. afterward, we calculate the query-key matching score for each answer and condition, and choose the best matches as the final output. unlike existing methods, our pipeline utilizes the structure of documents to generate questions and conditions, and can generate controllable multiple conditional answers. to verify the effectiveness of our method, we conduct experiments on the conditionalqa dataset @xcite . the experimental results showed that our method outperformed all baseline models in terms of answer and condition accuracy, indicating that our method can provide accurate answers and corresponding conditions to effectively answer conditional questions. in summary, our contributions are three-fold: ## preliminaries: structured documents structured documents contain complex and rich structural information, which is beneficial for learning conditional question answering. in this work, our model is trained on html documents, a widely used type of structured document. html documents are easily accessible and often contain rich semantic information, including tables, lists, and more. the underlying structure of an html document is represented by the document object model (dom) tree, wherein the entire document constitutes the root node, and individual elements are organized as child nodes within the hierarchy. a diagram of a dom tree is shown in figure 2 . since html does not always demonstrate a clear hierarchy among elements, we adopt a tag precedence order to convert html documents into trees, thus making the relationships between elements explicit. we order commonly used tags as: ⟨title⟩ -⟨h⟩ -⟨p⟩ -⟨li⟩ / ⟨tr⟩. each node's parent is the closest preceding higher-level node. for example, the ⟨h1⟩ tag is a section title and is the parent of ⟨h2⟩ subsection titles. the ⟨h2⟩ tag is a subheading and is the parent of ⟨p⟩ text elements. we omit tags that do not contain important information, such as ⟨b⟩ (bold),⟨i⟩ (italic), and ⟨a⟩ (hyperlink) tags. with our approach, each sentence within the html document can be clearly represented as a node in the document tree. c c l 2 0 2 3 claiming child @xmath0... </@xmath1.</@xmath2,000, but you may be taxed on the benefit. </@xmath3,000 but yours is higher, you're responsible for paying the tax charge. </@xmath4,000 you lose all of your benefit through tax. </@xmath5... to compile a corpus of structured documents for the cqa task, we consider the following criteria: based on these criteria, we propose to train our model to learn on national government websites, which are known for their formal and authoritative nature. we conduct web scraping to gather documents, filtering for policy documents, laws and regulations, and administrative guidelines, as they tend to exhibit clear logical structures and contain specific conditions relevant to the cqa task. for additional details regarding the construction of the corpus, which is referred to as dataset, please refer to appendix a. ## our approach in this section, we will introduce our proposed method lsd, which includes a conditional question generation module and a contrastive learning method for self-supervised learning on structured documents. after that, we will illustrate our pipeline that generates multiple conditional answers by calculating the matching score of answers and candidate conditions with query and key vectors. the overall process of our method is shown in figure 3 . ## analysis in this section, we conduct an ablation study to investigate the impact of our document modeling designs and contrastive learning. we further analyze the question generation process by evaluating the quality of generated questions and the accuracy of generated labels. ## conclusion and limitations in this paper, we present learning on structured documents (lsd), a self-supervised learning method for conditional question answering. lsd uses a conditional question generation method to leverage massive structured documents while improving conciseness, and applies contrastive learning to learn effective semantic representations from complex documents. we further propose a pipeline that could generate multiple answers and conditions to better handle the cqa task. we verify the effectiveness of the proposed method on the conditionalqa dataset. for future work, we plan to investigate how to better generate conditional questions and improve our model's performance in providing correct answers. despite the effectiveness of lsd in utilizing the structure of massive unsupervised data, there are still some potential points for improvement. one issue is that the state generator is only trained on answerable questions, leading to a distribution bias that there might be unanswerable questions. in addition, our pipeline can still not handle the position where a sentence has more than one answer, which limits our model's performance for broader scenarios. we will resolve these issues in future work.