How Do We Answer Complex Questions: Discourse Structure of Long-form Answers (2022)

Abstract: Long-form answers, consisting of multiple sentences, can provide nuanced and comprehensive answers to a broader set of questions. To better understand this complex and understudied task, we study the functional structure of long-form answers collected from three datasets, ELI5, WebGPT and Natural Questions. Our main goal is to understand how humans organize information to craft complex answers. We develop an ontology of six sentence-level functional roles for long-form answers, and annotate 3.9k sentences in 640 answer paragraphs. Different answer collection methods manifest in different discourse structures. We further analyze model-generated answers – finding that annotators agree less with each other when annotating model-generated answers compared to annotating human-written answers. Our annotated data enables training a strong classifier that can be used for automatic analysis. We hope our work can inspire future research on discourse-level modeling and evaluation of long-form QA systems.

## introduction while many information seeking questions can be answered by a short text span, requiring a short span answer significantly limits the types of questions that can be addressed as well as the extent of information that can be conveyed. recent work @xcite @xcite explored long-form answers, where answers are free-form texts consisting of multiple sentences. such long-form answers provide flexible space where the answerer can provide a nuanced answer, incorporating their confidence and sources of their knowledge. thus the answer sentences form a discourse where the answerers provide information, hedge, explain, provide examples, point to other sources, and more; these elements need to be structured and organized coherently. we take a linguistically informed approach to understand the structure of long-form answers, designing six communicative functions of sentences in long-form answers (which we call roles). 2 our framework combines functional structures with the notion of information salience by designating a role for sentences that convey the main message of an answer. other roles include signaling the organization of the answer, directly answering the question, giving an example, providing background information, and so on. about a half of the sentences in long-form answers we study serve roles other than providing an answer to the question. we collect discourse annotations on three long-form question answering (lfqa) datasets, eli5 @xcite , webgpt @xcite and natural questions (nq) @xcite . figure 1 contains an example annotation on each dataset. while all three contain paragraph-length answers needed for complex queries, they are collected in distinct mannersanswers in eli5 are written by reddit users; answers in webgpt are written by annotators who searched documents on a web interface and heavily quoted those documents to form an answer, and answers in nq are pre-existing paragraphs from wikipedia corpus. we collect three-way annotations for 3.9k sentences (∼700 question-answer pairs across three datasets). we also annotate a small number of model-generated answers from a recent long-form question answering (lfqa) system @xcite and provide rich analysis of their discourse structure. in all three datasets, we observe appearance of most proposed functional roles, but with different proportions. answers in eli5 contains more examples and elaborations, while answers extracted webgpt: how much money is needed in order to not have to work for the rest of your life? i worked on a window-washing robot that cleaned acres of rooftops over a huge commercial greenhouse. worked great, except when it didn't, and would either break down completely or just get lost and start climbing the wrong parts of the structure. then repair techs and manual window washers still have to be employed. i think this ends up being a cost/benefit problem where the reliability of our robots and price of implementation isn't quite at the point where it to determine how much money you need to never have to work again for the rest of your life, some calculation is needed to arrive at a dollar number tailored to you @xcite . you need to consider the amount you spend yearly, the effect of inflation on your savings, and the income you need from an investment portfolio to keep ahead of inflation. @xcite a reliable savings range is your current yearly spending multiplied by 28 to 36, with more security and comfort the higher the number is. @xcite for example, if you spend 30,000 per year, savings of 1,000,000 is likely a good number. @xcite however, another method of calculating is based on the median household income in the united states. @xcite at the median income of 59,000 with a yearly withdrawal rate of 4 and an investment return rate of 10%, the magic number to never have to work again is $1,475,000. @xcite nq: what does it mean to be a subject matter expert a subject -matter expert ( sme ) or domain expert is a person who is an authority in a particular area or topic. the term domain expert is frequently used in expert systems software development , and there the term always refers to the domain other than the software domain. … the development of accounting software requires knowledge in two different domains : accounting and software. some of the development workers may be experts in one domain and not the other. a sme should also have basic knowledge of other technical subjects . from wikipedia passages (nq) contain more auxiliary information. analyzing a subset of eli5 and webgpt, we also identify a big gap in lexical overlap between long-form answer and evidence passages across all functional roles. lastly, we found that human agreement of the discourse roles of model-generated answers are much lower than human-written ones, reflecting the difficulty for humans to process model-generated answers. with the data collected, we present a competitive role classifier, which performs on par with human when trained with our annotated data and can be used for automatic discourse analysis. we further envision using functional roles for controllable long-form generations, concise answer generation, and improved evaluation metrics for lfqa. ## defining answer discourse structure we study the discourse structure of long-form answers based on functional roles of sentences in the paragraph. functional structures characterize the communicative role a linguistic unit plays; as such, they vary across genres as the goals of communication also vary. in scientific or technical articles, these roles can be background, method, findings @xcite @xcite , while in news, they can be main event or anecdotes @xcite . these structures are related to, though distinct from, coherence discourse structures @xcite . the latter characterizes how each unit (e.g., adjacent clauses or sentences) relates to others through semantic relations such as temporal, causal, etc.; such structures can be trees that hierarchically relate adjacent units @xcite or graphs @xcite . in contrast, functional roles describe how information is organized to serve the communication goal, in our case, providing the answer. we developed our ontology by examining longform answers in online community forums (subreddit explain like i'm five (eli5)) and wikipedia passages, hence answers derived from different domains (e.g., textbooks) can contain roles beyond our ontology. we describe our six sentence-level discourse roles for long-form answers here: ## discourse analysis of long-form answers with our annotated data, we study the differences between the three types of long-form answers, namely answers provided by users in online community (eli5), answers written by trained annotators through web search (webgpt), and answers identified in wikipedia passages (nq). q/a validity webgpt boasts the lowest invalid rate, showing the high quality of their collected answers. ## discourse structure of model-generated answers having analyzed discourse roles of human-written long-form answers, we investigate the discourse structure of model-generated answers. this will allow us to quantitatively study the difference in terms of discourse structure across gold and generated answers, which we hope will cast insights to the linguistic quality of system outputs. ## automatic discourse analysis we study how models can identify the discourse role for each sentence in long-form answer in a valid qa pair. foot_7 such a model can be beneficial for large-scale automatic analysis. ## related work discourse structure. our work is closely related to functional structures defined through content types explored in other domains; prior work has affirmed the usefulness of these structures in downstream nlp tasks. in news, choubey et al. (2020) adopted van dijk (2013)'s content schema cataloging events (e.g., main event, anecdotal), which they showed to improve the performance of event coreference resolution. in scientific writing, content types (e.g., background, methodology) are shown to be useful for summarization @xcite , information extraction @xcite , and information retrieval @xcite . the discourse structure of argumentative texts (e.g., support, rebuttal) @xcite @xcite has also been applied on argumentation min-ing. to the best of our knowledge, no prior work has studied the discourse structure of long-form answers. question answering. recent work (cao and wang, 2021) have investigated the ontology of questions, which includes comparison questions, verification questions, judgement questions, etc. we construct the ontology of functional roles of answer sentences. one of the roles in our ontology is summary, yielding an extractive summarization dataset. this shares motivation with a line of work studying query-focused summarization @xcite . concurrent to our work, @xcite studies improving faithfulness of long-form answer through predicting and focusing on salient information in retrieved evidence document. lastly, our work build up on three datasets containing longform answers @xcite @xcite and extends the analysis of long-form answers from earlier studies @xcite . ## conclusion we present a linguistically motivated study of longform answers. we find humans employ various strategies -introducing sentences laying out the structure of the answer, proposing hypothetical and real examples, and summarizing main points -to organize information. our discourse analysis characterizes three types of long-form answers and reveals deficient discourse structures of modelgenerated answers. discourse analysis can be fruitful direction for evaluating long-form answers. for instance, highlighting summary sentence(s) or sentence-level discourse role could be helpful for human evaluators to dissect long-form answers, whose length has been found to be challenging for human evaluation @xcite . trained role classifier can also evaluate the discourse structure of model-generated answers. future work can explore using sentences belonging to the summary role to design evaluation metrics that focuses on the core parts of the answer @xcite , for assessing the correctness of generated the answer. exploring controllable generation, such as encouraging models to provide summaries or examples, would be another exciting avenue for future work.