Querent Intent in Multi-Sentence Questions (2020)

Abstract: Multi-sentence questions (MSQs) are sequences of questions connected by relations which, unlike sequences of standalone questions, need to be answered as a unit. Following Rhetorical Structure Theory (RST), we recognise that different “question discourse relations” between the subparts of MSQs reflect different speaker intents, and consequently elicit different answering strategies. Correctly identifying these relations is therefore a crucial step in automatically answering MSQs. We identify five different types of MSQs in English, and define five novel relations to describe them. We extract over 162,000 MSQs from Stack Exchange to enable future research. Finally, we implement a high-precision baseline classifier based on surface features.

## prior work prior work on qa has focused on either single questions contained within dialogue @xcite @xcite , or questions composed of two or more sentences crowd-sourced by community qa (cqa) services @xcite . our definition of msqs is similar to the latter, but it should be noted that sentences in existing cqa datasets can be declarative or standalone, while in our case they must be a sequence of questions that jointly imply some user intent. popular tasks on cqa have only considered the semantics of individual questions and answers, while we are more focused on interactions between questions. @xcite and @xcite classify questions to improve qa performance, but their work is limited to standalone questions. @xcite was the first to identify msqs as a distinct phenomenon, and curated a small dataset consisting of 300 msqs extracted from yahoo answers. however, this dataset is too small to enable significant progress on automatic classification of msq intent. ## large-scale msq dataset stack exchange is a network of question-answering sites, where each site covers a particular topic. questions on stack exchange are formatted to have a short title and then a longer body describing the question, meaning that it is far more likely to contain msqs than other question answering sites, which tend to focus attention on the title with only a short amount of description after the title. there is a voting system which allows us to proxy well-formedness, since badly-formed questions are likely to be rated poorly. it covers a variety of topics, meaning that we can obtain questions from a variety of domains. to obtain the data, we used the stack exchange data explorer foot_0 , an open source tool for running arbitrary queries against public data from the stack exchange network. we chose 93 sites within the network, and queried each site for entries with at least two question marks in the body of the question. we removed any questions with t e x and mark-up tags, then replaced any text matching a regex pattern for a website with '[website]'. from this cleaned text, we extracted pairs of msqs by splitting the cleaned body of the question into sentences, then finding two adjacent sentences ending in '?'. we removed questions under 5 or over 300 characters in length. finally, we removed any question identified as non-english using langid.py @xcite . many of the questions labelled as 'non-english' were in fact badly formed english, making language identification a useful pre-processing step. after cleaning and processing, we extracted 162,745 questions from 93 topics foot_1 . a full list of topics and the number of questions extracted from each is given in appendix a. we restrict the dataset to pairs of questions, leaving longer sequences of msqs for future work. ## msq type as a proxy for speaker intent msqs are distinct from sequences of standalone questions in that their subparts need to be considered as a unit (see (1) in section 1). this is because they form a discourse: a coherent sequence of utterances @xcite . in declarative sentences, the relationship between their different parts is specified by "discourse relations" @xcite , which may be signalled with discourse markers (e.g. if, because) or discourse adverbials (e.g. as a result, see @xcite ). we propose adapting the notion of discourse relations to interrogatives. a particularly useful approach to discourse relations in the context of msqs is rhetorical structure theory (rst) @xcite , which understands them to be an expression of the speaker's communicative intent. listeners can infer this intent under the assumptions that speakers are "cooperative" and keep their contributions as brief and relevant as possible @xcite . transposing this theory to interrogatives, we can conceptualise the querent's communicative intent as a specific kind of answer. reflecting this intent, the relation suggests an answering strategy. we introduce five types of "question discourse relations" with a prototypical example from our data set, highlighting the inferred intent and the proposed answering strategy in separable example what's the recommended kitten food? how often should i feed it? intent two questions on the same topic (the querent's kitten). strategy resolve coreference and answer both questions separately. reformulated example is himalayan pink salt okay to use in fish tanks? i read that aquarium salt is good but would pink salt work? intent speaker wants to paraphrase q1 (perhaps for clarity). strategy answer one of the two questions. disjunctive example is it normal for my puppy to eat so quickly? or should i take him to the vet? intent querent offers two potential answers in the form of polar questions. strategy select one of the answers offered (e.g. "yes, it is normal") or reject both (e.g. "neithertry feeding it less but more often"). ## classification using contrastive features since @xcite found that using conventional discourse parsers created for declaratives is not suitable for extracting discourse relations from msqs, we design our own annotation scheme and use it to implement a baseline classifier. following previous work on extracting discourse relations @xcite , we use discourse markers and discourse adverbials alongside other markers indicative of the structure of the question (listed in appendix c) to identify explicitly signalled relations. 3 we construct a high-precision, low-recall set of rules to distinguish the most prototypical forms of the five types using combinations of binary contrastive features. to derive the relevant features, we consider the minimal edits to examples of msqs required to break or change the type of discourse relation between their parts. we then define a feature mask for each msq type which denotes whether each feature is required, disallowed or ignored by that type. each mask is mutually exclusive by design. given a pair of questions, the system enumerates the values of each feature, and compares to the definitions in appendix b. if a match is found, the pair is assigned the corresponding msq label, otherwise annotation and our classifier. while separable msqs appear to be the most prevalent, the classifier identifies only a small fraction of them, implying that they are likely to be implicitly signalled. disjunctive and condi-tional are the most likely to be explicitly signalled. figure 1 it is assigned unknown. this process is illustrated in appendix d. to evaluate our classifier, 420 msqs from our test set were annotated by two native speakers. we then evaluate the classifier on the subset of 271 samples for which both annotators agreed on the msq type. the resultant confusion matrix is shown in figure 1a , with the classifier achieving 82.9% precision and 26.5% recall. overall, we find that our classifier performs well for a heuristic approach, but that real world data contains many subtleties that can break our assumptions. during the annotation process, we found many instances of single questions followed by a question which fulfils a purely social function, such as "is it just me or this a problem?" (a phatic question, see @xcite ). msqs can also exhibit more than one intent, presenting a challenge for both our classifier and the expert annotators (see appendix e). a limitation of our classifier is the focus on explicit msqs, which can be identified with well-defined features. the low recall of our classifier indicates that msqs are often implicit, missing certain markers or not completely fulfilling the distinguishing requirements. figure 1b shows that while disjunctive and conditional msqs are often explicitly signalled, the other types are likely to be implicit. ## conclusion inspired by the role of discourse relations in msq answering strategies, we propose a novel definition of five different categories of msqs based on their corresponding speaker intents. we introduce a rich and diversified multi-sentence questions dataset, which contains 162,000 msqs extracted from stack exchange. this achieves our goal of providing a resource for further study of msqs. additionally, we implement a baseline classifier based on surface features as a prelimininary step towards successful answering strategies for msqs. future work could improve on our classifier by considering implicit msqs, with one potential approach being to transform explicit msqs into implicit examples by removing some markers while ensuring the relation is still valid. other areas for further work include implementing appropriate answering strategies for different types of msqs, and investigating whether and how longer chains of msqs differ compared to pairs of connected questions. who, what, where, when, why, how, which c.5 conditional ("if") markers if so, accordingly, then, as a result, it follows, subsequently, consequently, if yes, if not, if the answer is yes, if the answer is no c.6 elaborative markers for instance, for example, e.g., specifically, particularly, in particular, more specifically, more precisely, therefore c.7 separable markers also, secondly, next, related, relatedly, similarly, furthermore