Is that really a question? Going beyond factoid questions in NLP (2021)

Abstract: Research in NLP has mainly focused on factoid questions, with the goal of finding quick and reliable ways of matching a query to an answer. However, human discourse involves more than that: it contains non-canonical questions deployed to achieve specific communicative goals. In this paper, we investigate this under-studied aspect of NLP by introducing a targeted task, creating an appropriate corpus for the task and providing baseline models of diverse nature. With this, we are also able to generate useful insights on the task and open the way for future research in this direction.

## introduction recently, the field of human-machine interaction has seen ground-breaking progress, with the tasks of question-answering (qa) and dialog achieving even human-like performance. the probably most popular example is watson @xcite , ibm's qa system which was able to compete on the us tv program jeopardy! and beat the best players of the show. since then and particularly with the rise of neural networks (nn), various high-performance qa and dialog systems have emerged. for example, on the qqp task of the glue benchmark @xcite , the currently best performing system achieves an accuracy of 90.8%. despite this success, current qa and dialog systems cannot be claimed to be on a par with human communication. in this paper we address one core aspect of human discourse that is underresearched within nlp: non-canonical questions. research in nlp has mainly focused on factoid questions, e.g., when was mozart born?, with the goal of finding quick and reliable ways of matching a query to terms found in a given text collection. there has been less focus on understanding the structure of questions per se and the communicative goal they aim to achieve. state-of-the-art parsers are mainly trained on wikipedia entries or newspaper texts, e.g., the wall street journal, genres which do not contain many questions. thus, the tools trained on them are not effective in dealing with questions, let alone distinguishing between different types. even within more computational settings that include deep linguistic knowledge, e.g., parc's bridge qa system @xcite which uses a sophisticated lfg parser and semantic analysis, the actual nature and structure of different types of questions is not studied in detail. however, if we are aiming at human-like nlp systems, it is essential to be able to efficiently deal with the fine nuances of non-factoid questions @xcite . questions might be posed • as a (sarcastic, playful) comment, e.g., have you ever cooked an egg? (rhetorical) the importance of these communicative goals in everyday discourse can be seen in systems like personal assistants, chatbots and social media. for example, personal assistants like siri, alexa and google should be able to distinguish an ability question of the kind can you play xyz? from a rhetorical question such as can you be even more stupid? similarly, chatbots offering psychotherapeutic help @xcite should be able to differentiate between a factoid question such as is this a symptom for my condition? and a self-addressed question, e.g., why can't i do anything right? in social media platforms like twitter, apart from the canonical questions of the type do you know how to tell if a brachiopod is alive?, we also find non-canonical ones like why am i lucky? @xcite show that 42% of all questions on english twitter are rhetorical. to enable nlp systems to capture non-factoid uses of questions, we propose the task of question-type identification (qti). the task can be defined as follows: given a question, determine whether it is an information-seeking question (isq) or a non information-seeking question (nisq). the former type of question, also known as a canonical or factoid question, is posed to elicit information, e.g., what will the weather be like tomorrow? in contrast, questions that achieve other communicative goals are considered non-canonical, noninformation-seeking. nisqs do not constitute a homogeneous class, but are heterogeneous, comprising sub-types that are sometimes difficult to keep apart @xcite . but even at the coarsegrained level of distinguishing isqs from nisqs, the task is difficult: surface forms and structural cues are not particularly helpful; instead, @xcite and dayal (2016) find that prosody and context are key factors in question classification. our ultimate objective in this paper is to provide an empirical evaluation of learning-centered approaches to qti, setting baselines for the task and proposing it as a tool for the evaluation of qa and dialog systems. however, to the best of our knowledge, there are currently no openly available qti corpora that can permit such an assessment. the little previous research on the task has not contributed suitable corpora, leading to comparability issues. to address this, this paper introduces rquet (rocket), the resource of question types, a collection of questions in-the-wild labeled for their isq-nisq type. as the first of its kind, the resource of 2000 annotated questions allows for initial machine-/deep-learning experimentation and opens the way for more research in this direction. in this paper, we use this corpus to evaluate a variety of models in a wide range of settings, including simple linear classifiers, language models and other neural network architectures. we find that simple linear classifiers can compete with state-ofthe-art transformer models like bert @xcite , while a neural network model, combining features from bert and the simple classifiers, can outperform the rest of the settings. our contributions in this paper are three-fold. first, we provide the first openly-available qti corpus, aiming at introducing the task and comprising an initial benchmark. second, we establish suitable baselines for qti, comparing systems of very different nature. finally, we generate linguistic insights on the task and set the scene for future research in this area. ## relevant work within modern theoretical linguistics, a large body of research exists on questions. some first analyses focused on the most well-known types, i.e., deliberative, rhetorical and tag questions @xcite @xcite , to name only a few). recently, researchers have studied the effect of prosody on the type of question as well as the interaction of prosody and semantics on the different types @xcite @xcite @xcite , to name a few). it should also be noted that research in developing detailed pragmatic annotation schemes for human dialogs, thus also addressing questions, has a long tradition, e.g., @xcite ; @xcite ; @xcite ; @xcite . however, most of this work is too broad and at the same time too fine-grained for our purposes: on the one hand, it does not focus on questions and thus these are not studied in the desired depth and on the other, the annotation performed is sometimes too fine-grained for computational approaches. thus, we do not report further on this literature. in computational linguistics, questions have mainly been studied within qa/dialog systems, (e.g., @xcite ; @xcite ), and within question generation, (e.g., @xcite ; @xcite ). only a limited amount of research has focused on (versions of) the qti task. one strand of research has used social media data -mostly twitter -training simple classifier models @xcite @xcite . although this body of work reports on interesting methods and findings, the research does not follow a consistent task definition, analysing slightly different things that range from "distinguishing informational and conversational questions", "analysis of information needs on twitter" to the identification of rhetorical questions. additionally, they do not evaluate on a common dataset, making comparisons difficult. furthermore, they all deal with social media data, which, despite its own challenges (e.g., shortness, ungrammaticality, typos), is enriched with further markers like usernames, hashtags and urls, which can be successfully used for the classification. a different approach to the task is pursued by @xcite , who crowdsources human annotations for a large amount of twitter questions, without applying any automatic recognition. more recently, the efforts by zymla (2014), @xcite and @xcite are more reproducible. the former develops a rulebased approach to identify rhetorical questions in german twitter data, while @xcite implements a machine-learning system to identify rhetorical questions in the switchboard dialogue act corpus. in @xcite a rule-based multilingual approach is applied on a parallel corpus based on the bible. the above overview of relevant work indicates that creating suitable training datasets is challenging, mainly due to the sparsity of available data. social media data can be found in large numbers and contains questions of both types @xcite , but often the context in which the questions are found is missing or very limited, making their classification difficult even for humans. on the other hand, corpora with well-edited text such as newspapers, books and speeches are generally less suitable, as questions, in particular nisqs, tend to appear more often in spontaneous, unedited communication. thus, to create a suitable benchmark, we need to devise a corpus fulfilling three desiderata: a) containing naturally-occurring data, b) featuring enough questions of both types, and c) providing enough context for disambiguation. ## rquet as a benchmarking platform we used the rquet corpus to evaluate a variety of models, 3 establishing appropriate baselines and generating insights about the nature and peculiarities of the task. ## conclusion in this paper, we argued for the need of the question-type identification task, in which questions are distinguished based on the communicative goals they are set to achieve. we also provided the first corpus to be used as a benchmark. additionally, we studied the impact of different features and established diverse baselines, highlighting the peculiarities of the task. finally, we were able to generate new insights, which we aim to take up on in our future work.