Exploring the Role of Context to Distinguish Rhetorical and Information-Seeking Questions (2020)

Abstract: Social media posts often contain questions, but many of the questions are rhetorical and do not seek information. Our work studies the problem of distinguishing rhetorical and information-seeking questions on Twitter. Most work has focused on features of the question itself, but we hypothesize that the prior context plays a role too. This paper introduces a new dataset containing questions in tweets paired with their prior tweets to provide context. We create classification models to assess the difficulty of distinguishing rhetorical and information-seeking questions, and experiment with different properties of the prior context. Our results show that the prior tweet and topic features can improve performance on this task.

## introduction questions are common in social media forums, but they can serve many pragmatic functions. questions are often information-seeking, but social media posts also frequently contain questions that do not expect any information for what the question literally asks about. we will use the term rhetorical question (rq) broadly to refer to all questions that do not seek any information. for example, rhetorical questions can express criticism (e.g., "can't you do anything right?"), sentiment (e.g., "how fun is that?"), sarcasm (e.g., "who knew?"), and agreement/disagreement (e.g., "is the pope catholic?"). distinguishing rhetorical and information-seeking questions is important for dialogue processing and conversational analysis, but only recently has begun to receive attention in the nlp community. our research has two main contributions. first, we created a new resource for this understudied problem. we have compiled a collection of nearly 5,000 tweets that contain a question that is respond-ing to an initial tweet, and labeled the questions as rhetorical or information-seeking with crowdsourcing. we found that 53% of the questions are information-seeking (iq) and 47% are rhetorical (rq), confirming that both types of questions are prevalent in social media. second, our research examines whether the initial tweet prior to a question can help to predict whether a question is information-seeking or rhetorical. most prior work has focused only on the question itself, but we investigate whether the topic of the discussion may be a valuable indicator too. our intuition was that rhetorical questions are common in contexts associated with argumentation and debate, such as politics. conversely, we expect information-seeking questions to be prevalent in contexts about products and services, where people are actively seeking information. in this paper, we first describe our twitter dataset and human annotations. next, we present classification models that exploit both the question and the initial tweet prior to the question. we explore several ways of extracting topic information from tweets to capture the prior context. our results show that the prior context does improve performance for this task. ## related work rhetorical questions have been studied in linguistics, primarily focused on linguistic properties and pragmatic functions @xcite @xcite @xcite . however there has been relatively little work on rhetorical questions in the nlp community until recently. work by @xcite identified the information need of questions in twitter by extracting features from the question tweets. however, their work did not explore the usefulness of prior context in dis-tinguishing rhetorical questions from informationseeking questions. @xcite modeled the contextual overlap between a question and the most recent status message (mrsm) of the same user in twitter, with the hypothesis that a rhetorical question shares context with its mrsm more than a random question with its mrsm. @xcite found that n-gram features from utterances immediately preceding and following a question could help identify rhetorical questions. our work differs from both works in several aspects. first our evaluation dataset contains humanassigned gold labels and a rich mix of both rq and iq. in contrast, @xcite automatically assigned their dataset with labels according to some heuristic rules, which may be noisy, and @xcite used the switchboard dialog act corpus @xcite , where only 5% of questions are rhetorical. second, neither of these works took preceding context and topic information into account. @xcite studied rhetorical questions in the context of sarcasm in debate forums, but they did not study the problem of distinguishing rhetorical questions from information-seeking questions. in contrast, we focused on distinguishing the information need of general questions in twitter. @xcite further explored distinguishing rhetorical questions from informationseeking questions. but their gold standard data consists of rhetorical questions automatically extracted from debate forums using heuristic rules. in contrast, our gold standard data consists of questions that have been manually labeled as rhetorical or information-seeking. another difference is that @xcite did not consider the prior context for questions, which we focus on in this work. ## data we began by collecting tweets that contain question marks from january to december 2014. 1 we then applied a few filters to remove tweets that (1) are not in english (based on twitter's language code), (2) @xmath0, (3) are retweets or have quotation marks around the question, because these questions did not originate with the tweeter, (4) contain urls or media (e.g., photos), because the question may refer to the linked content, (5) contain multiple questions, which could be difficult to tease apart, or (6) were posted by a vip ("verified") account. questions posted by vip accounts (entities in the public interest) were predominately rhetorical questions in advertisements, and we did not want these to skew our data. we will refer to the resulting tweets as question tweets (qtweets). we also collected the preceding tweets, which we will refer to as prior tweet (ptweets). our hypothesis is that the preceding context can be important because (a) the question alone can be ambiguous, and (b) knowledge about the topic of discussion can affect the likelihood that a question will be rhetorical or information-seeking. consequently, we only kept question tweets that responded to a prior tweet. we also required that the prior tweet was the initial tweet in the conversational thread because conversational threads often have topic shifts and questions may refer back to earlier comments. detangling discourse threads is a challenging problem in its own right. this process produced 5,064 question tweets foot_1 , each paired with its prior tweet as context. ## classifying questions as rhetorical or information-seeking we designed a variety of classification models to assess the difficulty of distinguishing rhetorical and information-seeking questions, and to examine the role of prior context for this task. first, we applied the cmu twokenizer @xcite , removed urls and hashtags, and replaced acronyms with their corresponding full words or phrases using a twitter acronym list foot_3 . next, we applied the stanford corenlp parser @xcite to obtain lemmas and partof-speech tags. for the embedding features, we used glove vectors @xcite pretrained on 2b tweets. we experimented with both 25 and 100 dimensional vectors, and show the best results in section 5. we then extracted three sets of features: word features, question features, and topic features. ## experimental results we randomly split our data into 3 partitions: training (3,200), development (797), and test (1,000). all classifiers were trained on the training set and tuned with the development set. we report results on the test set as precision, recall, & f1 scores, all macro-averaged over the rq and iq classes. first, we evaluated models that used features derived only from the qtweet (qt). the first two rows of ## analysis to better understand how topics interact with rhetorical and information-seeking questions, we ana8 we also tried adding unigrams but the embedding worked better. 9 none of the question features other than the subjectivity features are applicable to ptweets lyzed the distribution of rq and iq over topics, based on the topic labels produced by the google content classifier applied to the psnippets from our training and development sets. we also manually inspected some questions that seemed to be difficult for our system to label correctly. ## conclusions a contribution of this work is that we have created a new dataset containing nearly 5,000 question tweets labeled as rhetorical or informationseeking coupled with their prior tweets. to our knowledge, this is the first twitter-based dataset for studying rhetorical questions that has both humangenerated gold labels and includes prior context for each question. we also presented classification models to benchmark performance on this task, and showed that including the tweet prior to a question improves performance. we also showed several ways to capture topic information, and that topic information represented in the preceding context seems to be useful for this task. our hope is that this work will lead to further research on the role of context for recognizing rhetorical and informationseeking questions in social media.