Interpreting Answers to Yes-No Questions in User-Generated Content (2023)

Abstract: Interpreting answers to yes-no questions in social media is difficult. Yes and no keywords are uncommon, and the few answers that include them are rarely to be interpreted what the keywords suggest. In this paper, we present a new corpus of 4,442 yes-no question-answer pairs from Twitter. We discuss linguistic characteristics of answers whose interpretation is yes or no, as well as answers whose interpretation is unknown. We show that large language models are far from solving this problem, even after fine-tuning and blending other corpora for the same problem but outside social media.

## introduction social media has become a global town hall where people discuss whatever they care about in real time. despite its challenges and potential misuse, the language of social media can be used to approach many problems, including mental health @xcite , rumor detection (ma and gao, 2020), and fake news @xcite . there is extensive literature on question answering, but few previous efforts work with social media. in particular, yes-no questions are unexplored in social media. figuring out the correct interpretation of answers to yes-no questions (yes, no, or somewhere in the middle) from formal texts @xcite , short texts @xcite , and conversations @xcite is challenging. this is the case in any domain, but especially in social media, where informal language and irony are common. additionally, unlike previous work on yes-no questions outside social media, we find that yes and no keywords are not only rare in answers, but also that they are poor indicators of the correct interpretation of the answer. consider the twitter thread in figure 1 . the question is mundane: whether people preheat the oven before using it. the answers, however, are complicated to decipher. the first answer states that it is very strange that somebody else (the author's girlfriend) does not preheat the oven. note that the correct interpretations is yes (or probably yes) despite the negation. the second answer explains why it is unnecessary to preheat the ovenwithout explicitly saying so-thus the correct interpretation is no. the third answer includes both yes and no yet the correct interpretation is unknown, as the author does not commit to either option. in this paper, we tackle the problem of interpreting natural answers to yes-no questions posted on twitter by real users, as exemplified above. the main contributions are: foot_0 (a) twitter-yn, a new corpus of 4,442 yes-no questions and answers from twitter along with their interpretations; (b) analysis discussing the language used in the answers, and in particular, showing that keyword presence in answers is a bad predictor of their interpretations, (c) experimental results showing that the problem is challenging-even for large language models, including gpt3-and blending with related corpora is useful; and (d) qualitative analysis of the most common errors made by our best model. motivation traditionally, question answering assumes that there are (a) correct answers to questions or (b) no answers @xcite . in the work presented in this paper, answers are available-the problem is to interpret what they mean. as figure 1 shows, answers capture personal preferences rather than correct answers supported by commonsense or common practice (e.g., oven manuals state that preheating is recommended). interpreting answers to yes-no questions opens the door to multiple applications. for example, dialogue systems for social media @xcite must understand answers to yes-no question to avoid inconsistencies (e.g., praising the author of the second reply in figure 1 for preheating the oven is nonsensical). aggregating interpretations could also assist in user polling @xcite . as the popularity of chat-based customer support grows @xcite , automatic interpretation of answers to yes-no questions could enhance the customer experience. lastly, interactive question answering and search, in which systems ask clarification questions to better address users' information needs, would benefit from this work @xcite . clarification questions are often yesno questions, and as we shall see, people rarely use yes, no or similar keywords in their answers. rather, people answer with intricate justifications that are challenging to interpret. ## previous work question answering outside social media has a long tradition. there are several corpora, including some that require commonsense @xcite or understanding tables @xcite , simulate open-book exams @xcite , and include questions submitted by "real" users @xcite . models include those specialized to infer answers from large documents @xcite , data augmentation (khashabi et al., 2020), multilingual models @xcite , and efforts to build multilingual models @xcite . none of them target social media or yes-no questions. yes-no questions and interpreting their answers have been studied for decades. several works target exclusive answers not containing yes or no @xcite . @xcite work with yes-no questions in the context of navigating maps @xcite and find that answers correlate with actions taken. more recent works @xcite target questions from swda @xcite . @xcite present yes-no questions submitted to a search engine, and louis et al. (2020) present a corpus of crowdsourced yes-no questions and answers. several efforts work with yes-no questions in dialogues, a domain in which they are common. annotation efforts include crowdsourced dialogues @xcite and transcripts of phone conversations @xcite and friends @xcite . question answering in the social media domain is mostly unexplored. tweetqa is the only corpus available @xcite . it consists of over 13,000 questions and answers about a tweet. both the questions and answers were written by crowdworkers, and the creation process ensured that it does not include any yes-no question. in this paper, we target yes-no questions in social media for the first time. we do not tackle the problem of extracting answers to questions. rather, we tackle the problem of interpreting answers to yesno questions, where both questions and answers are posted by genuine social media users rather than paid crowdworkers. ## twitter-yn: a collection of yes-no questions and answers from twitter since previous research has not targeted yes-no questions in social media, our first step it to create a new corpus, twitter-yn. retrieving yes-no questions and their answers takes into account the intricacies of twitter, while the annotation guidelines are adapted from previous work. ## analyzing twitter-yn twitter-yn is an unbalanced dataset: the interpretation of 70% of answers is either yes or no ( are keywords enough to interpret answers? no, they are not. we tried simple keyword-based rules to attempt to predict the interpretations of answers. the rules we consider first check for yes (yes, yea, yeah, sure, right, you bet, of course, certainly, definitely, or uh uh), no (no, nope, no way, never, or n't), and uncertainty (maybe, may, sometimes, can, perhaps, or if ) keywords. then they return (a) yes (or no) if we only found keywords for yes (or no), (b) probably yes (or probably no) if we found keywords for uncertainty and yes (or no), (c) unknown otherwise. as figure 3 shows, such a strategy does not work. many instances are wrongly predicted unknown because no keyword is present, and keywords also fail to distinguish between no and probably no interpretations. ## automatically interpretation of answers to yes-no questions we experiment with simple baselines, robertabased classifiers, and gpt3 (zero-and few-shot). experimental settings randomly splitting twitter-yn into train, validation, and test splits would lead to valid but unrealistic results. indeed, in the real world a model to interpret answers to yes-no questions ought to be trained with (a) answers to different questions than the ones used for evaluation, and (b) answers to questions from non-overlapping temporal spans. in the main paper, we experiment with the most realistic setting: unmatched question and unmatched temporal span (train: 70%, validation: 15%, test: 15%). we refer the reader to appendix c for the other two settings: matched question and matched temporal span and unmatched question and matched temporal span. baselines we present three baselines. the majority class always chooses the most frequent label in the training split (i.e., yes). the keyword-based rules are the rules presented in section 4. we also experiment with a version of the rules in which the lexicon of negation cues is replaced with a negation cue detector (hossain et al., 2020). ## results and analysis we present results in the unmatched questions and unmatched temporal span setting in regarding the baselines, using a negation cue detector to identify no keywords rather than our predefined list is detrimental (f1: 0.27 vs. 0.37), especially with yes label. this result leads to the conclusion that many answers with ground truth interpretation yes include negation cues such as affixes im-and -less, which are identified by the cue detector but are not in our list. zero-and few-shot gpt3 outperforms the baselines, and obtains moderately lower results than the supervised model trained on twitter-yn (question and answer; f1: 0.53 vs. 0.56). including the annotation guidelines in the zero and few-shot prompts obtains worse results (appendix g, the supervised roberta-based systems trained with other corpora yield mixed results. circa, swda-ia, and friends-qia are the ones which outperform the keyword-based baseline (0.53, 0.47 and 0.51 vs. 0.37). we hypothesize that this is due to the fact that boolq includes yes-no questions from formal texts (wikipedia). unsurprisingly, training with twitter-yn yields the best results. the question alone is insufficient (f1: 0.29), but combining it with the answer yields better results than the answer alone (0.56 vs. 0.48). it is observed that friends-qia and circa are the only corpora worth blending with twitter-yn. both obtain better results than training with only twitter-yn (f1: 0.58 in both vs. 0.56). finally, including in the input the disambiguation texts automatically generated by gpt3 further improves the results (f1: 0.61) blending with circa. on the other hand, disambiguation texts are detrimental when blending with friends-qia. these results were surprising to us as disambiguation texts are often nonsensical. appendix g ( regarding statistical significance, it is important to note that blending circa and using disambiguation texts yields statistically significantly better results than few-shot gpt3. gpt3, however, is by no means useless: it provides a solid few-shot baseline and the disambiguation texts can be used to strengthen the supervised models. error type question answer p,g yes distractor y,n (4.4%) no distractor n,y (28.6%) social media u,y lexicon (20.7%) humor pn,py (13.2%) condition or u,py contrast (29.1%) external entities u,y (14.9%) unresponsive y,u (11.4%) ## conclusions we have presented twitter-yn, the first corpus of yes-no questions and answers from social media annotated with their interpretations. importantly, both questions and answer were posted by real users rather than written by crowdworkers on demand. as opposed to traditional question answering, the problem is not to find answers to questions, but rather to interpret answers to yes-no questions. our analysis shows that answers to yes-no questions vary widely and they rarely include a yes or no keyword. even if they do, the underlying interpretation is rarely what one may think the keyword suggests. answers to yes-no questions usually are long explanations without stating the underlying interpretation (yes, probably yes, unknown, probably no or no). experimental results show that the problem is challenging for state-of-the-art models, including large language models such as gpt3. our future plans include exploring combination of neural-and knowledge-based approaches. in particular, we believe that commonsense resources such as conceptnet @xcite , atomic @xcite , and common-senseqa @xcite ) may be helpful. the challenge is to differentiate between common or recommended behavior (or commonly held commonsense) and what somebody answered in social media. going back to the example in figure 1 , oven manuals and recipes will always instruct people to preheat the oven. the problem is not to obtain a universal ground truth-it does not exist. instead, the problem is to leverage commonsense and reasoning to reveal nuances about how indirect answers to yes-no questions ought to be interpreted.