Interpreting Answers to Yes-No Questions in Dialogues from Multiple Domains (2024)

Abstract: People often answer yes-no questions without explicitly saying yes, no, or similar polar key-words. Figuring out the meaning of indirectanswers is challenging, even for large language models. In this paper, we investigate this problem working with dialogues from multiple domains. We present new benchmarks in three diverse domains: movie scripts, tennis interviews, and airline customer service. We present an approach grounded on distant supervision and blended training to quickly adapt to a new dialogue domain. Experimental results show that our approach is never detrimental and yields F1 improvements as high as 11-34%.

## introduction while state-of-the-art models obtain results as high as 93% f1 @xcite with questionanswering benchmarks such as squad @xcite , challenges remain. for example, natural questions submitted to search engines remain challenging @xcite . similarly, existing models face challenges with open-ended questions checking for comprehension @xcite , yes-no questions that require deriving an answer (yes or no) from text @xcite , false presuppositions and assumptions in the question @xcite , and negation @xcite . many questions in dialogues expect a yes or no for an answer. yet many follow-up turns answer this kind of questions without explicitly saying yes, no, or similar polar keywords. @xcite analyze 18 hours of speech @xcite and report that 27% of questions fall in this category. indirect answers to yes-no questions are used to ask follow-up questions or provide explanations for negative answers @xcite , prevent incorrect interpretations of direct answers @xcite , or show politeness @xcite . a1: i understand, but i noticed that the fire marshall is here with you. is this somehow related to the fire department? b1 : i really can't give out any information right now at this point. a2: okay. but i do understand that your partner, leon jackson's been injured. is that correct? b2 : he was hurt, but not seriously. he'll be fine. a3: do you have the suspect in custody? b3 : [. . . ] is not a good time, okay. detective jackson's hurt. he's fine. i've got a fire marshall shot, detective jackson is hurt but not seriously. figure 1 : movie dialogue with three yes-no questions (a 1 , a 2 , and a 3 ). answers are indirect as they do not include polar keywords (yes, no, etc.). in b 1 and b 3 , the author declines to answer, whereas in b 2 the author indirectly answers no by minimizing the incident (injured requires loss of function, while hurt does not). consider the dialogue from the movie 15 minutes in figure 1 . none of the questions are answered explicitly. speaker b declines to answer the first and third questions. the answer to the first question states that b is not allowed to answer. the answer to the third question restates information known from previous utterances but provides no answer. on the other hand, the answer to the second question implicitly denies that leon was injured by stating that he was (only) not seriously hurt. this paper tackles the problem of interpreting indirect answers to yes-no questions (i.e., answers that do not contain yes, no, or other polar keywords). our contributions are: 1 1. demonstrating that the problem of identifying yes-no questions in dialogues can be automated with high precision, even in out-ofdomain dialogues; 2. creating three new benchmarks (300 instances each) to evaluate models to interpret answers to yes-no questions in three domains; ## terminology and existing corpora we use the term yes-no question to refer to a question that expects a yes or no for an answer. answers to yes-no questions may not include yes, no, or other polar keywords (e.g., positive: sure, of course, etc.; negative: not at all, no way, etc.). we refer to answers with and without polar keywords as direct and indirect answers. we make a distinction between the source of dialogues-who the speakers are and why they communicate. we use the term synthetic dialogue to refer to dialogues between people who are instructed (and usually paid) to talk about a given topic. the speakers in synthetic dialogues include crowdworkers. we use genuine dialogue to refer to naturally-occurring dialogues between people. finally, in this paper we work on two problems related to yes-no questions. given a dialogue, identifying yes-no questions pinpoints where the yes-no questions are. on the other hand, interpreting answers to yes-no questions figures out the underlying meaning of the answer (yes, no, or middle). unlike traditional question answering, answers are readily available-the problem is to figure out what the answer means. existing corpora we work with several existing dialogue corpora in multiple domains. in order to identify yes-no questions, we work with the following as training corpora (in-domain): swda @xcite , telephone conversations with dialogue act annotations (122k turns); mrda @xcite , meeting transcripts with dialogue act annotations (43k turns); daily-dialog @xcite , multi-turn conversations written by crowdworkers to simulate human daily conversations (87k turns); friends @xcite , scripts of the tv show (58k turns); and mwoz @xcite , task-oriented dialogues written by crowd workers (105k turns). for evaluation purposes (out-of-domain), we use the following: tennis @xcite , transcripts of postmatch interviews of tennis players (164k turns); movie @xcite , movie transcripts (304k turns); and air @xcite , task-oriented dialogues with topics limited to travel and flights (3,805k turns). in order to interpret answers to yes-no questions, we work with the following as training corpora: circa @xcite , 34k yes-no questions and indirect answers written by crowdworkers; and swda-ia @xcite , 2.5k yesno questions and indirect answers from the swda. both corpora include manual annotations of the interpretations of answers. for evaluation purposes, we use the same corpora than for identifying yes-no questions: tennis, movie, and air. specifically, we create new benchmarks (300 questions and indirect answers from each corpus) and use the rest (questions and direct answers) for training purposes via distant supervision. ## related works yes-no questions have received considerable attention recently. boolq @xcite is a collection of 16,000 yes-no questions and wikipedia articles from which answers (yes or no) can be derived. @xcite enhance boolq with questions that cannot be answered. unlike them, in this paper we target yes-no questions in dialogues, which are more open-ended (figure 1 ) than the fact-seeking questions (e.g., has the uk been hit by a hurricane?). two recent works target yes-no questions in dialogues @xcite . unlike us, both of them work with synthetic dialogues written by crowdworkers and are constrained to a handful of scenarios. yes-no questions in genuine dialogues have been studied before. the work presented here is closest to circa @xcite , direct @xcite , and swda-ia @xcite . unlike us, circa works with synthetic yes-no questions and answers without any conversational context. direct also works with synthetic dialogues. swda-ia works with telephone conversations from swda. to our knowledge, we are the first to explore yes-no questions in multiple dialogue domains. we show that existing corpora are beneficial, and more importantly, that combining additional training data obtained via distant supervision in the new dialogue domains brings additional improvements across all domains. ## identifying yes-no questions we first tackle the problem of identifying yes-no questions in dialogue. to our knowledge, previous work on yes-no questions is limited to interpreting the answers. we first present our rule-based approach to collect yes-no questions. then, we describe how to leverage these rules to build a classifier to automate the task. ## interpreting answers to yes-no questions armed with the highly precise classifier to identify yes-no questions, we move to interpreting answers to yes-no questions in dialogues from multiple domains. to our knowledge, there are two publicly available corpora: circa and swda-ia (section 2). we aim to explore multiple dialogue domains, so we first create new benchmarks. three new benchmarks we create three new benchmarks for evaluation purposes in new domains. specifically, we randomly select 300 yesno questions followed by an indirect answer from each corpus (tennis, movie, and air; 900 total). then, we manually annotate the interpretation of the answer using three labels: yes, no, or middle. our definition of yes includes what circa and swda-ia define as probably yes, which include sometimes yes and yes under certain conditions. for example, we annotate q: do you like mexican food? a: i am fine with tacos if my friends suggest mexican with yes. similarly, our definition of no includes what others define as probably no. for example, we annotate q: do you want to go out for dinner? a: i have a deadline and i may skip dinner with no. on the other hand, our definition of middle follows previous work: we choose it when the answer does not lean toward yes or no. as we shall see, however, we benefit from using distant supervision with all the dialogue corpora. inter-annotator agreements the three benchmarks were annotated in-house by two graduate students. inter-annotator agreements (cohen's κ) for tennis, movie, and air are 0.68, 0.69, and 0.66 respectively. these coefficients indicate substantial agreement @xcite ; above 0.8 would be (nearly) perfect. 87.0% of disagreements are between (a) yes or no and (b) middle, while only 13.0% are between yes and no. these percentages suggest that most disagreements are minor. after annotating individually, the annotators discussed the disagreements in order to adjudicate them and create the final ground truth. we refer the reader to appendix b for more details. ## conclusions indirect answers to yes-no questions in dialogue are common. in this paper, we have presented an approach to identify yes-no questions in dialogues (distant supervision and bert), and more importantly, to interpret indirect answers to yesno questions. experimental results show for the first time that the identification problem is rather simple. the second problem, on the other hand, remains challenging-f1 scores are 0.49 and 0.59 with movies and tennis. these results lead to the conclusion that synthetic dialogues may not be representative of more open-ended conversations. crucially, we have shown that distant supervision to obtain additional examples with direct yes and no answers is beneficial to interpret indirect answers. indeed, combining the additional instances with blended training is never detrimental and yields substantial improvements with our new, out-of-domain benchmarks (tennis, movie, and air). in other words, the proposed methodology can be used to adapt to new domains without requiring substantial human involvement, unlike annotating additional examples. our future plans include addressing the most common errors. in particular, we believe that exploring dialogue coherence and pretrained language models customized to the dialogue domain are two lines of research worth exploring.