Disentangling Indirect Answers to Yes-No Questions in Real Conversations (2022)

Abstract: In this paper, we explore the task of determining indirect answers to yes-no questions in real conversations. We work with transcripts of phone conversations in the Switchboard Dialog Act (SwDA) corpus and create SwDA-IndirectAnswers (SwDA-IA), a subset of SwDA consisting of all conversations containing a yes-no question with an indirect answer. We annotate the underlying direct answers to the yes-no questions (yes, probably yes, middle, probably no, or no). We show that doing so requires taking into account conversation context: the indirect answer alone is insufficient to determine the ground truth. Experimental results also show that taking into account context is beneficial. More importantly, our results demonstrate that existing corpora with synthetic indirect answers to yes-no questions are not beneficial when working with real conversations. Our best models outperform the majority baseline by a substantial margin, but the task remains a challenge (F1: 0.46).

## introduction dialogue systems have become a reality enabled by large datasets and deep neural networks. taskoriented @xcite and open-domain @xcite dialogue systems are commonplace. neural approaches are popular @xcite although systems based on logical inference and rules outperform networks in open-domain dialogue @xcite . state-of-the-art systems face challenges keeping track of a conversation and avoiding inconsistencies @xcite . evaluation is also an open issue as automated metrics have several drawbacks @xcite ). an alternative is to evaluate dialogue systems based on whether they can collaborate with humans to solve a problem @xcite or elicit some action such as donating to a cause @xcite . people do not explicitly say what they mean when they speak to each other yet they seamlessly a: do you work outside of the home? b: no, i am not working currently. ## previous work yes-no questions and indirect answers have been studied for decades @xcite . @xcite report that 27% of all questions in 18 hours of spontaneous speech are yesno questions with indirect answers @xcite . indirect answers are often used to ask follow-up questions or provide explanations for negative answers @xcite , prevent wrong interpretations of direct answers @xcite or show politeness @xcite . yes-no questions have received considerable attention recently. outside of the dialogue genre, @xcite present boolq, a corpus of 16,000 yes-no questions submitted to a search engine (e.g., does france have a prime minister and a president?) along with wikipedia articles that may contain an answer. two recent corpora consist of dialogues generated by crowdworkers on demand after being given some text to talk about: quac @xcite and coqa @xcite . unlike the work presented here, the dialogues in these corpora are limited to questions and answers about the text provided to crowdworkers. more importantly, we work with unrestricted, natural conversations rather than dialogues between paid annotators who are asked to have a conversation that resembles a questionnaire geared towards checking for understanding of a piece of text. yes-no questions in real conversations have been studied before at a small scale. de @xcite work with 224 questions-answer pairs involving gradable adjectives. we borrow from them the five labels to annotate underlying direct answers. de @xcite present a typology for 623 yes-no questions from swda. our corpus is four times larger. unlike them, we show that context is crucial to determine underlying direct answers and present experimental results. the work by @xcite is the closest to ours. they present circa, 34,268 yes-no questions and indirect answers written by crowdworkers given one of ten scenarios (e.g., scenario: talking to a friend about music preferences, q: do you like guitars? a: i practice playing every weekend). unlike us, circa assumes that the turn following a yes-no question is enough to determine the underlying direct answer. furthermore, we show that (a) determining underlying direct answers to yesno questions is much harder in real conversations and (b) their synthetic data only bring minor improvements when working with real conversations. ## swda-ia: a corpus of yes-no questions and indirect answers we present swda-ia, a corpus consisting of (a) the 2,544 yes-no questions in switchboard with indirect answers and (b) manual annotations indicating the underlying direct answers. three characteristics set our work apart. first, we work with real conversations as opposed to artificial, synthetic ones (section 2). second, we show that determining underlying direct answers requires taking into account context beyond the yes-no question and the next turn. third, annotators pinpoint which turns within the context around a yes-no-question are useful to determine the underlying direct answer. ## corpus analysis: the role of context the annotations we collected in both phases differ substantially. thus context beyond the yes-no question and indirect answer (i.e., the following turn) is needed to determine the ground truth in real conversations. to our knowledge, previous work does not take into account context (section 2). we show examples of changes in ground truth when annotators do not have and have access to context in the third example (bottom left) shows how follow-up questions in context can clarify the under- ). determining the underlying direct answers requires taking into account more conversation context than the question and indirect answer: the ground truth annotated by humans changes depending on whether they also have access to context (t -3 , t -2 , t -1 , t 2 , and t 3 ). which turns around a yes-no question are the most important? we show which turns annotators selected as useful to determine the underlying direct answers in figure 4 . unsurprisingly, the turn immediately after the yes-no question (after 1 ) is almost always useful (96.4%). the next turns by either speaker are often useful, including the follow-up turn by the speaker asking the yes-no question (after 2 , 20.9%). the turns before the yesno question (prev 3 -prev 1 ) are less often important. we also found that the turn before the yes-no question is the most useful (prev 1 , 12.5%) out of all the turns before the question. note that annotators almost always (96.4%) deem useful the turn immediately following the ## experiments and discussion we split swda-ia into training, development and test splits (70/15/15) randomly but making sure no split contains yes-no questions from the same conversation. we consider each conversation snippet (yes-no question, three turns before, and three turns after) an instance, and build models to predict the underlying direct answer. following previous work on (short) text classification and specifically on the same task with synthetic data @xcite , we build transformer-based classifiers fine-tuned using several strategies. we experimented with three transformers: bert @xcite , roberta @xcite , and tod-bert @xcite released by huggingface @xcite . note that the first two are pretrained with general-purpose english while the third is fine-tuned for task-oriented dialogues and includes an attention mechanism designed to keep track of dialogues. while the three transformers obtained roughly the same results (within 0.02 f1), roberta outperformed the other two and we only report results with roberta. the ground truth depends on context beyond the yes-no question and indirect answer (section 4). a roberta-based classifier, however, may not benefit from context. in order to experiment with context, we conduct three types of experiments: feeding to the models (a) only the question and the turn immediately following the question (i.e., indirect answer), (b) the full context (i.e., seven turns), and (c) only the annotator-selected turns during the annotation process. we use a separator token to indicate a new turn in the input sequence. fine-tuning roberta we fine-tuned the roberta-based classifier using corpora for related tasks and swda-ia, our corpus. specifically, we used the following corpora in our experiments: mnli is a corpus for natural language inference @xcite . it contains premisehypothesis pairs where the premise entails, is neutral with respect to, or contradicts the hypothesis. it contains 392k pairs for training, 9k for development, and 9k for test. the fine-tuned roberta classifier obtains 83% development accuracy. following @xcite , we rewrite questions into declarative statements and map entailment to yes, neutral to middle, and contradiction to no. boolq is a corpus for yes-no question answering @xcite . questions were submitted to a search engine and wikipedia articles containing and not containing the answer are included for each question. it contains 9.4k questions for training, 3.2k for development, and 3.2k for test. the fine-tuned roberta classifier with boolq obtains 73% development accuracy. we map their yes and no (correct and incorrect) to our yes and no. circa is a corpus with yes-no questions and indirect answers @xcite . unlike swda-ia, which includes questions and answers from real conversations, circa collected questions and answers from crowdworkers who were given one of ten scenarios (section 2). in other words, circa does not include naturally occurring questions and answers. additionally, circa does not consider context as all answers are a single turn. circa contains 3,431 yes-no questions and up to 10 indirect answers for each (total: 34,268 question-answer pairs). the fine-tuned roberta classifier with circa obtains 78% development accuracy. we map their middle and i am not sure to our middle. our corpus: swda-ia we consider three versions of fine-tuning with our corpus: taking into account only the question, only the answer, or both (qa). intuitively, the indirect answer should be the most useful to determine the underlying direct answer but the question may also help. ## conclusions yes-no questions with indirect answers are common in real conversations-77.4% of 5-minute phone conversations include at least one. in this paper, we investigate the underlying direct answers to such questions in real conversations. we have presented swda-ia, the first corpus with annotations for this task on top of real conversations. we show that determining the underlying direct answer requires taking into account context, as the ground truth changes depending on whether we show annotators context around the question. our analysis also shows that context after the question is more useful, including turns by both the author of the question and the other speaker. experimental results demonstrate that solving the task with real conversations is challenging (f1: 0.46). more importantly, (a) doing so barely benefits from fine-tuning with related tasks (mnli and boolq) and (b) circa, a corpus of synthetic questions and synthetic indirect answers, barely outperforms the majority baseline with real conversations. our future plans include designing models that better encode conversational context in order to obtain better results. we are also interested in exploring applications in dialogue systems to avoid inconsistencies due to misunderstandings of indirect answers to yes-no questions.