“ I ’d rather just go to bed”: Understanding Indirect Answers (2020)

Abstract: We revisit a pragmatic inference problem in dialog: Understanding indirect responses to questions. Humans can interpret ‘I’m starving.’ in response to ‘Hungry?’, even without direct cue words such as ‘yes’ and ‘no’. In dialog systems, allowing natural responses rather than closed vocabularies would be similarly beneficial. However, today’s systems are only as sensitive to these pragmatic moves as their language model allows. We create and release the first large-scale English language corpus ‘Circa’ with 34,268 (polar question, indirect answer) pairs to enable progress on this task. The data was collected via elaborate crowdsourcing, and contains utterances with yes/no meaning, as well as uncertain, middle-ground, and conditional responses. We also present BERT-based neural models to predict such categories for a question-answer pair. We find that while transfer learning from entailment works reasonably, performance is not yet sufficient for robust dialog. Our models reach 82-88% accuracy for a 4-class distinction, and 74-85% for 6 classes.

## introduction humans produce and interpret complex utterances even in simple scenarios. for example, for the polar (yes/no) question 'want to get dinner?', there are many perfectly natural responses in addition to @xcite in previous attempts to interpret indirect yes/no answers have been small scale and without datadriven techniques @xcite . however, recent success on many language understanding problems @xcite , the impressive generation capabilities of modern dialog systems @xcite , as well as the huge interest in yes/no question-answering @xcite have created a conducive environment for revisiting this hard task. we introduce circa, a new dataset with 34k pairs of polar questions and indirect answers in the english language. this high quality corpus consists of natural responses collected via crowd workers, and goes beyond binary yes/no meaning to include conditionals, uncertain, and middle-ground answers. circa contains many phenomena of interest, although the first step, which we address here, is how to robustly classify a question-answer pair into one of the above meaning categories. we find that bert @xcite fine-tuned on entailment data is an effective initial approach, mirroring the success in question-answering work involving yes/no questions @xcite . it reaches an accuracy of 85-88% for responses in the same situational context, and 6-10% lower accuracy on held-out scenarios. the answer text itself (as in 'i would like that') contains strong cues leading to 78-82% accuracy, however the best results come from jointly analyzing the question and the answer. ## related work indirect answers to polar questions are reasonably frequent and warrant deep study. early work put the proportion at 13% in face-to-face and telephone conversations @xcite , and at 27% in an instruction giving/following map task @xcite . for a more recent and larger analysis, consider the cornell movie dialog corpus @xcite . we heuristically mined yes/no questions and their following utterances, finding 6,327 pairs. direct answers (i.e., answers with 'yes', 'no', 'maybe' and related terms such as 'okay', 'yup', etc.) only cover 53% of the pairs. this suggests that indirect responses could be even more frequent in natural open-domain dialogue. even when a direct answer is possible, speakers use indirect answers to be cooperative and address anticipated follow-up questions @xcite , to provide explanations in the case of a negative answer @xcite , to block misleading interpretations that may arise from a curt 'yes' or 'no' reply @xcite , and since indirect answers may appear more polite @xcite . but we lack a large corpus of such answers to study these multiple pragmatic functions. our work aims to fill this gap. on the computational side, there are impressive efforts towards planning, generation, and detection of indirect answers, albeit, on a small scale, and without data-driven approaches. @xcite 's early work leverages discourse relations for generating indirect answers. for example, an 'elaboration' may be relevant for a 'yes' response, and a 'contrast' might help convey a 'no' answer. de @xcite reason about such answers using markov logic networks. in subsequent work, de @xcite present one of the first data-driven studies into indirect answers containing scalar adjectives. they mine a set of 224 question-answer pairs from interview transcripts and dialog corpora. using polarity information from review data, and manual coding of test samples, they achieve an accuracy of 71% on three classes 'yes', 'no' and 'uncertain'. our work aims to collect a much larger and more diverse natural corpus, and demonstrates the first automatic approach using recent advances in natural language inference (nli). de marneffe et al. (2010) also demonstrated the first crowd annotation of indirect answers, and we draw on many aspects of their formulation for the creation of our corpus. an unexpected limelight on yes/no questions has also arisen in recent question-answering (qa) work. researchers have noticed that yes/no questions are complex and naturally arise (as high as 20%) when questions are posed one after the other in a conversation @xcite . their goal is to produce direct 'yes' or 'no' answers, but obtaining them requires inference against a paragraph or excerpt, an analogous task to our yes/no inference from indirect answers. very recent work @xcite has specifically sought to improve this ability in qa systems, by building a corpus of 16k yes/no factual questions paired with wikipedia passages from which the yes/no answer can be inferred. departing from factual texts, our focus is on single-turn conversational responses in everyday situations. the latter are faithful, cooperative, and grounded in world knowledge. still, transfer learning from factual corpora could prove useful and we explore this too. ## the circa corpus circa (meaning approximately) is our crowdsourced corpus for research on indirect answers. it contains 34,268 question-answer pairs, comprising 3,431 unique questions with up to 10 answers to each. each question-answer pair is also annotated with meaning based on a fixed set of categories. 2 we explain all aspects of the collection process below, but in the interest of space, further details (complete annotator instructions, interfaces, and examples) are in the appendix. our data consists of very short dialogues, each containing a question and its indirect answer. the texts are varied in semantic and syntactic forms, and grounded in 10 different situational contexts. s1: talking to a friend about food preferences. q: "do you like pizza?" a: "i like it when the toppings are meat, not vegetable." s2: talking to a friend about music preferences. q: "do you like guitars?" a: "i practice playing each weekend." s3: talking to a friend about weekend activities. q: "are you available this sunday evening?" a: "what did you have in mind?" s4: talking to a friend about book preferences. q: "are you a fan of comic books?" a: "i read an archie every time i have lunch." s5. your friend is visiting from out of town. q: "would you like to go out for dinner?" a: "i could go for some mexican." s6. two colleagues leaving work on a friday. q: "long week?" a: "i've had worse weeks." s7. you friend is planning to buy a flat in new york. q: "does the flat's price fit your long-term budget?" a: "well, if it doesn't i will definitely refinance my mortgage." s8. your friend is thinking of switching jobs. q: "do you have to travel far?" a: "my commute is about 10 minutes." s9. two childhood neighbours uexpectedly run into each other at a cafe. q: "are you going to the high school reunion in june?" a: "i forgot all about that." s10. meeting your new neighbour for the first time. q: "did you move from near-by?" a: "i am from canada." ## learning task obviously answers contain numerous cues for subsequent dialog flow, but in this first work we focus on meaning prediction. specifically, given a question-answer pair, we classify it into one of the meaning categories in tables 7 and 8 . we consider two experimental settings: in the matched setup, we assume that the response scenario is seen during training (randomly dividing our corpus examples into 60% training, 20% each for development/test). the unmatched setting is aimed at understanding the performance on unseen scenarios, i.e., whether models can learn the semantics of yes/no answers beyond the patterns specific to individual scenarios. as our data contains 10 scenarios, we carry out 10 leave-one-out tasks, each time holding out one scenario (for example, 'buying a flat in new york') as the test data, and use the remaining nine for training and development. for both the matched and unmatched setting, we consider two variants of the classification problem: strict (with 6 different labels, namely all except 'other' and 'n/a' in for the unmatched setting, these sizes are the average across the 10 leave-one-out sets. ## models building upon recent nli systems, our approach leverages representations from unsupervised pretraining, and finetunes a multiclass classifier over the bert model @xcite . however, we first consider other models for related tasks. ## experiments we use pre-trained bert models (with 12 layers, 768 hidden units, and 12 attention heads, 110m parameters) for all our experiments. the experiments were done on a single cloud tpu, and finetuning on our corpus takes under 30 minutes. ## conclusion we have presented a new dataset containing natural indirect yes/no answers, as well as other significant pragmatic moves in the form of conditionals and uncertain utterances. our first approach towards automatic interpretation is promising, but there is a significant gap especially for examples outside training scenarios. our model does not yet classify additional information in responses ('dinner? let's go for italian.' indicates not only a 'yes' answer, but also a preference for italian food.). moreover, we have explored the phenomena in english. there are exciting avenues for multilingual work to account for language and cultural differences.