Unravelling Indirect Answers to Wh-Questions: Corpus Construction, Analysis, and Generation (2023)

Abstract: Indirect answers, crucial in human communication, serve to maintain politeness, avoid conflicts, and align with social customs. Although there has been a substantial number of studies on recognizing and understanding indirect answers to polar questions (often known as yes/no questions), there is a dearth of such work regarding wh-questions. This study takes up the challenge by constructing what is, to our knowledge, the first corpus of indirect answers to wh-questions. We analyze and interpret indirect answers to different wh-questions based on our carefully compiled corpus. In addition, we conducted a pilot study on generating indirect answers to wh-questions by fine-tuning the pre-trained generative language model DialoGPT (Zhang et al., 2020). Our results suggest this is a task that GPT finds difficult.

## introduction indirect answers (inds) to questions hold a distinctive position in the realm of human communication, as they provide related or implied information instead of offering the speaker's intentions or knowledge directly through an utterance's grammatically governed content. (i.e., literal content) @xcite . grasping the intrinsic nuances of indirect answers and accurately deducing the expected direct answer from them is essential to facilitate effective communication and information sharing between dialogue participants. it is a natural part of human communication to produce and understand indirect answers. people use indirect speech to maintain politeness, avoid confrontations, adhere to social norms, or convey information without explicitly stating it @xcite . however, understanding and generating indirect answers to questions can be quite challenging for dialogue systems. to engage in human-like conversation, these systems must be able to grasp the conversational context, background information, and relationships between participants. by accurately interpreting the meaning behind an indirect answer, the system can then provide a more appropriate response, contributing to a more natural interaction. in the field of dialogue studies, considerable attention has been given to the interpretation and generation of indirect answers to polar questions @xcite @xcite @xcite @xcite . however, there still exists a gap when it comes to the identification and interpretation of indirect answers to wh-questions. studying indirect answers to wh-questions is a challenging task for several reasons: a). unlike polar questions that have only yes or no (or rather the propositions they convey in context) as direct, resolving answers, wh-questions can have a wide range of possible direct answers. this makes it harder to interpret indirect answers to wh-questions; b). compiling a corpus of indirect answers to wh-questions is a challenging task, since indirect answers to wh-questions are significantly less frequent than those of polar questions. it requires annotating a huge number of wh-questions within conversational context to collect a reasonable amount of whq-ind pairs for analysis and training machine learning algorithms; c). the implied meaning of indirect answers to wh-questions often depends heavily on the context of the conversation. it usually also involves nuanced linguistic features like sarcasm, irony, and figurative expressions which can be a challenge for humans (overhearers) to interpret, let alone for dialogue systems. the aim of this paper is, therefore, to conduct a preliminary study by constructing what is, to our knowledge, the first corpus of indirect answers to wh-questions, and to investigate how direct answers are deduced from indirect answers. this paper is structured as follows: section 2 consists of a literature review, whereas section 3 provides the requisite theoretical background. in section 4, we present the data collection and annotation process. in section 5 we propose possible information resources needed for interpreting indirect answers to wh-questions. following this, in section 6 we briefly describe a pilot study on generating indirect answers by using a pre-trained language model. our results suggest this is a task that gpt finds difficult. the final section offers conclusions and some potential future work. ## related work several studies exist concerning the interpretation and generation of indirect answers to polar questions: @xcite proposed both pragmatic and computational methods for understanding and generating indirect answers to polar questions. specifically, they introduced a discourse-plan-based strategy for implicatures and a combined reasoning model to simulate a speaker's incentive for offering pertinent, unsolicited information. furthermore, they designed a computational model that is capable of interpreting and generating indirect answers to polar questions in english. their model relies on shared knowledge of discourse strategies and coherence relations to recognize and formulate a responder's discourse plan for a complete response. @xcite released the corpus di-rect, which provides 71,498 indirect-direct pairs together with multi-turn dialogue history extracted from the multiwoz dataset, and conducted three experiments to examine the model's ability to recognize and generate indirect and direct utterances. the direct corpus provides triples of paraphrases for each user's utterance: original utterance, indirect utterance, and direct utterance. this is the first study that offers a large-scale corpus of pragmatic annotations, which is very useful for understanding users' intentions in dialogue systems. in another recent work, @xcite created and released the first large-scale english corpus of more than 34k polar question-indirect answer pairs, named circa. that is a collection of natural responses obtained by crowd-sourcing and contains responses with yes-no meaning, as well as uncertain, middle-ground, and conditional responses. the authors also conducted experiments by fine-tuning a multiclass classifier over the bert model @xcite , and then further finetuned those models with polar question-answer pairs from the circa corpus. they examined the performance of different models for the classification of polar question-indirect answer pairs into the following meaning categories: 1. strict labels: yes; no; probably yes / sometimes yes; yes, subject to some conditions; probably no; in the middle; neither yes nor no; i am not sure; other; n/a., and 2. relaxed labels: yes; no; yes, subject to some conditions; in the middle, neither yes nor no; other; n/a. 1 the study evaluated various baseline models and compared the performance of the models using only questions, only answers, and both questions and answers. the results indicated that joint models (that is, models trained both with questions and answers) outperformed answer-only models. the study also highlighted the challenges of classifying uncertain or ambiguous responses and suggested that incorporating the right information for the task remains a challenge. taking inspiration from the research of @xcite , @xcite studied how to understand indirect answers to polar questions. instead of crowdsourcing, they collected polar questions and indirect answers from the transcripts of the friends tv series. after manual annotations, they released the friends-qia dataset with 5,930 polar question-indirect answer pairs in english, both with the majority label and with the raw annotations. they further experimented with convolutional neural networks (cnns) with different word embeddings: cnn with glove embeddings and cnn with bert embeddings. furthermore, an additional crowd layer was added to enable the model to learn from the disagreement of human annotators. as a result, cnns trained with bert embeddings outperformed cnns trained with glove word embeddings when the model was trained both with questions and answers. furthermore, using convolutional neural networks (cnns) to evaluate the task, the authors showed that there was still room for improvement in the interpretation of indirect answers. however, they also found encouraging improvements when explicitly modeling human disagreement in the annotations. ## background the taxonomy of the response space to questions we use is formally characterized using the kos framework @xcite which provides a theory of dialogue context and dialogue management. the question-specific responses are the most important subgroup of the taxonomy of the response space to questions. this includes responses providing answers (direct answers and indirect answers), and dependent questions where the response to the original question depends on the response to the question-response to that original question. other subgroups of the taxonomy are the metacommunicative responses (clarification response and acknowledgement), and the evasion responses (motivation, ignore, change the topic, and difficult to provide an answer). detailed descriptions of each class are presented in appendix b. direct answers are defined as those that, given a proposition: p, a question: q, p is a direct answer to q, if and only if p is about q, and is entailed by either the meet of q's atomic or negative atomic answer set. 2 indirect answers are distinguished from direct answers under two basic conditions: a). the indirect answer p is not a direct answer to the question q, and b). the indirect answer p, together with a bridging proposition bridgeprop (some shared knowledge), entails r, which is a direct answer to the question q. the formal definition of indirect answers is stated as follows: given p : p rop, q : question, dgb : ## corpus collection we aim to collect the first publicly available corpus of indirect answers to various content questions in english dialogue. to start with, we follow the annotation guidelines for the entire response space of the questions presented in previous works by @xcite , and also updated their annotation guidelines by adding extra instructions specific to indirect answers to wh-questions. we annotated various wh-questions and their corresponding responses from four different english corpora. namely, bnc (burnard, 2007), cornell-moive corpus @xcite , coca (the corpus of contemporary american english, davies, 2010), and llc (the london-lund corpus of spoken english, @xcite . ## generation of inds to wh-questions as a pilot study, we fine-tuned the pretrained response generation model di-alogpt (medium) @xcite with our collected corpus of indirect answers to wh-questions (458 examples), and tested the fine-tuned model's ability to generate indirect answers to wh-questions in a new test set. ## conclusions and future work in this paper, we have addressed the challenge of interpreting indirect answers to wh-questions. we started by collecting indirect answers to wh-questions from four different english corpora (bnc, cornellmovie, coca, and llc), and constructed a small corpus of 458 whq-ind pairs along with pre-question utterances and postresponse utterances. building such a corpus is highly labour intensive, given the difficulty of the task of classifying responses as indirect, as we demonstrated in several inter-annotator studies, including ones involving expert annotators. in addition, we developed a scheme of 9 possible information sources used to infer direct answers from indirect answers and found-provisionally, given problems with the imbalanced nature of our data set-that basic linguistic knowledge, common sense, and nuanced linguistic knowledge are the three most frequently used information sources for the interpretation of indirect answers to whquestions. finally, we also conducted a preliminary experiment for generating indirect answers to wh-questions by fine-tuning a large-scale response generation language model, dialogpt. the results of this latter experiment are hampered by the small amount of our current data set, but also suggest that this is a tricky task for gpt. there are several clear limitations of the current study, which future work should improve on: (1). since the size of the collected corpus is small, there is a need to continue collecting a more balanced and larger corpus of indirect answers to whquestions; (2). the proposed 9 possible information sources need to be further evaluated, related to established components of context, and tested across annotators; (3). we hope to improve the performance of our generation model by fine-tuning it with a larger corpus. other methods, such as few-shot learning, data augmentation, and transfer learning techniques may help improve the model performance on generating indirect answers to whquestions.