A Taxonomy of Empathetic Questions in Social Dialogs (2022)

Abstract: Effective question-asking is a crucial component of a successful conversational chatbot. It could help the bots manifest empathy and render the interaction more engaging by demonstrating attention to the speaker’s emotions. However, current dialog generation approaches do not model this subtle emotion regulation technique due to the lack of a taxonomy of questions and their purpose in social chitchat. To address this gap, we have developed an empathetic question taxonomy (EQT), with special attention paid to questions’ ability to capture communicative acts and their emotion-regulation intents. We further design a crowd-sourcing task to annotate a large subset of the EmpatheticDialogues dataset with the established labels. We use the crowd-annotated data to develop automatic labeling tools and produce labels for the whole dataset. Finally, we employ information visualization techniques to summarize co-occurrences of question acts and intents and their role in regulating interlocutor’s emotion. These results reveal important question-asking strategies in social dialogs. The EQT classification scheme can facilitate computational analysis of questions in datasets. More importantly, it can inform future efforts in empathetic question generation using neural or hybrid methods.

## introduction questions constitute a considerable part of casual conversations and play many important social functions @xcite . asking follow-up questions about the speaker's statement indicates responsiveness, attention, and care for the partner @xcite . listeners who manifest such an empathetic and curious attitude are more likely to establish the common ground for meaningful communication 1 our code and the annotated dataset are publicly accessible at https://github.com/sea94/eqt . @xcite and appear more likable to the speakers @xcite . the vital role of questions in social interaction makes question-asking a desirable property for open-domain chatbots. these chatbots aim to engage in a natural conversation with the users while practicing active listening to deliver understanding and recognition of users' feelings @xcite . in fact, generating meaningful questions is so important that this has become one of the central objectives of such agents @xcite . however, asking questions effectively is challenging as not all questions can achieve a particular social goal, such as demonstrating attentiveness or empathy @xcite @xcite . given the task complexity, automatic conversational question generation is still gaining momentum, with only few results reported so far. @xcite suggested a way to control the number of questions produced by the model with conditional training. @xcite proposed a question-generation method to increase their semantic coherence with the answer, employing reinforcement learning followed by the adversarial training procedure. @xcite devised a model generating appropriate questions for a variety of topics by modeling the types of words used in a question (interrogatives, topic words, and ordinary words). these works presented approaches to produce contextually appropriate and diverse questions, but none of them considered the effect of questions on the interlocutor's emotional state. we attribute the deficiency in this research to the lack of resources allowing to analyze and model various question-asking strategies in affect-rich social exchanges. to address this gap, we present a categorization and analysis of questions in social dialogs, with four main contributions. first, we develop an empathetic question taxonomy, eqt, by manually annotating a subset of the empatheticdialogues (ed) dataset @xcite . eqt delineates the acts and intents of questions. question acts capture semantic-driven communicative actions of questions, while question intents describe the emotional effect the question should have on the dialog partner. for example, a listener may request information (question act) about the age of speaker's daughter by asking "how old is she?" after learning about her success with the aim to amplify speaker's pride of his child (question intent). second, we design and launch a crowd-sourcing annotation task to grow the original labeled seed subset tenfold ( §5). third, we devise an automatic classification model, qbert, to generate labels for the rest of the ed dataset to demonstrate one important application of the taxonomy ( §6). qbert can facilitate the development of chatbots that offer engaging and empathetic conversations by raising meaningful questions. finally, we inspect co-occurrences of acts and intents and their effect on the interlocutor's emotion using visualization techniques ( §7). the analysis illustrates the most prominent questionasking strategies in human emotional dialogs. to conclude, we discuss the implications of these results for future question generation approaches. ## related work previously proposed taxonomies of dialog acts frequently differ in types of assisted natural language tasks. the dialog act markup in several layers (damsl) tag set was designed to enable computational modeling of conversational speech using statistical methods @xcite . it consists of 42 communicative acts derived from a switchboard corpus. eight of these labels describe different question types according to their semantic role, e.g., wh-question or rhetorical-question. several works proposed hierarchical taxonomies of dialog acts, targeted at modeling users' intents in human-machine conversations. @xcite introduced their annotation scheme for a symbolic dialog system intended to improve the lives of the elderly, while yu and yu (2021) designed a scheme for facilitating general human-machine chit-chat. in both works the logs of human-machine interactions were used for producing the taxonomies. each of them features labels devoted to questions, characterizing them either by a question word, e.g., how or what, or the form of expected answer, e.g., open-ended or yes/no question. finally, @xcite suggested a taxonomy of empathetic response intents in dialogs from the ed dataset with the purpose of improving controllability in neural dialog generation approaches. it further stated that questioning is one of the most frequent intents of the empathetic listeners. however, none of these works focused on the fine-grained analysis of questions and their role in empathetic dialogs. meanwhile, several linguistic studies closely examined the pragmatics of questions and offered a number of classification schemes. @xcite developed a scheme of 18 tags based on the information sought by the question. their taxonomy applies well for transactional exchanges, but does not capture the social dimension. @xcite studied the correspondence between the social function of questions and their syntactic form. she established 16 social question functions occurring in dyadic spoken conversations between friends. in another research effort, a group of linguists explored the range of social actions performed by questions across 10 languages @xcite . the authors developed a coding scheme comprising 3 semantic question types and 7 social actions and applied it to questions in spontaneous spoken conversations @xcite . finally, @xcite developed a taxonomy of 6 question types to describe questions occurring in their dataset of chat-based conversations between strangers instructed to get to know each other. the described works provide an insightful basis for studying questions in social conversations. however, they do not consider the effect of questions on their addressee's emotional states, neither do they describe specific mechanisms to handle computational modeling. moreover, most of them apply to spoken dialogs, impeding the extension of their results to chat-based exchanges due to the inherent differences in these modalities. lastly, they relied mainly on manual annotation, yielding comparatively smaller datasets. in our study, we extended the derived taxonomy to a large corpus using crowd-sourcing and automatic methods and analyzed the emerging patterns on a large scale. we summarize the comparison of our question taxonomy with the existing schemes in ## dataset for taxonomy derivation, we sought a dataset that contains social dialogs with diverse emotional expressions and could be applicable to train a chatbot with advanced question-generating abilities. we avoided datasets featuring multi-modal dialogs (iemocap @xcite , meld @xcite ) as well as transcribed spoken conversations @xcite , switchboard @xcite ). such dialogs contain back-channel communication and other sensory signals that are not present in chat-based conversations and, therefore, are not well-suited for the modeling task. similarly, we rejected datasets that assist other tasks than social conversation modeling, such as squad @xcite (reading comprehension) or qoqa @xcite (information gathering). finally, we did not consider datasets from social media as they can contain toxic and aggressive responses @xcite . we opted for the empatheticdialogues (ed) dataset @xcite , a benchmark dataset for empathetic dialog generation containing 24,850 conversations grounded in emotional contexts. each dialog is initiated by a speaker describing a feeling or experience and continued by a listener who was instructed to respond empathetically. the dialogs are evenly distributed over the 32 emotional contexts, covering various speaker sentiments (e.g., sad, joyful, proud). we found the ed dataset to be a rich source of question-asking as over 60% of all dialogs contain a question in one of the listeners' turns, resulting in a total of 20k listener questions. basic statistics of the dataset are given in descriptor value # dialogs in total 24,850 # turns per dialog on avg. 4.31 # dialogs with at least one question from listener 15,253 (61.4%) # questions from listeners 20,201 ## defining empathetic question taxonomy given the community's interest in question-asking functionality for chatbots and its significance for empathetic response generation, we aimed at developing a taxonomy of listeners' questions asked in response to speakers' emotional inputs. for this purpose, being guided by prior literature review, we employed a qualitative coding method, which is an established approach for such tasks @xcite @xcite . qualitative coding is a process of grouping and labeling similar types of data and iteratively validating the labels. to cover a diverse range of speakers' emotions, we sampled several hundred dialogs uniformly from the 32 emotional contexts in the ed corpus. the sample size was chosen to balance the need for the diversity of questions with researchers' ability to consider each question carefully and was consistent with prior practice. the coding process was informed by previous question classification schemes ( overall, more than 310 questions were annotated. eqt consists of 9 labels for question acts and 12 labels for question intents. the granularity of the taxonomy was driven by earlier linguistic findings and empirical observations about the interplay of the labels in two branches. for example, question acts request information @xcite , ask about consequence @xcite , and ask about antecedent @xcite are related and could possibly be grouped. however, we de-cided to keep them separately as listeners use them with unequal frequencies in positive and negative emotional contexts and combine them with different question intents ( §7). similarly, the initial set of labels for question intents was created based on the variety of emotions present in the dataset. we further reduced it to a manageable size to make it more applicable for an annotation task, while still preserving sufficient expressiveness of labels to represent subtleties of the data @xcite . we present the labels with their definitions below and provide several examples in figure 1 . examples for each act and intent label are given correspondingly in tables 4 and 5 from appendix a. ## crowd-sourced annotation for further analysis, we annotated a larger subsample of the ed dataset with the eqt labels by designing and launching a crowd-sourcing task on amazon mechanical turk (mturk). the design was refined based on three pilot studies: one internal and two mturk-based. for the annotation, we sampled about 40% of dialogs from each of the original 32 emotional contexts. we only sampled the dialogs with at least one question in one of the listener's turns. the dialogs were then pre-processed so that each dialog ended with a question requiring a label. further, we distributed the dialogs into individual human intelligent tasks (hits) and launched them on mturk in a sequence of batches. for each hit we collected the annotations from three workers. the incentive for one hit varied from @xmath00.9 depending on the worker's performance and task configuration. we describe the details about the task design and the annotation procedure below; exhaustive explanations about dialog pre-processing and the task user interface are provided in appendix b. ## automatic labeling to show how eqt can be operationalized, we demonstrate the use of the taxonomy for annotating the reminder of the ed dataset. we first formulate the question act and intent prediction problems and then build two classification models to address them. before training, we augmented the labeled set using k-nearest-neighbors (k-nn) method. we also tried training the classifiers without data augmentation, but their performance was weaker (see appendix d for details). ## analysis of questioning strategies in this section we present the analysis of questioning strategies adopted by the empathetic listeners. we base our examination on human-annotated questions instead of the whole ed dataset to avoid any potential noise which might have been introduced by automatic classification. visualizations for the whole dataset are included in appendix e. here, by a questioning strategy, we imply a combination of act and intent labels assigned to each question. we first analyzed which labels from the two eqt branches form such strategies by plotting the cooccurrences of each pair (figure 2 ). larger circles represent more frequent strategies, while an empty cell indicates that people do not use the given act to deliver the corresponding intent. for example, to amplify partner's joy, one may request information for more details or ask about consequences of the event, but will unlikely raise a negative rhetorical question. several strategies are much more frequent than others. act request information and intent express interest dominate in our dataset, occurring together for 39% of questions. they define the most general type of questions, which are probably easy to ask, providing a reason why listeners use them often. at the same time, dialogs in the ed dataset are relatively short, and it can be difficult for listeners to fully understand the ideas and feelings of speakers in a couple of turns. in this case, requesting information and expressing interest demonstrates listener's attentive curiosity about the situation. once listeners feel more confident about the speakers' sentiments and contexts, they employ more specific question-asking strategies. we further analyzed this phenomenon temporally across dialog turns (figure 3 ). primarily, we studied how listeners' questioning strategies affect speakers' emotions by visualizing the mappings between them. for this visualization, we used 41 emotion and intent labels describing each turn in the ed dataset produced by @xcite . to avoid clutter, we mapped the original 41 labels to 3 coarser categories: positive, negative, and neutral using our best judgement (see appendix e for details). then, for the dialogs containing a question in the second turn, we plotted how speakers' emotions and listeners' questioning strategies shift over the first three turns. we computed the frequencies of all questioning strategies and, for the ones occurring in more than 0.5% of cases, we plotted the flow patterns. we restricted our analysis to the first three turns because over 70% of dialogs in the ed dataset have only four of them, excluding the possibility to study the influence of questioning strategies on further speakers' turns. in order to still get an intuition how listeners' question-asking behavior changes in the consecutive turns, we plotted the dynamics of the ratios of question act and intent labels across the dialog depth. figure 3a shows the flow rates between speakers' emotions and listeners' questioning strategies. as observed before, listeners most likely use follow-up questions to elicit more details about the situation by expressing interest and requesting information. in most of such cases, the speaker's emotion remains preserved in their consecutive utterance as the speaker elaborates on the first turn, maintaining the sentiment. when speakers explain themselves with sufficient clarity already in the first turn, listeners raise more precise questions, adapting the strategy to the affective context. if speakers share a positive experience, listeners try to amplify their emotions by requesting more information or asking about the consequences of the situation. on the contrary, when speakers disclose a negative sentiment, listeners try to validate and alleviate their feelings. they typically intend to express concern, sympathize, offer relief, or de-escalate the issue, and achieve it by asking about what preceded or fol- positive neutral negative amplify excitement, request information amplify excitement, ask about consequence amplify pride, request information amplify joy, request information express interest, request information express interest, ask for confirmation express interest, ask about antecedent express interest, ask about consequence express interest, suggest a reason express concern, request information express concern, ask for confirmation express concern, ask about antecedent express concern, ask about consequence express concern, suggest a reason sympathize, request information sympathize, ask about antecedent sympathize, ask about consequence offer relief, suggest a solution de-escalate, suggest a solution positive neutral negative 49.6% figures 3b and 3c demonstrate how ratios of different acts and intents evolve over two successive listeners' responses. even though the horizon of four dialog turns might be too short to trace all the patterns, a few observations can be made. with increasing depth of the dialog, the overall number of questions decreases, while two types get more prominent: general questions (request information, express interest) and questions aiming at suppressing speakers' negative emotions (e.g., suggest a solution, offer relief ). it may indicate that listeners employ specific strategies to react to positive speakers' emotions immediately after their disclosure, but in case of negative contexts they tend to ask for extra clarifications in the first place and deliver targeted emotional treatment only in the next turn. as dialogs converge to more neutral exchanges, reducing the need to manage speakers' feelings, the ratio of questions demonstrating listeners' general curiously about the subject increases. finally, we reflected on the scarcely represented labels. among acts, positive and negative rhetoric and irony appear least frequently. these labels can be broadly classified as rhetorical questions. they typically serve for self-expression than conversational engagement and, therefore, are less common than other forms of questions @xcite . moreover, negative rhetorical prompts may harm the conversation quality @xcite , which could also explain why listeners avoided them in empathetic dialogs. the same reasoning applies to the two infrequent intents, pass judgement and moralize speaker. another surprisingly rare intent is motivate. we believe that motivation might be difficult to express in the form of a question. moreover, for people who did not undergo special training, expressing motivation might be more challenging than other intents as it suggests a more thorough approach to solving one's problems. ## limitations and future work due to the nature of the ed dataset, some eqt labels are less represented than others. we kept them under consideration as we observed their distinctive role in managing speaker's emotions. their further analysis is crucial for further identifying and designing effective questioning strategies for empathetic conversations, such as promoting motivational questions and avoiding judgmental ones. eliciting additional samples for these categories could be possible by applying qbert classifiers to other datasets capturing social dialogs. our taxonomy does not cover the phatic role of questions typically occurring during greetings, e.g., "what's up?" or "how's it going?" such questions were very rare in the ed dataset. we chose not to analyze them, since these routine questions are the most superficial @xcite and unlikely to serve any emotion-regulation function. in the design of our annotation task, we opted for asking the crowd workers to choose a single most specific label from each of the two eqt branches. this was done with the aim of facilitating further analysis of questioning strategies withing the scope of this study. nevertheless, according to @xcite , most adequate classification schemes in the social sciences allow assigning an observation to multiple rather than only one category. this also applies to our case. for example, for the question "did you go through a breakup recently?" both suggest a reason and request information can be relevant. future work can explore the possibilities of using multiple applicable labels in addition to the most specific one. additional labels can be obtained either by tagging the samples manually or by taking top-n most confident predictions from the classifiers. the results of this paper can facilitate the development of question-asking mechanisms of conversational chatbots. one can employ conditional training @xcite to train an end-to-end neural model on a subset of most effective questioning strategies as defined by the co-occurrences of the eqt labels and their mappings with speakers' emotions (cf. figure 3 ). to achieve even greater interpretability and controllability, researchers can devise architectures that dynamically model the selection of appropriate questioning strategy before generating a question. the strategy can be selected based on the conversational history and speaker's emotion and further passed into the question gener-ation module. the main purpose of such modeling approaches is to lead an engaging empathetic conversation by raising meaningful questions, which deliver desirable effect on user's emotional state. moreover, eqt along with qbert models can be used to label questions originating from other corpora or chat logs and evaluate their effectiveness for regulating speaker's emotions, as described above. ## conclusion in this paper we introduced eqt, an empathetic question taxonomy depicting acts and intents of questions in social dialogs. we used crowdsourcing and automatic methods to tag all listeners' questions from the ed dataset with the eqt labels, which validated their interpretability and produced useful annotations for future research. further analysis of the dataset with the visualization techniques shed light on various question-asking strategies employed by listeners in response to speakers' emotionally-ridden inputs. we identified several useful question-asking behaviors for favorable emotional regulation. we expect that our findings will enable the development of more controllable and effective question-generation models.