Did they answer? Subjective acts and intents in conversational discourse (2021)

Abstract: Discourse signals are often implicit, leaving it up to the interpreter to draw the required inferences. At the same time, discourse is embedded in a social context, meaning that interpreters apply their own assumptions and beliefs when resolving these inferences, leading to multiple, valid interpretations. However, current discourse data and frameworks ignore the social aspect, expecting only a single ground truth. We present the first discourse dataset with multiple and subjective interpretations of English conversation in the form of perceived conversation acts and intents. We carefully analyze our dataset and create computational models to (1) confirm our hypothesis that taking into account the bias of the interpreters leads to better predictions of the interpretations, (2) and show disagreements are nuanced and require a deeper understanding of the different contextual factors. We share our dataset and code at http://github.com/elisaF/subjective_discourse .

## introduction discourse, like many uses of language, has inherent ambiguity, meaning it can have multiple, valid interpretations. much work has focused on characterizing these "genuine disagreements" @xcite @xcite and incorporating their uncertainty through concurrent labels @xcite and underspecified structures @xcite . however, prior work does not examine the subjectivity of discourse: how you resolve an ambiguity by applying your personal beliefs and preferences. our work focuses on subjectivity in questionanswer conversations, in particular how ambiguities of responses are resolved into subjective assessments of the conversation act, a speech act in conversation @xcite , and the communicative intent, the intention underly-so do you adjust your algorithms to prevent individuals interested in violence from being connected with like-minded individuals? sorry. could you repeat that? congressman, yes. that is certainly an important thing we need to do. ing the act @xcite . we choose conversation acts (or more broadly, dialogue acts) as a challenge to the view that dialog act classification may be an "easy" task that has never been approached from a subjective perspective. moreover, they are a good fit for our question-answering setting and are intuitive for naive annotators to understand. our data consists of witness testimonials in u.s. congressional hearings. in figure 1 , annotators give conflicting assessments of responses given by the witness mark zuckerberg (ceo of facebook) who is being questioned by congressman eliot engel. to make sense of our setting that has speakers (witness, politicians) and observers (annotators), we are inspired by the game-theoretic view of conversation in @xcite . the players (witness, politicians) make certain discourse moves in order to influence a third party, who is the judge of the game (the annotator). importantly, the judge makes biased evaluations about the type of the player (e.g., sincere vs. deceptive), which leads to differing interpretations of the same response. in our example, the two annotators are the biased judges with differing judgments on what type of player zuckerberg is: the first assumes sincere and the second deceptive. for zuckerberg's first response, the conversation act is interpreted un-ambiguously: both annotators agree he is signaling he can't answer the question. the intent, however, is ambiguous, where the cynical annotator interprets the clarification question as lying in order to stall vs. being honest. the second response yields both diverging conversation acts and intents: the first judge interprets the conversation act as an answer with the intent to provide a direct response, whereas the second judge perceives the conversation act as a shift to answer a different question with the intent to dodge the original, unfavorable question. we detail our full label set in section 3.2. we create the first discourse dataset with multiple, valid labels that are subjective. they do not hold concurrently and vary depending on the annotator; we collect annotator sentiments towards the conversants as a rough proxy for annotator bias. we further elicit annotator explanations for a window into their rationalization. a careful annotation protocol and qualification process ensure high quality crowd-sourced annotators with a strong understanding of the task. our dataset contains 6k judgments over 1k question-response pairs, with disagreements in 53.5% of the data. however, unlike our prior example, disagreements are not often trivially attributable to differing sentiments. uncooperative moves are sometimes warranted, regardless of annotator sentiment. interpretation of a response is further influenced by its question. a qualitative analysis of annotator explanations reveals strikingly different uses of subjective language across diverging interpretations. identifying all the possible interpretations of a response is a useful way of analyzing discourse in a realistic setting with multiple observers, and could aid in uncovering sociolinguistic aspects relevant to variations in discourse comprehension. with these goals in mind, we propose the task of predicting the complete set of annotator labels for a given response. we find a transformer-based model outperforms other neural and linear models. we confirm our assumption that incorporating the context of the judge helps the model make better predictions, but still leaves room for improvement. in summary, the task together with the dataset present a valuable opportunity to understand perceptions of discourse in a non-cooperative environment. more broadly, we show the need and value for considering the subjectivity of nlp tasks. our work introduces a framework for identifying, elic-iting, and analyzing these subjective elements, to enable application for other tasks. speech act theory originally did not consider insincere speakers, but later work recognized that even in non-cooperative settings, conversants adhere to the conventions of dialogue, or discourse obligations, such as responding to a question @xcite . for this reason, we explicitly separate judgments on conversation acts (that usually fulfill a specific obligation) from communicative intents, which can be perceived as deceptive (or sincere). prior work examines how writer intentions are often misaligned with reader perceptions @xcite , which further motivates our focus on the reader (our annotator). while our work focuses on subjectivity, ambiguity is studied in many nlp tasks, including natural language inference @xcite , evaluation of nlg @xcite , a recent semeval 2021 shared task, foot_0 as well as several discourse tasks @xcite @xcite @xcite . only one study strives to understand how these ambiguities are resolved: scholman (2019) shows different interpretations of ambiguous coherence relations can be attributable to different cognitive biases. however, our work focuses more generally on subjec-tivity rather than cognitive processes. related nlp tasks include dialog act classification, intent detection, deception detection and argumentation, though we importantly note these predict only a single interpretation. dialog acts are similar to conversation acts that apply at the utterance level. classification models typically combine representations of linguistic units (word, utterance, conversation-level) @xcite . in our work, we employ a hierarchical model to account for the levels in our label taxonomy. intent detection is traditionally applied to human-computer scenarios for task-specific goals such as booking a flight. our conversation data is not task-oriented, and we thus define our intents more closely aligned with beliefs in the sincerity of the speaker. detection of deception is, unlike many other nlp tasks, challenging even for humans @xcite . most datasets consist of instructed lies (where participants are told to lie). our work contains naturally-occurring deception where we include not just lying but other more covert mechanisms such as being deliberately vague or evasive (clementson, 2018), both frequent in political discourse @xcite . argumentation mining analyzes non-cooperative conversations, but typically requires expert annotators. recent work decomposes the task into intuitive questions for crowdsourcing @xcite , inspiring our annotation schemes that assume little to no training. closer to our setting is argument persuasiveness, where @xcite find prior beliefs of the audience play a strong role in their ability to be persuaded, which further motivates our focus on the annotator's bias. ## dataset we create the first dataset with multiple, subjective interpretations of discourse (summarized in ## experiment we propose the task of predicting all possible interpretations of a response (i.e., all perceived conversation act+intent labels) with the goals of analyzing discourse in a realistic setting and understanding sociolinguistic factors contributing to variations in discourse perception. we frame this task as a multilabel classification setting where 6 binary classifiers predict the presence of each of the 6 labels. 8 we evaluate with macro-averaged f1 which gives equal weight to all classes, unlike micro-averaging which in our imbalanced data scenario (figure 3 ) would primarily reflect the performance of the large classes. ## conclusion in this paper, we tackle the subjectivity of discourse; that is, how ambiguities are resolved. we present a novel english dataset containing multiple ground truths in the form of subjective judgments on the conversation acts and intents of a response in a question-response setting. we show the dataset contains genuine disagreements which turn out to be complex and not easily attributable to a single feature, such as annotator sentiment. the annotator rationales provide a window into understanding these complexities, and offer a rich source of linguistic devices. we propose a task to predict all possible interpretations of a response, whose results are consistent with our data analysis: incorporating the annotator bias helps the model significantly improve. we publicly release the dataset in hopes to spur further research by exploring the sequential nature of the hearings to employ crf-type losses and other forms of aggregating annotator judgments.