” I Never Said That”: A dataset, taxonomy and baselines on response clarity classification (2024)

Abstract: Equivocation and ambiguity in public speech are well-studied discourse phenomena, especially in political science and analysis of political interviews. Inspired by the well-grounded theory on equivocation, we aim to resolve the closely related problem of response clarity in questions extracted from political interviews, leveraging the capabilities of Large Language Models (LLMs) and human expertise. To this end, we introduce a novel taxonomy that frames the task of detecting and classifying response clarity and a corresponding clarity classification dataset which consists of question-answer (QA) pairs drawn from political interviews and annotated accordingly. Our proposed two-level taxonomy addresses the clarity of a response in terms of the information provided for a given question (high-level) and also provides a fine-grained taxonomy of evasion techniques that relate to unclear, ambiguous responses (lower-level). We combine ChatGPT and human annotators to collect, validate and annotate discrete QA pairs from political interviews, to be used for our newly introduced response clarity task. We provide a detailed analysis and conduct several experiments with different model architectures, sizes and adaptation methods to gain insights and establish new baselines over the proposed dataset and task.

## introduction in the era of mass information dissemination, question evasion and response ambiguity are widespread phenomena in political interviews and debates, rendering their detection an important aspect of political discourse studies. @xcite presents a meta-analysis of five studies on political interview q&as, concluding that politicians gave clear responses to only 39-46% of questions during televised interviews, while non-politicians 1 code and data can be found here: https://github. com/konstantinosftw/question-evasion . had a significantly higher 70-89% reply rate. in figure 2 we present statistics derived from our human annotations regarding response clarity among us presidents, revealing that politicians often avoid providing clear responses to journalists' questions. this phenomenon is known as equivocation or evasion in academic literature and describes a nonstraightforward type of communication, which is characterised by lack of clarity and includes speech acts such as contradictions, inconsistencies, subject switches, incomplete sentences, misunderstandings, obscure mannerisms of speech @xcite @xcite , rendering political speech susceptible to multiple interpretations from the perspective of the public. figure 1 presents an example of an interview featuring various interpretations, generated labels, and corresponding explanations using our proposed dataset. while the topic has been studied extensively in the field of linguistics, politics and communication, with several typologies proposed for classifying question responses @xcite @xcite , there has been little attempt to analyse whether such typologies are applicable to larger scale data and consistent with varying human perspectives and biases. in other words, the possibility of automatically classifying response clarity has not been explored in nlp, potentially due to the complexity of the task itself, as well as the underlying need to encode and reason over long context. however, recent advancements in language modelling boosted model performance for long-context inputs @xcite @xcite , paving the way for framing the task of automatically measuring response clarity. related to this endeavour, there is related work focusing on the responder's intent interpretation @xcite , or the answerability of questions for question-answering (qa) tasks @xcite @xcite @xcite . however, in both research directions, the focus deviates from directly assessing the clarity of the response, being obfuscated by perceptions of intent or question clarity. we address this by proposing the task of response clarity evaluation, focusing exclusively on assessing the effect of the response, building on relevant discourse typologies. we carry out a detailed analysis of proposed typologies, considering their overlap and consistency, the distribution of proposed classes in our collected data, and the feasibility of using them in an automated task, resulting in our proposed two-level response clarity detection taxonomy. specifically, the first level of the taxonomy accounts for a threeway evaluation of response clarity in terms of the number of interpretations the intended response holds. the second and more fine-grained level covers eleven common evasion phenomena in political literature, which explain in more detail the categorization of responses in the three-scale clarity classes. we use this taxonomy to annotate a dataset of political qa pairs and perform an analysis of the perspective variability among human annota-tors. we then evaluate different llms, exploring various training and inference frameworks, showing that simple prompting and instruction-tuning techniques using our dataset are highly capable of providing meaningful performance. moreover, we find that using the labels of the second level (evasion labels) in a two-step classification strategy helps boost performance for clarity classification. we argue that being able to detect answer ambiguity automatically will facilitate political speech discourse analysis, allowing for comparisons at scale. additionally, the proposed task can shed light on llm capabilities of reasoning over long contexts and prove useful for other downstream tasks in nlp such as question answering (see also §2.1). to sum up, our contributions are threefold: ## proposed taxonomy the typologies discussed in §2.1 are comprehensive and well-researched, but often exhibit compatibility issues @xcite @xcite as distinctions between categories vary among experts and sub-domains. for instance, a somewhat vague reply may be deemed as evasive by some while indirect yet coherent by others, especially since ambivalent responses are particularly prone to confirmation bias @xcite . to enhance objectivity, we focus on the clarity/ambiguity dimension, rather than a reply/nonreply distinction. this approach shifts annotators' attention from the bias-prone task of trying to decipher if an answer is "valid" or "invalid", to whether a response can be interpreted unambiguously or accepts a wider range of interpretations. extensive typologies such as @xcite include over 30 types of replies, resulting in a sparse dataset with few examples per category that further complicates the annotation task. we thus aimed to consolidate these typologies into fewer essential categories, while maintaining crucial distinctions. another necessary adjustment involved breaking down multi-part questions into their constituent questions, which led to the elimination of the category of "intermediate replies". as discussed in §2.1, most interviewers pose multi-barrelled questions and vagueness in a single answer towards a multi-part question results in classifying the entire response as an intermediate reply. to avoid skewing the dataset towards intermediate replies, we broke multi-barrelled questions into separate questions and asked the annotators to label each sub-question and answer separately. taking all of the above into consideration, we arrived at a two-level hierarchical taxonomy. the higher level includes 3 main response categories, namely 1 clear reply, containing replies that admit only one interpretation; 2 clear non-reply, containing responses where the answerer openly refuses to share information, and 3 ambivalent reply, where a response is given in the form of a valid answer but allows for multiple interpretations. at the second level these 3 categories further split into 9 sub-categories illustrated in figure 3 . as a brief exemplification, "q: have you seen my chocolates? a: the children were in your room this morning." would be considered an implicit reply (under the ambivalent category) since there is a rather clear implication on the culprit. yet, the answer does not commit to explicitly stating that "the kids ate it"which would have made for an explicit reply -but rather prompts for a reasoning step to reach the final assumption. instead, "a. i don't know", for the same question, would be labelled as a clear nonreply and specifically claims ignorance, since the respondent explicitly refuses to provide information; also, "a. you should not keep your chocolates all around the house" would be considered a deflection, i.e. an ambivalent answer, as it provides none of the requested information, yet it leverages the subject to pivot on a different point. for further analysis and examples see ## dataset creation as a first step, we collect presidential interviews of us presidents, provided by the official whitehouse website 2 . this resulted in 287 unique interviews spanning from 2006 until 2023 which we further analyse in app. a.1. we extracted a total of 3,445 questions and responses from these interviews, as described in the following sections. we leverage chatgpt to decompose the original interviews into qa pairs, aiming to separate multibarrelled questions into separate sub-questions and their respective response sub-parts. we use the automatically generated list of (sub-)questions to generate annotation instances, and then, upon validating the decomposition, annotators label the response to each sub-question separately. thus, for a given interview question, we may have several 2 interviews from https://www.whitehouse.gov/ . qa instances in the final dataset corresponding to distinct sub-questions, and the classification of the respective sub-responses. we henceforth refer to the generated sub-questions and sub-responses as singular qa pairs, "sqas" for short. human annotation process upon the aforementioned preprocessing of the interview questions, we specify the annotation task where the annotators are provided both with the original qas as well as the decomposed sqas, and asked to label the response for each sub-question separately. we opted for providing the sqas alongside the full text to reduce the effort of manually extracting distinct sqas from the original interviews, which would significantly increase the annotation time per sample. we further introduce counterfactual sqas to measure the annotators' potentially exclusive reliance on sqas, as explained in app. a.3. we were thus able to verify that all annotators followed our instructions and the introduction of sqas aids instead of hindering the annotation process. the prompt provided to chatgpt to create the original sqas and counterfactual sqas is shown in app. h. we employ 3 human annotators alongside an expert with a background in political science and political discourse analysis who acts as a validator of the outcome annotations. as a first "training" stage, we provide the annotators with a tutorial that includes annotated examples from each category of the taxonomy to allow them to familiarise themselves with the concepts introduced. then, the annotators are prompted to perform a series of annotation tasks in the following order: they have to 1 evaluate the sqas produced by chat-gpt as valid or not, and then 2 label each of the individual questions and answers, using the proposed taxonomy or indicate an erroneous question in sqas. finally, they should 3 add any missing questions, as well as the corresponding label. on average, each annotator evaluated 1150 samples. more information is provided in app. a.3. ## conclusion we introduce a novel task on response clarity classification focusing on political interviews. driven by studies of evasion techniques in political sciences, we propose a two-level hierarchical taxonomy for clarity classification that considers different evasion strategies at the lower (leaf) level. we also introduce a new dataset where question-answer pairs are manually annotated with the proposed taxonomy labels. we experiment with a range of different model architectures, sizes and adaptation strategies on our dataset, establishing several baselines. we empirically show that fine-grained labels facilitate classification in response clarity, while encoded model knowledge is strongly associated with classification performance. we aspire for this work to motivate future research in the topic, both from the nlp and political sciences communities.