Out of the Echo Chamber: D etecting Countering Debate Speeches (2020)

Abstract: An educated and informed consumption of media content has become a challenge in modern times. With the shift from traditional news outlets to social media and similar venues, a major concern is that readers are becoming encapsulated in “echo chambers” and may fall prey to fake news and disinformation, lacking easy access to dissenting views. We suggest a novel task aiming to alleviate some of these concerns – that of detecting articles that most effectively counter the arguments – and not just the stance – made in a given text. We study this problem in the context of debate speeches. Given such a speech, we aim to identify, from among a set of speeches on the same topic and with an opposing stance, the ones that directly counter it. We provide a large dataset of 3,685 such speeches (in English), annotated for this relation, which hopefully would be of general interest to the NLP community. We explore several algorithms addressing this task, and while some are successful, all fall short of expert human performance, suggesting room for further research. All data collected during this work is freely available for research.

## introduction recently, a publication on quantum computing described a quantum computer swiftly performing a task that arguably would require 10,000 years to be solved by a classical computer @xcite . a non-expert reader is likely to consider this claim as a hard-proven fact, especially due to the credibility of the venue in which this publication appeared. shortly afterwards, a contesting blog written by other experts in that field foot_1 argued, among other things, that the aforementioned problem can be simulated on a classical computer, using proper optimizations, in 2.5 days. clearly, out of potentially many texts questioning the promise of quantum computers (e.g. @xcite ), making readers of the former publication aware of that specific blog post, which directly contests the claims argued in that publication, will provide them with a more informed view on the issue. broadly, argumentative texts, such as articles that support a certain viewpoint, often lack arguments contesting that viewpoint. this may be because those contesting arguments are not known to the author of the text, as they might not even have been raised at the time of writing. alternatively, authors may also deliberately ignore certain known arguments, which might undermine their argumentative goal. regardless of the reason, this issue places readers at a disadvantage. lacking familiarity with opposing views that specifically challenge a given perspective, may lead to uninformed decisions or establishing opinions based on partial or biased information. therefore, there is merit to developing a system that can automatically detect such opposing views. motivated by this scenario, we propose a novel natural language understanding task: given an input text and a corpus, retrieve from that corpus a counter text which includes arguments contesting the arguments raised in the input text. while contemporary systems allow fetching texts on a given topic, and can employ existing tools to discern its stance -and so identify texts with an opposing view -they lack the nuance to identify the counter text which directly contests the arguments raised in the input text. the potential use-cases of the proposed system exist in several domains. in politics, it can present counters to partisan texts, thus promot-ing more informed and balanced views on existing controversies. in social media, it can alleviate the bias caused by the "echo chamber" phenomenon @xcite , by introducing opposing views. and in the financial domain, it can potentially help analysts find relevant counter-texts to predictions and claims made in earning calls. it may also help authors to better present their stance, by challenging them with counter texts during their writing process. lastly, it may aid researches to examine relevant citations by annotating which papers, out of potentially many, hold opposing views. note, however, that this paper focuses on counter text detection -a useful tool for these worthy goals, but not a complete solution. to pursue the aforementioned task, one needs a corresponding benchmark data, that would serve for training and evaluating the performance of an automatic system. for example, one may start with an opinion article, find a set of opinion articles on the same topic with an opposing stance, and aim to detect those that most effectively counter the arguments raised in the opinion article we started with. this path represents a formidable challenge; for example, reliable annotation of long texts is notoriously difficult to obtain @xcite , to name just one reason out of many. to overcome this issue, here we focus on a unique debate setup, in which the goal of one expert debater is to generate a coherent speech that counters the arguments raised in another speech by a fellow debater. specifically, as part of project debater foot_2 , we collected more than 3,600 debate speeches, each around four minutes long, recorded by professional debaters, on a wide variety of controversial topics, posed as debate motions (e.g. we should ban gambling). with this paper, we make this data available to the community at large. each motion has a set of supporting speeches, and another set of opposing speeches, typically recorded in response to one -and only one -of the supporting speeches. correspondingly, our task is defined as follows. given a motion, a supporting speech, and a set of candidate opposing speeches discussing the same motion, identify the opposing speeches recorded in response to the supporting speech. we analyze human performance on this chal-lenging task, over a sample of speeches, and further report systematic results of a wide range of contemporary nlp models. our analysis suggests that expert humans clearly outperform the examined automatic methods, by employing a potentially non-trivial mix of heuristics. in summary, our main contributions are as follows: (1) introducing a novel nlu task, of identifying the long argumentative text that best refutes a long argumentative text given as input. (2) suggesting to simulate the proposed general task in a well-framed debate setup, in which one should identify the response speech(es) that rebuts a given supporting speech. (3) sharing a large collection of more than 3,600 recorded debate speeches, that allow to train and evaluate automatic methods in our debate-setup task. (4) providing empirical results for a variety of contemporary nlp models in this task. (5) establishing the performance of humans in this task, conveying that expert humans currently outperform automatic methods. ## related work most similar to our work is the task of retrieving the best counter argument to a single given argument @xcite , also within the debate domain. however, in that setting counterarguments may discuss different motions, or have the same stance towards one motion. in our setting, identifying speeches discussing the same motion can be done using existing nlp methods, and being of opposing stances may be explored with various sentiment analysis techniques. our focus is on identifying the response to a supporting speech within a set of opposing speeches, all discussing the same motion. other than the different setup, our task also handles a more complex premise -speeches which are substantially longer than any single argumentative unit, and include multiple such units. an alternative to our approach is breaking the problem into three stages: (1) identifying specific arguments made in each debate speech; a major drawback of the above approach is that it requires a considerable labeling effort -the annotation of arguments mentioned within speeches -which has been shown to be a challenge @xcite . another is that the methods in the above studies which focus on establishing relations at the individual argument level may be limited when aiming to evaluate the perspective of long texts. specifically, a response speech may contain multiple arguments that relate to the supporting speech in different ways. for instance, the speaker in such a speech may choose to concede an argument, while still maintaining an opposite view. therefore simply mapping argument level relations may fall short when trying to generalize and assess full speeches. our task complements the above endeavors by facilitating a framework that would allow extending their granularity from the argument level to a full-text level. also, our main motivation is different -detecting whole long counter speeches, and not the exact counter arguments within the counter speech. the latter, perhaps more challenging goal, is out of scope for this work. new neural models have recently driven performance improvements across many nlp tasks @xcite , surpassing the level of non-expert humans in a diverse set of benchmark tasks @xcite . to facilitate the progress of further research @xcite introduced a benchmark aiming to pose a new series of rigorous tests of language understanding which are challenging for cutting-edge nlp technologies. our work is consistent with the motivation behind these benchmarks, as it suggests a challenging new nlu task, accompanied by a corresponding dataset and benchmarks. the rise of deliberate disinformation, such as fake news, highlights the erosion in the credibility of consumed content @xcite , and situations where one is exposed only to opinions that agree with their own, as captured by the notion of echo chambers, are becoming more prevalent @xcite . the task proposed in this work seems timely in this context. ## data we now detail the process of collecting the speeches, the structure of the dataset, and how it is used for our task. dataset structure each speech in the dataset discusses a single motion and is either a supporting speech -in which a single speaker is arguing in favor of the discussed motion, or an opposing speech -in which the speaker is arguing against the motion, typically in response to a supporting speech for that motion. as described below, debaters recording an opposing speech typically listen to a given recorded supporting speech, and then design and record their own speech in response to it. this counter speech is either explicit -including a rebuttal part in which the speaker directly addresses arguments raised in the rebutted speech, or implicit -including no such dedicated rebuttal section, but tacitly relating to the issues raised in the supporting speech they respond to. the data contains multiple counter speeches to each supporting speech, among which some, none or all may be explicit or implicit. figure 1 depicts the structure of this dataset. examples of one explicit and one implicit counter speeches are included in the appendix. ## human performance recently, with deep learning techniques achieving human performance on several nlu tasks, and even surpassing it, there is growing interest in raising the bar @xcite . that is, to facilitate advancing nlu beyond the current state-ofthe-art, there is a need for novel tasks which are solvable by humans, yet challenging for automatic methods. to assess our proposed task in this context, we performed an annotation experiment, as described below. setup each question presented one supporting speech and between 3 to 5 candidate opposing speeches, all discussing the same motion. annotators were instructed to read the speeches, and select one opposing speech which they thought was a counter speech of the supporting speech. when they could not identify such a counter, they were asked to guess and mention that they had done so. following this step, one of the authors read the explanations provided by the experts and formed a set of reason categories. then, another 60 questions were sampled and given to 3 crowd annotators, using the figure-eight 6 crowdsourcing platform. the crowd annotators were from a dedicated group which regularly participates in annotations done by our team. after choosing a counter speech, they were instructed to choose the reason (or multiple reasons) for their choice from the set of reason categories. the crowd payment was set to 2.5$ per question. to encourage thorough work, a post-processing bonus was given for each correct answer, doubling that pay. the full guidelines given to the expert and crowd annotators are provided in the appendix. ## counter speech identification having established that experts perform well on this task, the question remains whether present nlp methods can match that performance. ## further research potential the data presented here was collected to facilitate the development of project debater, and we chose the novel counter speech detection task to showcase this data and make it available to the community. however, the unique properties of our data -recorded speech which is more organized and carefully construed than everyday speech -make it interesting to revisit well-known nlp and nlu tasks. several examples are listed below. author attribution: all speeches in the dataset are annotated for the debater who recorded them. it could be particularly interesting to study author attribution on our dataset as it contains persuasive language, relevant to opinion writing and social media. additionally, we provide voice recordings and transcripts for all speeches, enabling to study multi-modal methods for this task. topic identification: this is a well established research area which can be examined here in various aspects, including clustering speeches by topic, matching speeches to topics or extracting the topic of a speech without prior knowledge. whereas previous work often requires annotating the topics of texts and deducing a consensual label, in our data the topic of a speech is given by design. sentence ordering or local coherence: the sentence ordering task @xcite is concerned with organizing text in a coherent way and is especially relevant for natural language generation. our dataset allows to study this using spoken natural language of a persuasive nature, that often relies on a careful development of an argumentative intent. the data also provides a unique opportunity to study the interplay between a coherent arrangement of language and the associated prosodic cues. other tasks the large scale of the dataset, over 200 hours of spoken content and their manuallycorrected transcripts, enables its use in other speech-processing tasks that require such data. some examples include speech-to-text, text-tospeech, and direct learning from speech of word @xcite or sentence @xcite embeddings. such tasks often use large scale datasets of read content (e.g. @xcite ), and our data allows their exploration in the context of spoken spontenous speech. in addition, with further annotations of the dataset, it lends itself to other potential tasks. one example is the extraction of the main points of a speech or article. this can facilitate various downstream tasks, such as single document summarization in the context of spoken language. another example is the annotation of named entities within the transcript texts, facilitating direct identification of those entities in the audio, similarly to the work of @xcite . ## conclusions we presented a novel nlu task of identifying a counter speech, which best counters an input speech, within a set of candidate counter speeches. as previous studies have shown, and consistent with our own findings, obtaining data for such a task is difficult, especially considering that labeling at scale of full speeches is an arduous effort. to facilitate research of this problem, we recast the proposed general task in a defined debate setup and construct a corresponding benchmark data. we collected, and release as part of this work, more than 3,600 debate speeches annotated for the proposed task. we presented baselines for the task, considering a variety of contemporary nlp models. the experiments suggest that the best results are achieved using jensen-shannon similarity, for speeches that contain explicit responses (accuracy of 80%) and using conditional mutual-information on speeches that respond to the input speech in an implicit way (accuracy of 43%). we established the performance of humans on this task, showing that expert humans currently outperform automatic methods by a significant margin -attaining an accuracy of 92% on speeches with an explicit true counter, and 76% on speeches with an implicit one. noteworthy is that some of the automatic methods outperform the results achieved by the crowd, suggesting that the task is difficult, and may require a level of expertise beyond layman-level. the reported gap between the performance of expert humans and the results achieved by nlp models demonstrate room for further research. future research may focus on the motivation we described, but may also utilize the large speeches corpus we release as part of this work to a variety of additional different endeavors.