Breaking Down the Invisible Wall of Informal Fallacies in Online Discussions (2021)

Abstract: People debate on a variety of topics on online platforms such as Reddit, or Facebook. Debates can be lengthy, with users exchanging a wealth of information and opinions. However, conversations do not always go smoothly, and users sometimes engage in unsound argumentation techniques to prove a claim. These techniques are called fallacies. Fallacies are persuasive arguments that provide insufficient or incorrect evidence to support the claim. In this paper, we study the most frequent fallacies on Reddit, and we present them using the pragma-dialectical theory of argumentation. We construct a new annotated dataset of fallacies, using user comments containing fallacy mentions as noisy labels, and cleaning the data via crowdsourcing. Finally, we study the task of classifying fallacies using neural models. We find that generally the models perform better in the presence of conversational context. We have released the data and the code at github.com/sahaisaumya/informal_fallacies.

## introduction argumentation plays a critical part in our lives as it helps us make decisions and reason about the world around us. studies @xcite have shown that learning how to argue increases the ability to identify weak arguments and decreases the tendency to use verbal aggressiveness. fallacies are weak arguments that seem convincing, however, their evidence does not prove or disprove the argument's conclusion. fallacies are usually divided into formal and informal, where the former can be easily described using logical representations, while for the latter, an analysis of the content is more appropriate. fallacies are prevalent in public part of this work was done while the first author was an intern at inria, france. creating a dataset of fallacious arguments is difficult, given that there are over 100 types of fallacious arguments @xcite . there have been several attempts to create comprehensive datasets: @xcite proposed a game in which players add fallacies in the hope of fouling other participants, in @xcite ad hominem fallacies are found using a subreddit's rule violations, while @xcite fallacies are annotated together with other propaganda techniques in news articles. however, our work is the first to propose a viable solution for finding fallacious arguments belonging to many different fallacy types. in this work, we study fallacies in public discussions on online forums. our salient contributions are: i) we align informal fallacies mentioned on reddit within the pragma-dialectic theory of argumentation @xcite ; ii) we design a methodology for mining and labeling easily fallacies in online discussions; iii) we construct a large and balanced dataset of fallacious arguments; iv) finally, we evaluate several neural models on the task of predicting fallacious argu-ments, and we find that taking into consideration additional conversational context is important for this task. ## fallacies on reddit finding a large sample of fallacious arguments is a challenging task as it assumes going through long conversations, finding arguments, and then verifying if the arguments are sound. another major issue, even if we recognize the argument is flawed, is to find the exact fallacy that is committed, given that more than 100 types of fallacies have been proposed in the literature @xcite . our goal is to construct an annotated dataset of fallacies using a mixed strategy: i) first, as noisy labels, we leverage user comments that mention the name of a fallacy, and second, ii) we clean this dataset by removing false-positive samples via crowdsourcing. our intuition is that a person will mention a fallacy as a reply to another comment to highlight that the previous comment's argument is fallacious, as shown in figure 1 . this might not always be the case, as users could discuss fallacies in general, hence the need to further label the discussion using crowdsourcing. we use the pushshift reddit api @xcite to retrieve data from reddit. the api allows searching comments and submissions by their ids or by a set of keywords. we start by making an exhaustive list of fallacies informed by wikipedia. we chose wikipedia as a resource for creating the list of fallacies as it is one of the most well-known sources of information, hence a reddit user could peruse it easily to understand what fallacy was committed in the discussion. for each fallacy we find all its different designations, for submission title: what is something massively outdated that humanity has yet to upgrade? link: https://www.reddit.com/r/askreddit/comments/b3nwm6/ the 5-day work week i know a lot of people don't like it, but a 9 to 5 office job is a pretty big step up from slavery, feudalism and indentured servitude. though i do agree with studies saying that working for less than 8 hours a day is more productive. ## models and discussion tasks. we address four tasks leveraging our annotated dataset, listed in the order of increasing granularity: i) comment-level (cl) fallacy identification (binary task of predicting if a comment is fallacious or not); ii) comment-level fallacy type identification (multi class prediction of the type of fallacy, with non-fallacious as one class in the 9 classes); iii) token-level (tl) fallacy identification (binary task of predicting if tokens in the coi belong to a fallacy or not); iv) token-level fallacy type identification (multi class prediction of tokens in the coi into one of the eight fallacy classes or the non-fallacy class). ## conclusion and future work in this work, we present a methodology for mining and labeling fallacious comments in online discussions. we find frequent fallacy mentions on reddit and the subreddits in which they are the most prevalent. we create a large corpus of annotated comments and experiment with several neural methods for classification. we explore methods that consider the context of the discussion, and we show that they give better results. there are several exciting directions for continuing this work. first, using our methodology, we can annotate more comments for the eight fallacies we studied in this paper, we can improve the negative example set or explore other types of fallacies. second, we can study another aspect of the discussion, the speech acts. according to the pragma dialectic theory, an argument is composed of several speech acts. investigating if certain speech acts are more prevalent in fallacious discussions might lead to improved detection of fallacies. lastly, in the pragma dialectic theory of argumentation, fallacies are violations of rules of critical discussion, for example, the fallacies we annotated violate two rules, as described in section 3. given the significant number of fallacy types, we believe that a hierarchical approach to their detection could prove more efficient: identifying if a conversation violates one of the ten rules of critical conversation, and then for that particular rule identifying the type of fallacy.