C o C o L o F a: A Dataset of News Comments with Common Logical Fallacies Written by LLM -Assisted Crowds (2024)

Abstract: Detecting logical fallacies in texts can help users spot argument flaws, but automating this detection is not easy. Manually annotating fallacies in large-scale, real-world text data to create datasets for developing and validating detection models is costly. This paper introduces CoCoLoFa, the largest known logical fallacy dataset, containing 7,706 comments for 648 news articles, with each comment labeled for fallacy presence and type. We recruited 143 crowd workers to write comments embodying specific fallacy types (e.g., slippery slope) in response to news articles. Recognizing the complexity of this writing task, we built an LLM-powered assistant into the workers’ interface to aid in drafting and refining their comments. Experts rated the writing quality and labeling validity of CoCoLoFa as high and reliable. BERT-based models fine-tuned using CoCoLoFa achieved the highest fallacy detection (F1=0.86) and classification (F1=0.87) performance on its test set, outperforming the state-of-the-art LLMs. Our work shows that combining crowdsourcing and LLMs enables us to more effectively construct datasets for complex linguistic phenomena that crowd workers find challenging to produce on their own.

## introduction logical fallacies are reasoning errors that undermine an argument's validity @xcite . common fallacies like slippery slope or false dilemma degrade online discussions @xcite and make arguments seem more dubious, fostering misinformation @xcite . automatically detecting logical fallacies in texts will help users identify argument flaws. however, automatically identifying these fallacies in the wild is not easy. fallacies are often buried inside arguments that sound convincing @xcite ; over 100 types of logical fallacies exist @xcite . the nature of the figure 1 : examples from cocolofa. for each news article, we hired crowd workers to form a thread of comment. each worker was assigned to write a comment with a specific type of logical fallacy (or a neutral argument) in response to the article. problem makes it expensive to build large-scale labeled datasets needed for developing fallacy detection models. prior works have created datasets for logical fallacies ( llm-assisted crowdsourced data creation. @xcite found that many crowd worker's submitted summaries were created using llms. we saw it as an interesting opportunity rather than a threat. integrating llm assistance directly into the worker's interface offers benefits for both workers and requesters. for workers, builtin llms can aid in complex writing tasks that might otherwise be too challenging and eliminate the need to switch between browser tabs to use external llms. for requesters, having a built-in llm allows for storing all prompts used and texts produced by the llm, ensuring a more transparent understanding of how llms' outputs are woven into the final data. previous work has integrated ai models into worker interfaces to help produce examples that trigger specific model behaviors, such as model-fooling examples @xcite . in this paper, we advocate using llms to help workers generate complex examples. ## cocolofa dataset construction we constructed cocolofa, a dataset that contains 7,706 comments in the online comment sections of 648 news articles. each comment is tagged for the presence of logical fallacies and, where applicable, the specific type of fallacy. 143 crowd workers, aided by gpt-4 integrated into their interface, wrote these comments. cocolofa also includes the titles and contents of the news articles, all of which are cc-by 3.0 licensed. we split the dataset into train (70%), development (20%), and test (10%) sets by article, ensuring a balanced representation of 21 topics across the splits. this section overviews the data construction steps. ## data quality assessments we hired two experts from upwork.com to assess the data quality. we specified that the experts should have abilities of identifying logical fallacies and writing the explanation to justify their annotations in our job description. cocolofa yielded slightly higher interannotator agreements, while experts often disagreed with each other. as shown in @xcite and @xcite , this level of κ value is normal in annotating logical fallacy data. we computed confusion matrices for experts' annotations and labels in both datasets. the confusion matrix comparing the two experts on cocolofa is shown in figure 3 , and the others are in appendix e. figure 3 shows that most disagreements occur in determining the presence of a fallacy rather than its type. we discuss the possible reasons for high disagreement in labeling logical fallacies further in discussion (section 6). concerns over argumentation scheme. during the annotation process, experts identified that some workers did not include fallacies in their comments. instead, they used argumentation schemes to make their argument "fallacy-like" but valid. to address such an issue, some previous work, such as ruiz-dolz and lawrence (2023), suggested using a series of critical questions of the corresponding argumentation scheme to assess the validity of an argument. however, having annotators or comment writers go through these questions for each comment will significantly limit the scalability of our approach. given that experts only identified 12 out of 237 comments to be "fallacy-like," we considered our approach a reasonable trade-off. ## experimental results we evaluated three types of baseline models with both detection and classification tasks on logic, logicclimate, reddit, and cocolofa dataset ( fallacy detection. given a comment, the model predicts whether the comment is fallacious or not. logic and logicclimate only have positive examples, so we only reported recalls. fallacy classification. given a known fallacious comment, the model classifies it into one of the eight fallacy types. in this task, we removed all negative samples. we only evaluated baselines on reddit and cocolofa because logic and log-icclimate used different fallacy type schemes. ## discussion throughout the project, we learned that annotators often disagree when labeling logical fallacies, as consistently shown by the low inter-annotator agreement reported in all related literature @xcite , including our own. this section outlines the three main sources of disagreement we identified and offers design suggestions for mitigating (or retaining) them. ## conclusion and future work this paper presents cocolofa, the largest known logical fallacy dataset, curated through a collaboration between llms and crowd workers. bert-based models fine-tuned using cocolofa achieved good performances in fallacy detection and classification tasks. in the future, we plan to develop models that use context and reasoning to identify fallacies, especially on out-of-distribution data. additionally, while cocolofa includes eight fallacy types, over a hundred exist. we aim to expand it to cover more.