A Dataset of Sustainable Diet Arguments on T witter (2022)

Abstract: Sustainable development requires a significant change in our dietary habits. Argument mining can help achieve this goal by both affecting and helping understand people’s behavior. We design an annotation scheme for argument mining from online discourse around sustainable diets, including novel evidence types specific to this domain. Using Twitter as a source, we crowdsource a dataset of 597 tweets annotated in relation to 5 topics. We benchmark a variety of NLP models on this dataset, demonstrating strong performance in some sub-tasks, while highlighting remaining challenges.

## introduction in natural language processing (nlp), impact on climate change is usually only framed in the context of efficiency @xcite ; @xcite ; puvis de @xcite . while efficiency improvements are welcome, we risk greenwashing nlp and further neglecting the field's potential to positively impact climate change. @xcite proposed to strive towards net positive climate impact of nlp by developing beneficial applications (see §3 for related work in this direction). in ibm's project debater @xcite , a large team of researchers created a system capable of autonomously debating a human in a structured environment. while the system could not convince many people to switch positions, it helped to educate people about certain topics. this can be regarded as a first step towards behavioral change @xcite . in this paper we propose to apply debating technology to promote behavioral change that benefits the environment and climate: namely, mining arguments that can convince people to undergo a shift to a more climate-friendly diet (see §2). our focus is on extracting and labeling argumentative structures used in online social media-specifically, twitterand compiling them into a domain-specific english dataset for green nutrition. our annotation focuses on subjective and anecdotal evidence, shifting away from traditional argument mining methods where more strict explicit evidence is preferred. this shift is motivated by sociological research that shows that anecdotal stories are more persuasive in changing people's opinion @xcite . finally, we train and benchmark baseline models on the dataset, showing promising results but also identifying important challenges. foot_0 ## sustainable diets to successfully transform our societies to become more sustainable, we need to focus on improving the sustainability of our diets. the eat-lancet report @xcite has marked this as a shift away from excessive consumption of animal protein-heavy diets. however, unfortunately, such diets are generally quite prevalent in many developed countries. the science behind the benefits of such a transition is quite well established @xcite , but there is still a lack of incentives to change habitual behaviors for people participating. to change such incentives and habits requires actions from all aspects of society, including individual consumers. @xcite argues that the social transition of our diets to become more sustainable requires us to continuously monitor and evaluate processes across all the societal facets to help solve issues and update practices. therefore to successfully transform our society to consume a sustainable diet for a successful green transition, we must change the social and cultural conditions and traditions around green nutrition. however, @xcite shows that there is solid evidence that established dietary preferences are hard to change for large consumer segments due to negative taste perceptions and lack of knowledge and skills about healthy and green foods. here, we address this challenge by aiming to collect arguments covering various aspects, beyond the obvious ones about health and climate. regardless of which aspects are more convincing, the end result will benefit the climate-our rationale is that the end will justify the means. ## related work positive environmental impact. machine learning and related fields have a substantial potential to help address climate change @xcite . some of the potential paths where nlp can be used for a positive impact include helping people understand their carbon footprint, facilitating behavior change towards more sustainable practices, informing policy and supporting education and financial regulation @xcite . cross-disciplinary research with social science can help improve the understanding of large-scale discourse spread over multiple channels regarding climate change @xcite . successful examples include compliance verification of corporate reports: bingler et al. (2022) examined annual corporate reports and found many engage in "cheap talk" (greenwashing), e.g., lacking specificity in climate goals and activities. @xcite created a dataset for and detected images with misleading captions on twitter for several topics, including climate change. these efforts allow for better policy shaping and steering of the online discourse around climate change, which we hope to achieve with our work too. ## annotation scheme we define an argument mining annotation scheme based on previous work @xcite @xcite , consisting of topics and the annota-tion labels argumentative, claim, evidence, evidence type and pro/con. ## dataset here we describe the procedure for collecting and annotating our dataset of tweets containing arguments related to the topics described in §4. scraping. the corpus used for annotation is a collection of tweets scraped from twitter using tweepy foot_2 by iteratively creating queries by a combination of keywords foot_3 and n-grams from an initial set of topics. for each query, we scrape a maximum of 1000 tweets. we remove retweets, quote tweets, links and videos, as well as tweets with less than three words, resulting in 31840 english tweets in total. user mentions are replaced with it is morally wrong to eat and cause animals pain to fulfill our nutritional needs. evidence: normative figure 1 : simplified examples of arguments for the topic t1 (we should reduce the consumption of meat). in (a) the evidence type is fact, since no source is given. in (b) it is normative, as it describes a belief but is more elaborate than a claim. note that the level of granularity in our dataset is a whole tweet rather than spans within a tweet. here, spans are indicated to illustrate which part of the tweet suggests that it should have a particular label. evidence type example topic(s) pro/con anecdotal we are on the green bean diet here, too! i love them. mom hasn't tried broccoli t1, t2, t3, t4, t5 pro expert many fruit & veg (which contain natural acid) don't trigger flare ups-the list is long and varied (obs this may not apply to you) but after a little digging i found some doctors do reccomend a plant based diet to ease the inflammation. going meatless is even recommended by ica t2, t4 pro study according to a 2022 study, eating an optimal #diet of whole grains, legumes, fish, fruits, vegetables and nuts can improve life expectancy by how many years? t2, t4 pro fact hey eco-friendlies! the well known high-carbon company mcdonalds produces 1.5 million tonnes of food packaging alone ! fun fact carbon footprints are important! tune in for more behind closed door stats! t5 pro normative the dangerous of this thing is that our vegan extremists will start interfering in this.. t4 con unrelated/no evidence give your children healthy food to avoid the dad bod haha ## experiments to evaluate the ability of existing models to mine arguments according to our scheme, we conduct a series of experiments with various approaches. ## results the results are shown in input encoding for xgboost. the different methods of combining topics and tweets for xg-boost (see §6) have a relatively small impact on performance. the concatenation method outperforms the [sep] method in pro/con and claim, and therefore we only report results using it in the tables. xgboost vs. fine-tuning. overall, xgboost performs well compared to the fine-tuned roberta. this could be due to training issues or a lack of data: we only have about 600 unique tweets, with only a fraction of them being annotated as containing claims and evidence, causing issues of sparsity and dataset imbalance. this could be mitigated by using a different training approach or annotating more examples in the future. ## discussion and limitations while we frame our dataset around sustainable diets, it is, in fact, focused on plant-based diets. many other aspects are relevant for sustainability, including production, geographical location, genetic modification, transportation, water consumption, land preservation and health. we leave these issues to future work. the topics used in this paper are simple by design. they are all quite similar, which might cause some correlation issues when training models. for instance, t1 (discussing meat consumption) has a significant overlap with t3 (discussing meat alternatives) of 22% for claim and 26% for evidence. however, it can also indicate the presence of other topics that are similar to both. for instance, when arguing for reducing meat, people might use animal welfare as evidence. therefore, topic exploration and expansion could be done further to improve the spectrum of topics in the dataset and explore how relationships between topics are made and related in debates. the dataset is a starting point for training argument mining models. it is balanced in the distribution of the claims and evidence across the topics, with a minor overlap between topics of roughly 20%. our annotation guidelines are robust enough to be used for crowdsourced and expert annotation. the low agreement in the crowdsourced annotations for evidence may be improved by better guidelines or a different annotation methodology, but they may simply be a reflection of inherent subjectivity. this will be investigated in future work. one issue with this dataset is its relative lack of context for many of the tweets due to them referencing outside tweets or responding to other users in a discussion. there is good potential here to utilize this external context for further argument mining or further improve the detection of claims and evidence in the primary tweet. this could initially be done by annotating the current tweets as ## conclusion we defined an annotation scheme for an argument mining task tailored for social media with a focus on argumentation for sustainable nutrition. we proposed two new types of evidence: normative and fact. with this scheme we scraped and annotated a dataset containing 597 tweets for five different topics, resulting in a dataset of 2985 annotated tweettopic pairs. xgboost is a strong starting point for argument mining, and ibm project debater api is a robust zero-shot model for argumentation tasks.