Can Social Media Inform Dietary Approaches for Health Management? A Dataset and Benchmark for Low-Carb Diet (2023)

Abstract: Social media offers an accessible avenue for individuals of diverse backgrounds and circumstances to share their unique perspectives and experiences. Our study focuses on the experience of low carbohydrate diets, motivated by recent research and clinical trials that elucidates the diet’s promising health benefits. Given the lack of any suitable annotated dataset in this domain, we first define an annotation schema that reflects the interests of healthcare professionals and then manually annotate data from the Reddit social network. Finally, we benchmark the effectiveness of several classification approaches that are based on statistical Support Vector Machines (SVM) classifier, pre-train-then-finetune RoBERTa classifier, and, off-the-shelf ChatGPT API, on our annotated dataset. Our annotations and scripts that are used to download the Reddit posts are publicly available at https://data.csiro.au/collection/csiro:59208 .

## introduction current practice and general perception recommend carbohydrates as a major contributor to dietary energy intake-acceptable macronutrient distribution ranges specify carbohydrates 45% -65% of energy intake @xcite . however, recent research spanning clinical trials, systematic reviews, and meta-analyses substantiates the benefits of low-carb diets-< 26% of total energy intake from carbohydrate or less than 130g of carbohydrate per day-especially for individuals with type ii diabetes, including weight loss, blood glucose control and reducing cardiovascular disease risk and remission of type ii diabetes @xcite . a low-carb diet typically involves reducing the intake of carbohydrates and increasing the proportion of dietary protein and fats @xcite . whilst a large body of controlled clinical trials has been conducted @xcite , there is limited research on the experience of customers following a low-carb diet. the term experience entails health effects observed in practice, the challenges or barriers that people may face, and the extent of social support or advice, etc. public perception is another aspect to probe. by understanding commonly perceived health effects and misconceptions of low-carb diets, health professionals can tailor strategies to educate and increase the accessibility of low-carb diets. we consider social media as a valuable potential source of real-world insight to support scientific research and inform health professionals on personal perspectives and experiences of low-carb diets. our aim is to quantify the extent of insights that can be mined from social media firsthand which could indirectly inform dietitians in clinical health management. given the lack of similar studies in the nlp community, the first step is to create a dataset that facilitates such research. we create a dataset that reflects the interests of health professionals and focus on identifying reddit posts about health responses, barriers, and advice. we manually annotate comments and submissions from the low carbohydrate diet community on reddit (r/lowcarb), ensuring reasonable inter-annotator agreement. finally, we benchmark the effectiveness of several classifiers, including support vector machine-based, pre-trainthen-finetune roberta-based, and off-the-shelf chatgpt-based classifiers. related work social media has been extensively investigated to inform health care professionals regarding epidemic intelligence @xcite , pharmacovigilance @xcite , or vaccination hesi-tancy @xcite . however, there are hardly any studies from the nlp community paying attention to dietary practices. @xcite investigated mining arguments for the transition to sustainable dietary practices (plant-based diets) on twitter with crowd-sourced annotations. they focused on identifying claims supported with sufficient evidence, including anecdotal, expert, study, fact, or normative. they also annotated their dataset for stance. their study highlighted the need for sustainability aspects to be considered for design of diet programs. they also mention a restriction in their dataset being lack of context given the nature of the tweets. to our knowledge, however, there is no study that focuses on identifying perspectives and experiences of the low-carb diet and that is the focus of our study. ## dataset we explored two data sources-twitter and reddit-in the early stages of our work. we found that different from tweets that were retrieved through keyword searching and thus are often off-topic, the low-carb subreddit (r/lowcarb) exhibits longer posts with guided discussion and richer information. note that the subreddit fosters a community for people interested in low-carb diets, and may have a selection bias toward more positive user experiences. however, there is still ample conversation on the challenges and obstacles of a low-carb diet. thus, reddit was selected as the main data source as we believe its pros outweigh the cons. we leave the investigation of other data sources for future work. the reddit data follows a tree-like structure. users can submit submissions with a title and an optional body paragraph, which initiates discussion on a defined topic relevant to low-carb diets. redditors may reply to submissions in the form of comments, which can also be directly replied to. we use the pushshift api ( https://github. com/pushshift/api ), which can be used to search all publicly available comments and submissions on reddit, to collect data. 1 when performing the search, we specify only the name of the subreddit (i.e., lowcarb), and the api will search for the most recent comments and submissions within the low-carb subreddit. the searches were conducted several times in 2022-december and 2023-january, and finally, we collect 1570 unique comments and 1210 unique submissions, respectively. annotation schema initial exploration of the reddit data provided a glimpse into its nature and characteristics. we note that many posts are advisory, responding with recommendations to a query or experience, with low-carb recipes frequently posted. nutritionists are often interested in understanding what challenges impede the progression of the low-carb diet and what health response users experience. therefore, we defined three categories of interest: (1) health response, (2) barrier, and (3) advice. any remaining post is categorised as "other". inter-annotator agreement there are in total 200 posts annotated by all five annotators, and we measure inter-annotator agreement on these multiannotated examples. for each category of interest, we compute pair-wise inter-annotator cohen's kappa coefficient @xcite . we observe that the health response category is relatively straightforward to annotate and annotators reach a moderate agreement level (averaged cohen's kappa across annotator pairs: 0.59). in contrast, barrier is the most challenging category, ## benchmark results to evaluate the viability of our annotated dataset, we build several representative classifiers and test their effectiveness on the dataset. ## summary we create a dataset that consists of reddit posts talking about low-carb diets. these posts contain rich information about health responses relating to low-carb diet consumption, barriers impeding the progression, and suggested actions to take. our benchmark results on the annotated dataset show that although chatgpt is promising at classifying reddit posts into defined categories of interest, it still underperforms supervised trained models by a large margin.