An Empirical Analysis of Human-Bot Interaction on R eddit (2020)

Abstract: Automated agents (“bots”) have emerged as an ubiquitous and influential presence on social media. Bots engage on social media platforms by posting content and replying to other users on the platform. In this work we conduct an empirical analysis of the activity of a single bot on Reddit. Our goal is to determine whether bot activity (in the form of posted comments on the website) has an effect on how humans engage on Reddit. We find that (1) the sentiment of a bot comment has a significant, positive effect on the subsequent human reply, and (2) human Reddit users modify their comment behaviors to overlap with the text of the bot, similar to how humans modify their text to mimic other humans in conversation. Understanding human-bot interactions on social media with relatively simple bots is important for preparing for more advanced bots in the future.

## introduction people across the world engage with each other on social media sites for personal, professional, and entertainment-related reasons. in recent years automated agents ("bots") have become more prevalent on social media @xcite . bots engage with human users on social media platforms via the platforms' application programming interfaces (apis). by listening to content using the api, bots are coded to engage based on specific keywords or phrases that are used by human users. as bots become more prevalent on social media, more and more humans find themselves engaging with these bots. these humans may or may not be aware of the fact that the bots are not humans. there is a need to study how humans and bots interact with each other, and to analyze how bots influence the way humans engage on the platforms. on certain platforms bots are opaque (that is, you do not know if a user is a bot). for example, on twitter most bots are opaque, and there is a steady stream of research on detecting bots and analyzing the effect they have on the behavior of human twitter users e.g., @xcite @xcite . however, on reddit many bots are open and explicit about their botness. reddit users engage with each other on topic-focused communities ("subreddits"). if users are knowingly interacting with a bot, does the bot influence what the users will comment? more specifically, our research question for this work is: how does engagement with a known bot influence human behavior on social media? in this work we present a case study of a bot that engages frequently on a small number of subreddits to do a deep dive into the way the bot interacts with humans on the subreddits. the bot's comments are pre-defined and randomly selected, so there is no true "interaction" per-se with respect to how the bot replies to human users. we are interested in seeing if humans reply to this simple bot in interesting ways. as bots become more advanced with improving nlp technologies, the effect on human-bot interaction will become more pronounced. therefore, it is important to understand interaction dynamics with simple bots to theorize about how these interactions may change with more advanced bots. as a result of our analyses we identify two interesting findings. first, we find that the sentiment of the (randomly generated) bot comment has a significant effect on the sentiment of the human reply comment. second, we compare the text content of the bot comments and human replies and find evidence of lexical entrainment, where humans overlap with bots in terms of their text comments, consistent with known patterns of conversation between humans @xcite . we selected the bobby-b-bot for our analyses because the bot is a purely "entertainment" bot, in contrast to many reddit bots that perform some utility (e.g., text summarization or subreddit moderation). the bot source code is open-sourced and available, so we can see how the bot identifies comments on reddit that it should reply to. to activate bobby-b-bot, a reddit user posting on one of the specified subreddits 2 must include some variation of the bot's name in their comment. once activated, the bot will reply to the comment with a randomly selected quote from the got books. to collect all bobby-b-bot comments from reddit, we pulled the bot comment data as comment triples from the pushshift reddit api 3 . in each triple, there is a human post ("parent"), followed by the bobby-b-bot reply, and finally another human post ("child"). we extracted 126,329 bot comments, spanning from 2017/10/23 gmt-4 to 2020/06/14 gmt-4. there are 95,206 (75%) positive parent comments and 31,123 (25%) negative parent comments. for bot comments, there are 75,634 (55%) positive bot comments and 50,695 (45%) comments. when accounting for those bots comments where another user replied, we were left with 16,124 parent-bot-child comment triples. among these child comments, we identified 12,109 (75%) positive comments and 4,015 (25%) negative comments. ## sentiment analysis we first analyze the reddit comment data using sentiment analysis @xcite . in particular, our goal is to determine whether the sentiment of a bot's comment has an effect on the sentiment of comments made in reply. we use the vader (valence aware dictionary and sentiment reasoner) lexicon as our sentiment tool @xcite . vader is a specific tool that is designed for analyzing social media texts. it can generate the sentiment score based on the unlabeled given texts showing the polarity of sentiment (how positive or negative). we fit a linear regression model to predict the vader sentiment score of the child comments in our data set. our three independent variables were: the parent sentiment score, the bot sentiment score, and an indicator variable for whether the parent and child in the triple are the same user. we included all interactions between independent variables in our model. the regression results provide some interesting observations ( when the parent and child users are the same, both parent and bot sentiment have a significant positive effect on the child comment sentiment ( ## textual overlap we next consider how reddit users modify the text of their comments in response to the bot (lexical entrainment). lexical entrainment is when speakers use words that overlap with words of their conversational partner @xcite . prior work has shown that humans will adapt their speech when interacting with computers @xcite @xcite . computer agents have been designed to encourage individuals to adapt their speaking rate or amplitude @xcite . however, here the bot presents randomly selected text, and the only aim of the bot, insofar as it has an aim, is to entertain. will humans adjust their comments when interacting with the bot? for our entrainment metric we use jaccard similarity @xcite . jaccard similarity is a ratio of overlapping tokens to total tokens between two comments, and from 0 (no overlap) to 1 (perfect overlap). entrainment metrics in prior work are token-focused, where entrainment for specific keywords are measured @xcite . here we are interested in a global measure, and we therefore use jaccard similarity to that end. we measure entrainment in our comment triples across three comment pairs: (1) between parent and child comments, (2) between bot and child comments, (3) and between parent and child comments where the trigger phrase has been removed. for cases 1 and 2 we can compare human-human entrainment with bot-human entrainment. however, we include case 3 so that we can remove examples that may artificially inflate our entrainment results. recall that the bobby-b-bot is activated by a user including the bot name in a comment. therefore the phrase "bobby b" (or a variation) appears in all parent comments. in certain cases the child comment also includes the phrase, as the child user also wants to summon the bot. to avoid counting these we remove all instances of "bobby b" from the parent and child comments for our case 3 analysis. in all cases we remove stop words before calculation. in each case we plot a histogram showing, of those triples where there is some overlap, the jaccard scores of the relevant pair in the triple (figure 1 ). for parent-child entrainment there are 10,723 comment pairs with overlap, for bot-child entrainment there are 4,527 pairs, and for parent-child entrainment with the bot triggers removed there are 2,728 pairs. first, the vast majority of overlap between humans (i.e., the parent-child comment pairs) is a result of the child comment user calling the bot again via the "bobby b" keyword. once this is removed the number of instances with overlap drops significantly. however, we still observe some positive amounts of overlap between humans. that this overlap occurs with another conversant parent comment bot comment child comment 1 bobby b is our true savior, he helped us cope with season 8. careful, ned! careful now! that's right, bobby b, we have to be careful when watching season 8 or we might lose braincells. 2 i'm usually always for more but i actually rewatched a clip from e2 the other day (when it's decided lady will be killed in nymeria's place) and forgot how disappointed i was in how bobby b handled it when ned questioned the order. been sad about it surrounded by lannisters! every time i close my eyes i see their blonde hair and their smug, satis-fied faces! i suppose you were surrounded at that time and couldn't do much else, i get it. this bot is alive rip lady 3 bobby b, c'mon man, you have to admit it's pretty similar to something you say all the time. gods what a stupid name! 'what? ros? well it's probably short for something like roselyn, or knowing grrm, maybe like rosmallysyn or some moderately real sounding anglo name that isn't actually a name. i'm surprised he went with something as normal as robert for you. handy nicknames like bobby b are pretty common though. like ned. ned is a normal name, but i don't think eddard is a name in real life. thoughts? 4 why do you keep calling me ned bobby b? a dothraki horde on an open field, ned! why do you keep calling me ned bobby b? we sampled instances where there is evidence of entrainment to see why human users are interacting with bobby-b-bot ( it is intriguing to see that the child comment adopts words both from the parent comment and the bot comment. even though the humans are interacting with a bot, and a simple bot at that, they still will try to behave like they are talking to other human beings. even when the overlaps are just one or two words within the text, those words serve as an important role for human referencing. by mentioning the key common words, humans can still efficiently keep the conversation going. ## related work much of the recent work regarding bots has been in the area of bot detection, in particular bots on twitter @xcite @xcite . on reddit there are some opaque bots @xcite , however in many cases bots are transparent, typically by including the word "bot" in the username. prior work has looked at human-bot cooperation for subreddit moderation @xcite , but to the best of our knowledge this is the first work to study human-bot interaction and comment pattern effects on reddit. this work also diverges from prior work on human-bot interaction @xcite @xcite . in most work where the influence of bots is investigated, the bot is purposefully programmed or designed to elicit some response or interaction. however, bobby-b-bot comments are randomly selected snippets of text, and the bot's purpose is to entertain. ## conclusion in this work we present a detailed analysis of the effects of a single social media bot on human communication. while our sample is limited, the num-ber of such social entertainment bots is only going to increase with more advanced text generation capabilities. the bobby-b bot we studied uses a simple random text selection algorithm to "interact" with users on reddit. we have shown that even a simple bot can have an effect on the way individuals communicate on the platform. understanding human-bot interaction when bots are simple is key for building theories of interaction for when the bot technology improves (e.g., a bot that is built on top of @xcite . in this work we consider a single reddit bot as a case study. our goal was to determine whether a bot built for entertainment, and not meaningful back and forth with human users, had an effect on human user sentiment and word selection. certain details of how the bobby-b-bot is implemented (e.g., randomly selecting what text will be posted) are not consistent with the normal flow of conversation. however, we see that in response to the bot's post, humans are matching keywords that the bot used. so in the context of short, discrete interactions on social media, it would be interesting to see if this behavior holds with other bots. an important direction for future work would be to extend this further and conduct similar studies on a wider range of human-bot interactions to see if the results are more broadly applicable. in particular, is this effect more prevalent in goal-driven bots (i.e., bots that seek to change opinion or raise awareness) than in bots that exist for entertainment purposes? typically, lexical entrainment is targeted, that is, specific keywords are investigated to see if there is overlap @xcite . however in this work we consider global entrainment, where any overlap (stop words excluded) is tracked. because the interactions are short, we consider all overlap meaningful. while jaccard similarity is an appropriate metric in terms of calculating the overall token overlap, more sensitive metrics could also be considered to incorporate weights for different types of tokens (e.g., rare words). prior work on human-machine lexical entrainment looked at conversations between humans and physical robots in a shared space, or conversational agents @xcite . more generally, the entrainment phenomenon is usually studied over the course of a conversation, to determine if conversation participants imitate each other's conversation styles. however in this work we look at discrete human-bot interactions on social media. we find evidence of entrainment in short interactions with a technically simple bot. that humans are imitating the bots in these discrete interactions is an interesting result. future work to investigate this on a larger bot data set is needed to determine if this behavior is wide-spread on reddit. even beyond reddit, bots on other social media platforms such as twitter may be able to influence human responses without an extended back-and-forth to establish trust.