Analyzing Pokémon and Mario Streamers’ Twitch Chat with LLM -based User Embeddings (2024)

Abstract: We present a novel digital humanities method for representing our Twitch chatters as user embeddings created by a large language model (LLM). We cluster these embeddings automatically using affinity propagation and further narrow this clustering down through manual analysis. We analyze the chat of one stream by each Twitch streamer: SmallAnt, DougDoug and PointCrow. Our findings suggest that each streamer has their own type of chatters, however two categories emerge for all of the streamers: supportive viewers and emoji and reaction senders. Repetitive message spammers is a shared chatter category for two of the streamers.

## introduction streamers playing video games online have gained popularity over the past years (see @xcite . platforms like twitch foot_0 and youtube foot_1 have enabled gamers to build massive audiences, with top streamers attracting millions of followers who tune in regularly to watch live gameplay and to engage in real-time chat (see @xcite . this rise of live-streaming platforms has introduced a novel arena for examining the dynamics of online interaction and community formation. the communicative practices observed in these environments are rich with potential for analysis, offering insights into how digital communities emerge, how chatter identities are performed and negotiated in virtual spaces, and how technology shapes discourse and social interaction (see @xcite . real-time chatters provide interesting data for research purposes. in this study, we aim to better understand what categories of chatters are there. we do this by building user embeddings for each chatter using a large language model (llm) and later on cluster these embeddings into chatter categories. by harnessing the capabilities of large language models (llms), we aim to construct a nuanced representation of individual chatters, encapsulating the multifaceted dimensions of their discourse. we study two pokémon streams, one by small-ant and one by pointcrow, and one mario stream by dougdoug. we aim to see whether there are any mutually shared chatter categories, and if any streamers has their own unique chatter categories. through this, we aspire to contribute to a deeper understanding of the sociocultural fabric of online gaming communities by shedding light on the complex interplay between content creators and their audiences in a rapidly evolving digital landscape. furthermore, we present a novel embeddings and clustering driven data-analysis method that is applicable as is in almost any digital humanities dataset. the code has been made available on zenodo foot_2 . ## related work twitch streams have been an object of study before our research as well. there is also a great body of literature on computational sociolinguistics @xcite @xcite . however, in this section we focus on some of the prior literature relating to studying twitch streamers. in the context of online live streaming, prior research @xcite has explored the intricate dynamics between broadcasters and audiences, focusing on the interaction that occurs through both spoken language and written chat during live broadcasts of video games. the research identifies a novel communicative behavior termed 'pivoting,' where both broadcasters and audiences produce context-dependent utterances in response to game events, demonstrating the highly interactive and situated nature of discourse in live streaming environments. a mixed-methods study @xcite ) explored the information behavior and perceptions of co-presence among twitch users, leveraging human information theory and social identity theory as the theoretical framework. through quantitative analysis using tools like liwc and spss, alongside qualitative thematic analysis with nvivo, the study identified that information reaction and production were prevalent behaviors across multiple streams, while also highlighting the strong sense of co-presence experienced by participants. the study by cabeza-ramírez et al. (2022) investigates the complex interplay between video game usage, live streaming and the potential adverse effects associated with excessive engagement in these activities. employing a multilayer perceptron model on a substantial sample of 970 video game users, the research identifies key factors influencing gaming behavior. specifically, the study highlights that motivations related to a sense of belonging to gaming platforms and the positive social uses, such as making friends and pursuing gaming as a profession, significantly contribute to the risk of pathological gaming. building on the foundation of affordance theory, a recent study @xcite examined the practices of the most popular streamers, revealing how these individuals leverage various communication modalities and social commerce elements to create compelling content. their work helps in uncovering the emerging trends and common strategies employed by streamers, offering insights into the evolving landscape of online video streaming as a business model driven by individual content creators. we study three streams: hardest nuzlocke (interrupted by a board game w/ magicthenoah, failboat, captainkidd) foot_3 by smallant, pokemon fusion johto league 150,000+ new fu-sions | !tts on foot_4 by pointcrow and can i beat 10 different mario 64 speedruns simultaneously? !suck foot_5 by dougdoug. obtaining chat data on twitch is rather difficult programmatically. we build a small python script that launches chrome through selenium foot_6 . this script monitors the chatbox and saves all chat messages to a file. this requires us to actually watch through the streams to collect the data. the additional benefit is that we do not need to write a crawler that might violate the terms of service of twitch -this way we simply log out all messages that appear during the streams naturally. smallant was playing a pokémon nuzlocke in his stream where he needed to do some calculations on pokémon stats. the stream featured a section where he was playing a virtual board game with other streamers. pointcrow's stream was about continuing his playthrough of pokémon infinite fusion game. during the stream, he took a break to eat dinner and watch online videos. dougdoug streamed super mario 64. the stream included a section of live coding and a message from his sponsor. figure 1 shows how active chatters were. chatters are grouped in categories based on how many messages they sent. as we can see, most of the chatters on all streams sent only 1-10 messages during the entire stream. ## user embeddings and clustering in order to create a user embedding for each chatter, we concatenate the chat messages for each user and separate each message with a line break. to ensure that we get enough text to represent each chatter well enough, we exclude all chatters who have sent less than 20 messages. if a chatter has not sent enough messages, not enough can be known about them to support the further analysis. we use the llm called palm-2 @xcite ) over google's vertexai foot_7 to embed each chatter's concatenated messages into a user embedding. these user embeddings serve as mathematical representation of the semantics of what each user was chatting about. in particular, we use text-embedding-004 model for seman-tic_similarity task. we picked this particular task for the embedder because we intend to compare the semantic similarity of each embedding. we cluster the user embeddings separately for each stream. for clustering, we use affinity propa-gation @xcite . it takes in an affinity matrix, which shows how close each embedding is to other embeddings, and it will automatically find an optimal number of clusters based on the affinities provided to the algorithm. we use cosine similarity to populate the affinity matrix. we use the methods provided in scikit-learn @xcite for affinity propagation and cosine similarity. similar clustering approaches have previously been used with word embeddings @xcite . some chatters end up clustered into their own clusters. we remove all clusters that have only one chatter, because we are more interested in the overall tendencies of chatter categories, not in individual deviant chatters. ## results the clustering algorithm created 6 clusters for smallant, 12 clusters for pointcrow and 31 clusters for dougdoug. on a closer inspection, we found that some of the clusters included mutually similar messages, so we proceeded to merge some clusters manually. this resulted in 5 cluster for smallant, 4 for pointcrow and 6 for dougdoug. cluster name size characteristics supportive viewers 8 the chatters are engaged with the stream in a laid-back fashion. the messages are generally positive. partykirby spammers 7 the chatters were mainly spamming partykirby emote emojis and reactions 5 the chatters mainly send emojis and short reactions such as "oh no" or "damn". strategic helpers 11 the chatters sent helpful messages to help the streamer plan their pokémon party better. anime and gaming enthusiasts 15 these chatters were talking about anime shows and other games as well and not just what smallant was playing. the results of our analysis on pointcrow's stream can be seen in ## conclusions in conclusion, our study introduces a novel approach within digital humanities by utilizing a large language model (llm) to create user embeddings for representing twitch chatters. by employing affinity propagation for automatic clustering and refining the results through manual analysis, we were able to effectively categorize chat participants from streams by smallant, dougdoug, and pointcrow. the method is generic enough to be used with any kinds of documents. in our study, we found that a bit of manual merging of the clusters was needed. this can be automatized in the future by running multiple iterations of the clustering algorithm by clustering clusters. this could be achieved easily by calculating a centroid embedding for each cluster and recalculating the affinity matrix based on the cluster centroids. our analysis revealed that, while each streamer attracts a distinct type of chatters, there are common categories across the streams. notably, all three streamers share categories of supportive viewers and emoji and reaction senders. additionally, a category of repetitive message spammers was found to be common among two of the streamers. these insights highlight the potential of llmbased embeddings for understanding and categorizing social interactions in digital environments. in the future, it might be interesting to gather more chat messages on multiple streams of the same streamer to assess the stability of the chatter categories across different streams. chat messages also include timestamps, which we did not take into consideration in this study. it might also be interesting to include the content of the actual stream in the study, as our current study was only limited to the chat messages.