Comparing a BERT Classifier and a GPT classifier for Detecting Connective Language Across Multiple Social Media (2024)

Abstract: This study presents an approach for detecting connective language—defined as language that facilitates engagement, understanding, and conversation—from social media discussions. We developed and evaluated two types of classifiers: BERT and GPT-3.5 turbo. Our results demonstrate that the BERT classifier significantly outperforms GPT-3.5 turbo in detecting connective language. Furthermore, our analysis confirms that connective language is distinct from related concepts measuring discourse qualities, such as politeness and toxicity. We also explore the potential of BERT-based classifiers for platform-agnostic tools. This research advances our understanding of the linguistic dimensions of online communication and proposes practical tools for detecting connective language across diverse digital environments.

## introduction the growth and popularity of social media over the past two decades has created many opportunities for natural language processing and computational social science researchers to study short-form text. during this time, researchers have built a wide variety of text classifiers to understand social media posts, including for sentiment analysis @xcite , discrete emotion detection @xcite , life events identification @xcite , and even depression detection @xcite . overwhelmingly, these efforts have focused on negative or unwanted online content. for example, research efforts have focused on the identification of misinformation, disinformation, or bot activity @xcite @xcite . similarly, there are hundreds of studies discussing nlp classifiers for malicious @xcite or toxic language @xcite . at face value, the emphasis on building classifiers for unwanted content makes sense: one very common use case for nlp classifiers is to identify content for removal, whether it be spam messages @xcite or content seen as toxic @xcite . and yet, there is little discussion regarding what desired language on social media would look like. although nlp research has focused a great deal on building classifiers to remove unwanted content on social media, it has paid less attention to classifiers that detect wanted or desired content. to fill this gap, we advocate for and build a classifier for one such language feature: connectivity. we define connective language as language features that express a willingness to talk with people who are not ideologically aligned, such as expressions of intellectual humility or openness to other perspectives. as we explain, connectivity is an essential aspect of human communication, and recent social science research highlights the importance of connective language to facilitate pro-democratic conversations @xcite . this research suggests that connective language can help facilitate discussion @xcite , empower citizens @xcite , and contribute to a healthier public square. a connective language classifier could be used in multiple ways, such as allowing users to filter or sort content, awarding a badge to users employing the language, or recommending content on a platform. these use cases could help people identify others who are interested in having thoughtful exchanges. drawing from the literature in communication research and in natural language processing, this paper introduces and illustrates the use of a multiplatform connective language classifier. first, we build a human-labeled training set using a mix of social media messages from reddit, twitter, and facebook. we use this novel training dataset to build a bert classifier and a generative ai (gpt-3.5 turbo) classifier for connective language. finally, we compare the connective language classi-fier to concepts for which there are existing classifiers, such as politeness, to show how they are semantically distinct. ## proposed method to build a connective language classifier, we apply the following approach: first, we build a multiplatform dataset consisting of content from users who are likely to be engaging in discussion on a topic about which they disagree. this includes a mix of political topics (e.g., for whom should a citizen vote?) and apolitical discussion (e.g., should pineapple be a pizza topping?). we then construct a gold-standard training set of connective language using human labelers. after achieving inter-coder agreement, four undergraduate students labeled 14,107 social media posts. we then use these messages to build a connective language bert classifier. we compare this classifier to one built using gpt 3.5 turbo, a large-language model. we also analyze how connective language is distinct from other similar concepts, including politeness and constructiveness. ## discussion connectivity emerged as an important attribute of online discussions. in this study, we proposed two types of classifiers to detect connective language from social media posts. first, we found that the bert classifier outperforms gpt-3.5 turbo in classifying texts into connective and non-connective categories. this indicates the superior effectiveness of bert in identifying connective language within political discussions. additionally, we found that connective language is conceptually distinct from other related concepts such as politeness, toxicity, constructiveness, reciprocity, among others, suggesting that connectivity represents a unique dimension of discourse quality. furthermore, our results demonstrate the ability to use bert to construct 3 see https://developers.perspectiveapi.com/s/about-theapi-attributes-and-languages multi-platform classifiers, enhancing the versatility and applicability of our approach and potentially laying the foundation for platform-generalizable classifiers. while our analysis did not necessarily find biases among the outputs of the gpt classifier, the decreased accuracy of this classifier may be a result of the more complex, nuanced, or new conceptualization of connectivity, as opposed to more overt or well-studied labels like sentiment or toxicity. this classifier could be used to test whether those using connective language have more deliberative conversations, as theory would suggest @xcite ; to evaluate the effects of exposure to social media posts that contain connectivity; and to examine practical ways of increasing connectivity to the extent that it has pro-democratic effects such as increasing understanding of alternative views. ## conclusion this work is foremost motivated by a desire to advance nlp classifiers that identify desirable language and contribute to quality discussion. drawing from literature on the importance of interactivity, respectfulness, and expressions of openness @xcite @xcite , our work is among the first to propose an nlp classifier to detect connective language. in addition to building a classifier for a relatively understudied concept, our connective language classifier also contributes to ongoing scholarly efforts to build multi-platform classifiers (e.g., @xcite . while single-platform analyses have served as a useful starting point, this work can fail to consider the ever-expanding nature of our multi-platform digital ecosystem. we consider this work to be "in conversation" with the plethora of nlp scholarship building classifiers for harmful or toxic language (e.g., @xcite . while the study of harmful or toxic language is certainly important, especially for removal efforts, it is equally important (and comparatively uncommon) to study and build classifiers for desired language styles. we hope this work inspires others to build and develop classifiers for both undesired and desired online content.