Unsupervised stance detection for social media discussions: A generic baseline (2024)

Abstract: With the ever-growing use of social media to express opinions on the national and international stage, unsupervised methods of stance detection are increasingly important to handle the task without costly annotation of data. The current unsupervised state-of-the-art models are designed for specific network types, either homophilic or heterophilic, and they fail to generalize to both. In this paper, we first analyze the generalization ability of recent baselines to these two very different network types. Then, we conduct extensive experiments with a baseline model based on text embeddings propagated with a graph neural network that generalizes well to heterophilic and homophilic networks. We show that it outperforms, on average, other state-of-the-art methods across the two network types. Additionally, we show that combining textual and network information outperforms using text only, and that the language model size has only a limited impact on the model performance.

## introduction stance detection is the task of determining the position of a text or person towards a certain target, often split into "for" and "against" with an optional third split being "neutral", "unknown", or "neither". the target can be an entity, a topic, or other subject, such as a claim or event. while our work focuses on unsupervised methods for stance detection, much of the recent work on this domain has been done with supervised methods @xcite @xcite @xcite . in contrast, only a limited range of methods have been proposed for unsupervised contexts (e.g. @xcite ). this is important for topics where annotation is costly, or results are timesensitive. like with supervised methods, some methods are mainly text-based, @xcite @xcite focusing on the text itself, while others are graph-based, focusing mainly on the users' interaction network @xcite . recently, @xcite proposed a method integrating both text and graph/network information. their work focused more on determining polarizing concepts and identifying the source of polarity in communities, which differs from the focus of our work. additionally, it is noteworthy that this approach builds upon the foundation set by earlier supervised studies (e.g. @xcite ) which have similarly attempted to utilize semantic and social graph information for different applications than stance detection, notably using graph neural networks. nevertheless, we show in our experiments that existing methods have the strong disadvantage of being tailored to a specific data source, i.e. a particular social media platform such as twitter, but can hardly generalize. in particular, the phenomena of homophily, where users tend to interact with other users who share their opinions or beliefs, and heterophily, where users interact with those who hold opposing beliefs to their own @xcite are important when considering the source of the data. for instance, platforms such as twitter tend more towards homophily @xcite , while other platforms, such as debate forums, fall more on the side of heterophily @xcite . current methods are designed for one or the other of these network types, raising difficulty when the data does not match the network type that the method was designed for, regardless of data source. to the best of our knowledge, we are the first to propose an unsupervised stance detection method that leverages both semantic information via text embeddings and network information using a graph neural networks (gnns) to handle both types of networks (homophilic and heterophilic). this method is trained with a controllable contrastive ## existing approaches our research focuses on an unsupervised framework, based on textual and/or network data, designed to predict user stance in social media networks. in the following, v denotes the collection of users, the interactions are modeled by a graph g. additionally, each user is associated to a set of documents they posted online. the nature of g varies according to the framework but the general approach consists of first finding a representation that best describes the behavior of the users based on their interactions and/or the content they published and then to cluster these vector representations of users in such a way that the cluster label assigned to a user corresponds to their predicted stance. as unsupervised approaches are rare, we have chosen three recent methods that can be considered as baselines to test against our approach. each is built for either homophilic or heterophilic networks, exploits either graph structure or textual data in some way, and has code made available: in-fovgae @xcite , stem @xcite and the method from @xcite . infovgae @xcite utilizes a different graph type than other methods we tested, opting for a bipartite graph @xmath0, t, e) with two types of nodes: the users (v ) and the tweets (t ). there is an edge (v i , t j ) ∈ e between a user v i and a tweet t j if the user has tweeted t j . the incidence matrix for this bipartite graph is then the input to a variational graph auto-encoder (vgae), which aims to recreate this matrix. after training, infovgae uses embeddings from the latent space as input to a k-means algorithm to cluster the users. thus, infovgae focuses on the user-tweet relationship instead of the textual content of tweets, disregarding semantic information. this model is tailored to perform well on homophilic networks. stem @xcite relies on the assumption that if one user responds to another, they do not hold the same stance. the input graph of stem is a weighted undirected graph of user interactions where an edge (v i , v j ) between two nodes v i and v j indicates a direct interaction between these two speakers and the weight of this edge corresponds to the number of interactions. the goal of stem is to find user embeddings that maximize the distance between two users who have interacted and thus have an edge between them. to do so, it first reduces the graph to its 2-core, then solves a relaxation of the max-cut algorithm in order to find the node embeddings. stem then finds, in this vector space, a random hyperplane that passes through the origin to split the vectors into two opposed groups. the labels obtained for the nodes belonging to the 2-core are then propagated to the nodes that sit outside of the 2-core using the initial assumption that if one user responds to another, they should have opposing stances. due to its underlying hypothesis, stem is more particularly dedicated to nonhomophilic or even heterophilic networks. in addition, it only applies when the number of stances is limited to two (for and against). finally, stem disregards textual information due to its emphasis on structural embeddings. ## a generic model for unsupervised stance detection (gusd) this section presents our model, called gusd whose architecture is given in figure 1 . we have made the code freely available foot_0 . in this framework, @xmath1, e) is a simple undirected graph and there is a weighted edge (v i , v j ) ∈ e between v i and v j if they have interacted in some way (retweet, mention, reply, etc) in this network. more precisely, the edge weight is the count of the interactions between the users. let f be a matrix of size |v | × d where each row f i corresponds to a vectorized representation of the user's textual production. we leverage pretrained language models (encoder transformers in our experiments) to build f . more precisely, we use an average of the [cls] token representation of each document (a tweet or a post) produced by a user as done in @xcite . we use a graph neural network trained in a selfsupervised setting to build informative representations of the users that incorporate both the graph and text information. specifically, we leverage graph attention networks (gat) @xcite . we recall that the computation of the embedding of node v i in layer l we write z l @xmath2: with σ an activation function, w l a learnable weight matrix shared across all nodes, and α ij a learned attention coefficient for node v j with respect to node v i that captures the importance of neighbors. finally, the initial representation is the text embedding previously built, so z 0 @xmath3. as no node annotation is provided in the unsupervised setting, we follow prior works @xcite and use a self-supervised objective to train the node representation. we perform graph reconstruction using a contrastive approach. the aim is to maximize the probability of observed edges and minimize the probability for a set of negative examples (a subset of the unconnected nodes of the graph). we use a soft contrastive loss as presented in @xcite : note that this self-supervised strategy is quite standard @xcite , therefore not well suited to deal with both homophilic and heterophilic graphs. to circumvent this issue, we propose two versions of this training objective. one fits a homophilic network, while the second can handle a heterophilic network. for the former, the homophilic version, the positive examples are connected nodes and negative examples are drawn among unconnected pairs of nodes. more precisely, for a positive pair, i.e. an edge between nodes u and v, we draw one random node w that is not connected to u, and add the pair (u,w) to the set of negative pairs. for the latter, the heterophilic version, we build an alternative adjacency matrix a ′ from the original adjacency a of g. with c(.), the function that changes positive non-zero values to 1 and negative values to 0, we compute a ′ = c(aa t )c(a). in a ′ there is an edge between two users that interacted with at least one common user (e.g. that debated with the same person in a debate setting). it also removes every direct interaction from the initial graph. this process transforms the debate/heterophilic graph into a homophilic graph. positive values in a ′ form the positive examples, while for negative examples we use positive values in c(a) to give a strong signal of users that should be placed apart in the latent space. after training the model in this self-supervised setting, we perform k-means clustering on the representations z, providing groups of users that we believe cover their opinion proximity and therefore their stance. ## experimental protocol the first aim of our experiments is to compare our model, implemented with bert-large, with the state-of-the-art on various datasets with different characteristics and to show its capacity to adjust to different network types (homophily and heterophily, specifically), confirming that it constitutes a solid baseline for further works. then, to investigate how the size of a pre-trained language model utilized to represent textual information, affects the performance of our model gusd, we conducted additional experiments using distilbert @xcite , bert-base, and bert-large @xcite . finally, we study the interest of combining both text and graph data for unsupervised stance detection. before presenting the obtained results, we detail our experimental protocol. ## results it should be noted that a high standard deviation associated with this overall average means a high variability of the scores obtained by a model over the different datasets. as shown by the results, the baselines have specific contexts where they perform best, but outside of those contexts they often perform on par with random chance or are unable to produce results. infovgae struggles with heterophilic networks, as it uses graph convolutions on the bipartite graph and thus aggregates up to the two-hop neighborsusers that shared the same content. it performs best on homophilic networks with a higher percentage of retweets because of its use of a bipartite graph and graph convolutions. this is because a graph with many unique texts (meaning texts that have been posted by a single user without any retweets, such as with createdebate) will produce a less connected graph and lower the model's effectiveness. stem, on the other hand, was built for heterophilic interaction networks as confirmed by its poor performance on the homophilic datasets. some attempts were made to adjust stem to homophilic data, such as adjusting the max cut objective, however these tests were unsuccessful. the method from darwish et al. works best on homophilic twitter data that can build more informative frequency vectors, meaning that the average tweets, hashtags, or mentions per user are relatively high, such as in datasets conref and timme (see data statistics in while gusd does not outperform every model on every dataset, that was not its goal but it does outperform the other models on average across the datasets. because it can be run on both homophilic and heterophilic interaction networks, it is able to adjust to the needs of these two different types of networks, where assumptions made by other models do not hold across both types. additionally, unlike the other models gusd runs without any issues on all the datasets and is less variant to the additional noise that the timme-all dataset contains in comparison to timme-pure, which is filtered to only data on politicians. ## conclusion while unsupervised models do exist for stance detection, they struggle to generalize to network types that do not hold to the assumptions the methods are based on. as such, none of them can act as baselines across multiple datasets with opposing characteristics of homophily or heterophily. to that end, we have conducted an analysis of state-of-theart models on varying datasets and proposed a new baseline model. unlike existing work, it exploits both text content and graph structure by using text embeddings propagated via graph neural networks, which makes it more generalizable to different network types. gusd outperforms the other unsupervised models on average and is robust to changes in the number of parameters of the language model used to construct the text embeddings.