Putting Context in Context: the Impact of Discussion Structure on Text Classification (2024)

Abstract: Current text classification approaches usually focus on the content to be classified. Contextual aspects (both linguistic and extra-linguistic) are usually neglected, even in tasks based on online discussions. Still in many cases the multi-party and multi-turn nature of the context from which these elements are selected can be fruitfully exploited. In this work, we propose a series of experiments on a large dataset for stance detection in English, in which we evaluate the contribution of different types of contextual information, i.e. linguistic, structural and temporal, by feeding them as natural language input into a transformer-based model. We also experiment with different amounts of training data and analyse the topology of local discussion networks in a privacy-compliant way. Results show that structural information can be highly beneficial to text classification but only under certain circumstances (e.g. depending on the amount of training data and on discussion chain complexity). Indeed, we show that contextual information on smaller datasets from other classification tasks does not yield significant improvements. Our framework, based on local discussion networks, allows the integration of structural information while minimising user profiling, thus preserving their privacy.

## introduction online conversations are a main channel through which phenomena such as fake news, rumors and hate speech can spread @xcite , political leaning is expressed @xcite ) and one's health conditions can be revealed @xcite . all these phenomena can be captured to some degree automatically, provided that we have reliable nlp systems able to classify the content of the messages. most classification approaches focus on the textual content of single comments (or a pair, in the case of stance detection), however little has been done to include the figure 1 : representation of input data in kialo dataset: the discussion chain (in bold) is extracted from the discussion tree, and each claim has a textual content c, a user id and a timestamp. a support (green) or contrast (red) label w.r.t. the previous statement is assigned to each claim. the initial claim c 0 has no stance (blue). this representation can be easily generalized to experiments on other datasets. indeed, while the actual content of comments gives us information about what was written, knowing whether and how often two users interact with each other can give us a wider picture of how the dialogue is evolving. furthermore, temporal information allows us to identify peaks or "waves" of comments, suggesting the occurrence of a triggering event, as seen in relation to online toxicity @xcite and fake news @xcite . previous nlp studies already investigated how contextual information can be included in the classification of online conversations, mainly following three distinct directions: integrating textual context, i.e. the previous thread of a given post @xcite , modelling user-related context @xcite , or including structural context in terms of conversation structure @xcite , or external knowledge @xcite . regardless of which type of context was considered, one major issue is represented by the limited size of many benchmarks, from which models can hardly learn contextual information @xcite . another drawback is that, in order to develop classification models embedding contextual information, complex and computationally-intensive architectures are needed @xcite . we address the above challenges by proposing an approach integrating textual, temporal and structural context in a simple, unified architecture, where such information is expressed in natural language and is captured by a transformerbased model @xcite for classification, without separately modelling the latent structural information of the interactions. in this framework, we avoid to explicitly provide user-related information, which may lead to privacy issues, but we rather represent users as "local discussion ids", meaning that a user is assigned a new id for each discussion they participate in. as a consequence, if a user is active in several discussions, this information is not available and user profiling at global network level is not possible, thus enforcing privacy preservation. since previous studies highlighted that training size is crucial to make models aware of contextual information, we mainly perform our experiments on a task of stance detection using a large dataset extracted from the kialo platform @xcite . while the dataset is described in detail in section 4, we report in figure 1 an example of discussion structure taken from this resource. to better understand the contribution of the training set size, we perform also an analysis of the learning curve (section 8) and we evaluate the performance of our models on local discussion networks (ldns) of different complexity and of varying length (section 9). as a comparison, we also test our approach on two smaller datasets for stance detection and abusive language detection, confirm-ing the effect of dataset size (section 7). the data are available upon request only for research purposes, in compliance with kialo's terms of service. we follow a data minimisation principle, sharing only the information needed to replicate our experiments after user anonymisation. 1 2 related work despite the fact that social network discussions involve more information than just a sequence of texts, such as user interactions and temporal evolution, researchers have only made few attempts to combine linguistic information with structural and temporal information. some attempts have been made for tasks like fake news detection (e.g., @xcite , hate speech detection @xcite , stance detection (e.g., @xcite and rumour verification @xcite . user-related information has also been successfully exploited in abusive comment moderation @xcite . all these tasks are closely related to the dynamics of human behavior, but the involvement of linguistic information, network information and temporal information altogether has been difficult because of: i. the fusion of heterogeneous knowledge, by combining computationally-expensive models such as pretrained language models and graph neural networks (gnns) @xcite , like in @xcite ; ii. the access to large-scale private data, that cannot be freely released; iii. the training of human annotators on this data; iv. the deletion of social media posts over time, leading to gaps in discussions, especially in hate speech and fake news datasets @xcite . for few shared tasks, datasets that also include contextual information such as user ids and timestamps have been created @xcite . still, researchers have mostly worked only on the textual content. one of the reasons why contextual information has been marginally explored in classification tasks is that it has not been proved beneficial in a consistent way. as shown by @xcite , exploiting the textual context does not lead to any increase in performance for abusive language detection, even if the dataset was re-annotated by looking at the full context. these results have been confirmed by @xcite , who further show that the outcome of contextual models strongly depends on the intrinsic characteristics and the dimension of the training set. @xcite show that adding a short context (only parent and target comments) improves hate speech classification. however, they do not consider any structural context but only textual one. similar to our work, @xcite model contextual information through natural language. however, they consider as "context" external contextual knowledge such as structured knowledge bases, causal relationships, or information retrieved from a large pretrained model, and not the conversation structure. as regards stance detection, @xcite propose a graph-based inference model to predict the stance of a comment versus its own parent, exploiting the concept of graph walk to add context. they perform experiments on a dataset retrieved from kialo, as we do in this work (see details in section 4). a similar task is rumour verification, where the goal is to evaluate the truthfulness of a rumour based on the reaction caused by it. in this case, since the focus is on the effects produced by the claim, the context is represented by the claims following the target claim (i.e., the right context), rather than the claims preceding it (i.e., the left context). to address this task, @xcite propose a combination of bert with a particular graph neural network called gat @xcite . they retrieve both linguistic and extra-linguistic context, but they consider the full discussion tree and perform classification only of the initial claim. to summarize, existing past works that tried to integrate contextual information to classification tasks either were not able to outperform text-only approaches, or yielded an improvement using computationally expensive models such as graph neural networks (gnns). furthermore, they tended to give in input to the model all possible information, including user data. with our approach, instead, context benefits classification, while modelling the diverse types of input in natural language and being privacy-preserving. ## problem statement the definition of discussion is not unique. depending on the social network, different discussion the enforcement of the criminalisation of drugs has harmed communities around the world and should be stopped criminalising the activity of drug users has negative effects on individual, their families, and society the legalisation of drugs would not likely stop the over-policing of affected communities in countries like the us. structures can arise, from discussion chains to discussion trees, or allowing branches only at specific levels. in the following, discussion chain indicates a linear thread of ordered claims, where each claim is the reply to the previous one. this definition allows us to assume that the author of the n th claim has read all the previous n 1 claims. moreover, using the single chain instead of the discussion tree allows us to reduce the complexity of the discussion structure. from a discussion chain we can retrieve a local discussion network (ldn), i.e. a multi-edge directed network of interaction among the users, with a timestamp label for each edge. formalization. let @xmath0, d 1 , d 2 , ..., d m } be a set of discussions, where each discussion is made of an ordered sequence of claims d @xmath1, c1 , c2 , ..., cn } where c0 is called initial claim and each claim ci is a response to the claim ci 1 8i ## kialo dataset for stance detection kialo foot_1 is an online platform where people can debate around a main topic, with moderators being in charge of checking the grammaticality of the claims, evaluating the level of support or of contrast between a target claim and its parent claim, and even moving claims to make conversations more consistent. for these reasons, kialo typically con- in kialo, the author of each comment is required to assign a stance label to it with respect to the parent comment. this label (support or contrast) is then checked by the moderator, who can change it if needed (an example of supportive and contrastive stance from the dataset is displayed in figure 2 ). furthermore, being clearly structured, it is possible to easily retrieve from discussions the reply-tree structure and the distribution of support/contrast comments. datasets extracted from kialo have already been used in the past to study the linguistic characteristics of impactful claims @xcite or perform polarity prediction @xcite . we obtained access to the dataset based on kialo presented in @xcite , which was used for binary stance detection. we extract from their data only a subset containing chains longer than 1 (i.e., having at least the initial claim and one reply). in this way, we obtain 122, 681 training instances, 7, 447 validation instances and 8, 211 test instances. each instance includes: i. the target claim; ii. the discussion chain, from the initial claim to the target claim; iii. the stance of each claim versus its parent claim; iv. the user id of each claim; v. the timestamp of each claim. given a discussion @xmath2, c1 , ..., cn } of length n + 1, the goal is to classify correctly the stance of cn with respect to cn 1 , choosing between support (s) or contrast (c). we report in for each discussion tree we extract all the discussion chains going from the initial claim to the leaves. consequently, it is possible for portions of these chains to overlap, while the target claims, with their respective labels, remain unique. this approach allows the model to process instances in which different discussion progressions result in different outcomes. furthermore, to mitigate potential data contamination effects, the dataset is split according to the initial claim c 0 . as a result, all chains originating from the same initial claim are exclusively assigned to either training, validation, or test set. ## context definition and modelling in past works, context has been integrated in social media classification tasks using two main approaches: by combining linguistic and network information through the combination of node or network embeddings and textual embeddings @xcite or by using textual embeddings as features in a network system, and retrieving a general representation using gnns or node/network embedding techniques @xcite . we follow a third approach by expressing information on structural and temporal context using natural language, and then giving it in input to a transformer-based model. we use a robertabased model @xcite to perform the task. this allows us to keep the same classification framework while only changing the input data to progressively add contextual information, adopting a simple yet effective solution which is computationally lightweight. given a discussion chain @xmath3, c1 , ..., cn } of length n + 1, where @xmath4, u i , t i }, we can identify 3 different types of context: a linguistic (textual) context, c i , and two extra-linguistic (temporal and structural) contexts, t i and u i . ## models and experimental settings we implement and compare eight different classification models trained on the sdk dataset, which can be divided into three categories: dummy, baselines and contextual. dummy models predict the label ignoring the input (i.e., majority class or random class). instead, for baselines and contextual we always use a pre-trained roberta-based model @xcite to embed the input. then we extract the final [cls] contextual embedding and feed it into a multi-layer perceptron (mlp) module to perform the classification task (for details of the architecture, see appendix a.1). we use optuna @xcite for hyperparameter optimization of the learning rate and the dropout applied to the mlp (details in appendix a.2). in figure 3 we report a schematic view of the input configuration employed for the c l s c l s s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p s e p c 0 c 3 1 2 c 7 3 c 11 4 c 12 figure 3: schematic view of the input configuration for each model tested. we display the position of each textual content c i , the [cls] tokens, the [sep] tokens, the user prefix and the time prefix. baseline models and the contextual models. we describe below the different classification models, divided into the three following categories. dummy. we implement two "dummy" models: ## learning curve analysis while our experiments show that the discussion context on the sdk dataset is beneficial to stance detection, we aim to assess the impact of the training set size. our intuition is that, when contextual information is embedded in the model, more training instances are needed than for text-only models. indeed, the model must be given enough training instances to understand what is the role of the special tags and what type of information is included between two specific separators. results. figure 4 shows the results obtained when increasing the training set size as the average over 3 runs (the full results and experimental details are reported in appendix a.4). we exclude the dummy models, since they never outperform baseline and contextual models. with 5% of the training data, all the contex-tual models are beaten by the worst baseline model (i.e., single), with a drop in m-f1 ranging from 10.8 (tc) to 16.3 (tc+u+t) compared to using the whole training set. at the same time, the pair model achieves the best result in this setting, with a performance drop of only 4.6. however, as soon as we add more data, the scenario changes. with 10% training set and 20% training set, con-textual models overcome the single model and progressively approach the pair model. with 40% training set, tu + u and tc + u + t outperform the pair model and with more data they substantially increase their gap with the latter. to sum up, these results show that contex-tual models need between 20% and 40% of the training data (i.e., from 24 thousand to 49 thousand training examples) to achieve comparable results with the pair model, while they need more data to outperform it. ## analysis of discussion structure beside assessing the impact of training set size on classification performance, we are also interested in analysing the role played by the topology of local discussion networks (ldns) in terms of complexity and discussion length. to this aim, we merge consecutive claims written by the same author in a discussion chain into a unique turn, and create a corresponding turn chain. in this way, two consecutive turns have always different authors, and the corresponding ldn does not have self-loops. for further details we refer to appendix a.5. we first divide ldns in the sdk dataset into two groups: simple ldns, which are characterized by chains where users write only one turn, and complex ldns, with a user writing several turns. we run the stance detection experiment with the setting presented in section 7 and compare the results obtained on simple vs. complex chains. we also analyse how the number of claims and of users affects classifier performance on complex ldns (with and without context). results are reported in figure 5 , which displays the m-f1 score obtained with the different models. the thickness of the line represents the standard deviation over 5 runs. the analysis shows that extra-linguistic context gives an important contribution to the classification of complex ldns, in particular the tc + u + t model. this contribution is more limited on simple chains, with the pair model and the contextual models achieving comparable results. as regards the impact that the number of turns has on the classification of complex ldns (middle graph in figure 5 ), we first group the turns into three bins based on their length: from 2 to 5 (dark blue), from 6 to 10 (blue) @xmath5. the comparison among the three groups clearly demonstrates that the inclusion of temporal and structural context consistently results in a performance improvement, regardless of the number of ## discussion the results reported in section 7 and section 8 show that adding extra-linguistic context is beneficial to improve performance on stance detection. however, this benefit arises only if the contex-tual models have access to enough data, which in our experiments on the sdk dataset means between 24, 000 and 49, 000 items. this result explains also the different performance obtained on smaller datasets (section 7.2). as regards the analysis of local discussion chains, the more complex is the ldn, the more evident are the benefits from the structural context. this suggests that our transformer-based model is able to capture the structure given by the interactions among the users, even if implicit, when enough data are available. our analyses show also that capturing contextual information is particularly beneficial with longer chains of turns, and discussion chains with more users. when all contextual information (both linguistic and extra-linguistic) is included in the model, the classifier performs equally well on long and on short chains, making the results more consistent and the model more robust to chain length and user activity. as regards the temporal context, we show that it is still useful to achieve a better performance, but we argue that in kialo it may not be particularly relevant because this is a platform where users are more likely to ponder their responses and take some time to reflect before posting, also thanks to a strict moderation policy @xcite . ## conclusions in this paper we have tested the effectiveness of using linguistic and extra-linguistic contexts for text classification. our results show that full linguistic context alone worsens or does not significantly improve the results with respect to the non-contextual baseline. instead, with extra-linguistic context, the performance improves, especially with the contribution of structural context. further analysis shows that such results strongly depend on the amount of data on which the models are trained. moreover, we found that extra-linguistic context makes results more robust across discussion networks of different lengths and more or less active users. our experiments show also that transformer-based models are able to embed structural features, which can be effectively given in input to the model in the form of simple natural language statements.