M e LT : Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detection (2021)

Abstract: Much of natural language processing is focused on leveraging large capacity language models, typically trained over single messages with a task of predicting one or more tokens. However, modeling human language at higher-levels of context (i.e., sequences of messages) is under-explored. In stance detection and other social media tasks where the goal is to predict an attribute of a message, we have contextual data that is loosely semantically connected by authorship. Here, we introduce Message-Level Transformer (MeLT) – a hierarchical message-encoder pre-trained over Twitter and applied to the task of stance prediction. We focus on stance prediction as a task benefiting from knowing the context of the message (i.e., the sequence of previous messages). The model is trained using a variant of masked-language modeling; where instead of predicting tokens, it seeks to generate an entire masked (aggregated) message vector via reconstruction loss. We find that applying this pre-trained masked message-level transformer to the downstream task of stance detection achieves F1 performance of 67%.

## introduction generated by people, natural language data inherently spans multiple levels of analysis, from individual tokens, to documents (or messages), and to sequences of messages. while the multilevel aspect is rarely looked at beyond words-todocuments, some work has suggested benefits to modeling language as a hierarchy, such as building document representations from a collection of its sentences or a user vector given a history of their language @xcite @xcite @xcite . we consider stance detection, a message-level task, where the social or personal context in which the message appears (e.g., such as a person's profile) has been shown relevant to capturing the stance of the message @xcite . however, such work explicitly integrated user-or social-context into the stance model, as a separate component. we ask if there is a more direct integration of user context when processing a target message. to this end, we process the target message as a part of the sequence of messages from the user. this way of using historical language from a person enables us to both model within message information (word-level) and to process the message within the author context (message-level). while there have been some models that take advantage of hierarchy through words and sequences of messages @xcite @xcite there has been little work in providing generic pre-training routines for large capacity transfer learning style models beyond the word-level. instead, many of these hierarchical models are either applied directly to a downstream task or, if pre-trained, on an adjacent version of the downstream task. being able to pre-train general message-level models could enable inclusion of message-level contextual information that is not easily obtainable with task-specific training that is limited in data sizes as compared to larger unlabeled corpora available for modeling at the message-level. in this study, we propose a hierarchical messagelevel transformer (melt) trained over a novel pretraining routine of masked document modeling foot_0 , where the goal is to encode documents in latent space using surrounding contextual documents. we then fine-tune melt to a stance detection dataset derived from twitter as defined in the semeval 2016 shared task @xcite . our contributions include: (1) introduction of a new pre-training routine for hierarchical message-level transformers 2 , (2) demonstration of efficacy of our pre-training routine for stance detection, and (3) exploratory analysis comparing model size with respect to the number of additional message-level layers and amount of user history leveraged in finetuning. ## related work our approach is inspired by the success word-todocument level transfer learning has had since popularized by the bert language model @xcite . offering the idea of a "contextual embedding" allows models to properly disambiguate words based on their surrounding context. while other types of language models are also used, usually autoregressive based such as @xcite , many models are variants of the bert autoencoder style @xcite . both @xcite and @xcite use hierarchical encoder models for summarization tasks. while both models encode sentences using some surrounding context, their pre-training tasks are still that of text generation rather than latent modeling. @xcite encode global context in conversation threads on social media by generating a history vector (concatenated representations of each sub-thread) during the fine-tuning step and @xcite propose a capsule network to aggregate fine-tuned word representations to perform automatic stance detection. stance detection is an ideal task to develop melt because while it is labeled at the message-level, the stance itself is presumed to be held by the author with a history of messages. previous successful approaches to stance detection have used topic modeling, multi-task modeling via sentiment, multidataset training @xcite @xcite , or user-level information @xcite . our work builds on this by using a pre-trained transformer trained to model message representations in latent space across author histories to encode global user knowledge into individual messages. ## hierarchical message modeling messages are made up of individual words that come together to give each other context and meaning. comparably, a collection of messages can come together to show topics of conversation. directly encoding the interactions of messages and their underlying words can prove beneficial when modeling language at the document or person-level. for example, processing post history of a social media user within context of their own language. ## stance detection with melt we use the stance dataset available from the se-meval 2016 shared task @xcite . this data includes tweets that were annotated either against, neutral, or favoring of a specific target mentioned within the tweet, across 5 distinct targets in the dataset. however, this data only includes labeled tweets from users and not any history, so we use the extended dataset from @xcite . during fine-tuning we keep a max history length of 40 and a temporal ordering within sequence. we figure 2 : average weighted-f1 performance across our models when we fine-tune using different amounts of user history. both size melts improve when more history is available, with a plateau occurring on the 2layer model. ## results we show a comparison of our best melt model against other approaches in the average of 40 recent messages. this allows the model to have a global context within user. we also include the top participant from the shared task @xcite which uses a different f1 score as defined for the shared task, referred to here as semeval f1 foot_2 . lastly, we compare our results to the approach of @xcite , from whom we received the extended history dataset, which uses the labeled tweet and a list of accounts the author follows. however, they only report the weighted-f1 score for their best performing model. we find that fine-tuning distilbert directly to the task of stance detection proves difficult, only scoring a modest f1. however when we include some context language from the user, an average representation of their recent language concate-nated into the fine-tuning layer, there is a noticeable boost in performance highlighting that stance prediction is aided by knowing the context of the message. we find that melt can utilize this contextual information best and out-performs other approaches. next, we break down the performance of various configurations of our models in we also find that the 2-layer melt performs competitively -in figure 2 we show that it performs better or on-par with the large model until 40 messages of history is reached, due to the 2layer model saturating at history of 30. suggesting that the larger the model, the more history it can efficiently track. lastly, we investigate using a different wordlevel model for our experiments. we choose dis-tilroberta, for similar reasons to our original choice of distilbert, and apply the same techniques as done with distilbert shown in ## conclusion with a large number of tasks in nlp that rely on social media as a domain, methods which can model language as a multi-level phenomena, from words to documents to people, can offer a higherlevel contextual representation of language. in this work, we presented a new hierarchical pre-training routine that, when fine-tuned to stance detection, outperforms other models utilizing both message and user-level information as well as improves re-4 both melt-rands learn the mfc baseline sults upon solely using the word-level model on which we build melt. we also find that during fine-tuning, it was always beneficial to unfreeze the word layers even though they had to be frozen during pre-training. melt can be attached to the top of a word-level language model in order to directly encode sequences of message vectors, thus allowing the modeling of historical context and leading towards a way of approaching language modeling that integrates its personal context.