Transformer-based classification of premise in tweets related to COVID -19 (2022)

Abstract: Automation of social network data assessment is one of the classic challenges of natural language processing. During the COVID-19 pandemic, mining people’s stances from their public messages become crucial regarding the understanding of attitude towards health orders. In this paper, authors propose the transformer-based predictive model allowing to effectively classify presence of stance and premise in the Twitter texts.

## introduction modern natural language processing methods emerged as tools of outstanding performance in many classic machine learning tasks. along with their other achievements, the introduction of transformer architecture @xcite allowed to develop of domain-specific models in the interlingual transfer of social media texts @xcite , identification of drug similarity @xcite , and detection of adverse drug effects @xcite . one of the prospective domains of language model development is an automatic extraction and further assessment of insights gained from twitter texts @xcite . in addition, stance and premise classification tasks are frequently interpreted as critical challenges in social media text analysis @xcite . ## data available labeled data @xcite includes 4155 tweets divided into train and test samples in a ratio of 17:3. regarding the premise classification, the dataset contains a subset of 2445 tweets with a positive label and 1710 tweets with a negative label. as an additional metadata, there are 1402 tweets tagged as stay at home orders, 1526 related to the face masks, and 1227 tweets marked as opinions about school closures. established text data could be assessed from the perspective of trending word frequencies, figure 1 . ## method in order to reach the best score, we performed analysis of the state-of-the-art architectures for text classification and selected the following models: bert (kenton and toutanova, 2019), roberta @xcite , albert @xcite , deberta @xcite , rem-bert (chung et al., 2020), and longformer (beltagy et al., 2020). generally, these models encode each tweet to the fixed-size vector and further apply the classifier layers in an end-to-end manner. the output activation function (softmax) converts the hidden representation of the text to the desired class probabilities which are further used during the computation of binary cross entropy loss function: ## experiments we measure the performance of the models on the main task via three commonly used metrics for the 1 https://pypi.org/project/tweet-preprocessor/ binary classification: roc auc, accuracy, and f1-score. as the optimizer for the selected models, we used adamw @xcite . the models were trained along the 20 epochs with learning rates varying from 0.001 to 0.00001 and batch sizes from the range @xcite @xcite @xcite . experiments were done with 3 tesla v100 gpus and 512 gb of ram. the training time of the models lies in the interval from 3.5 hours up to 5 hours. for the premise classification, the best performance was achieved by the large roberta model, to ensure the statistical significance of applied preprocessing procedures, we leverage the mann-whitney u test regarding the null hypothesis that the f1 scores obtained on the initial and preprocessed tweets belong to the same distribution. we rejected the null hypothesis with a p-@xmath0.05. ## conclusion in this work, we have explored the application of different transformer models to the task of premise classification. we extensively evaluated bert variants and obtained the best architecture during the computational experiments. in future work, we intend to focus on the ensembling methods applied to the presented task.