Fraunhofer SIT @ SMM 4 H ’22: Learning to Predict Stances and Premises in Tweets related to COVID -19 Health Orders Using Generative Models (2022)

Abstract: This paper describes the system used to predict stances towards health orders and to detect premises in Tweets as part of the Social Media Mining for Health 2022 (SMM4H) shared task. It takes advantage of GPT-2 to generate new labeled data samples which are used together with pre-labeled and unlabeled data to fine-tune an ensemble of GAN-BERT models. First experiments on the validation set yielded good results, although it also revealed that the proposed architecture is more suited for sentiment analysis. The system achieved a score of 0.4258 for the stance and 0.3581 for the premise detection on the test set.

## introduction during the pandemic many countries, such as the us, china, and germany, introduced health mandates to cope against the fast spread of the coronavirus. in this time, social networks like facebook and twitter enabled users to share their opinion regarding the implemented health orders. these social media posts can be used as part of argument mining to find reasons for and against those. further, it also allows analyzing whether any opposing opinions are based on false information and intentionally spread disinformation. in task 2. of the social media mining for health 2022 (#smm4h) shared task @xcite , one objective was to develop a classification system that can predict the stance towards imposed health mandates during the pandemic, such as face masks, school closures and stay at home orders, in tweets. another was to detect whether a tweet contained statements that could be used for reasoning. in this paper, the proposed approach of our team, fraunhofer sit, is described. it takes advantage of two generative models to generate new labeled samples as well as a generative adversarial network for classification and to also fully utilize unlabeled samples during training. these are then incorporated into an ensemble classification scheme to improve the classification accuracy. the paper is organized as follows: in section 2 the proposed system developed for the shared task is presented. section 3 showcases and discusses the experimental results achieved by the proposed components on the validation set. the paper then concludes in section 4. ## proposed approach the main component of our classification system to estimate stances and to identify premises in tweets was built around an ensemble of fine-tuned gan-bert @xcite models. gan-bert is a generative adversarial network @xcite ) that allows to fine-tune models on labeled and unlabeled data. its generator network produces artificially crafted embeddings, while the discriminator tries to distinguish them from real embeddings extracted from transformer models. here, bert uncased, bert cased @xcite and albert v2 @xcite served as transformer models due to their performance on the validation set. in the case of real samples, the discriminator also tries to predict their classes. the class-labels hereby consisted of l f av or and l again st for the stance estimation task and l 0 and l 1 for the premise detection task. the prediction probabilities of all the three fine-tuned gan-bert models were averaged to provide a unified classification decision. to provide additional labeled training data and to prevent data scarcity having an impact on the classification accuracy, new samples were generated through gpt-2 @xcite . to ensure that the generated data represent a particular class, one text-generation model was trained on data of a specific class. the data then underwent several preprocessing steps, where emojis were converted to their descriptive terms, urls and user mentions were exchanged with generic tokens. ## experimental results the classification system was implemented using python. for data preprocessing, the emoji 1 package as well as libraries provided by python were used. the gpt-2 models were trained on the preprocessed train split of the dataset provided along the shared task @xcite using the aitextgen framework 2 . to avoid overfitting, the models were only trained for 3500 steps with a learning rate of 1e -3 . the words the, face masks, school closures, stay at home orders, corona and covid19 served as prefixes for the text-generation. further, various temperatures were considered, such as 0.8, 1.0 and 1.2. for each configuration consisting of a label, prefix and temperature, 50 samples were generated and occurring duplicates were removed. the following examples showcase tweets generated by our trained model: stance -favor: covid19 @user what a frightening lot of people. no one is wearing a mask. and related ways of managing it. it's so disappointing. stance -against: the death figures are inflated. a major percentage of deaths are those who died with covid at home, and not from it. for training the gan-bert models in a semisupervised manner, data from the stance detection in covid-19 tweets dataset @xcite served as unlabeled data. the sequence length for the embeddings was set to 280 and the models were trained using a batch size of 16 and learning rates of 5e -5 for both the generator and the discriminator. as optimizer adam (kingma and ba, 2015) was used to take advantage of adaptive learning rates and for regularization, a dropout rate of 0.3 was 1 https://github.com/carpedm20/emoji/ 2 https://github.com/minimaxir/aitextgen chosen. to further avoid overfitting, the training procedure took advantage of model checkpoints to keep only the best performing models while the models were trained for 20 epochs. ## conclusion in this paper, a classification system to estimate stances of health orders and to detect premises in tweets is proposed. the system incorporates two types of generative models to allow training on unlabeled data and to generate new labeled data. the f1-scores achieved by the proposed model on the validation set yielded promising first results. however, it also showed, that the classification architecture is more suited for sentiment estimation rather than premise detection.