KUL @ SMM 4 H ’22: Template Augmented Adaptive Pre-training for Tweet Classification (2022)

Abstract: This paper describes models developed for the Social Media Mining for Health (SMM4H) 2022 shared tasks. Our team participated in the first subtask that classifies tweets with Adverse Drug Effect (ADE) mentions. Our best-performing model comprises of a template augmented task adaptive pre-training and further fine-tuning on target task data. Augmentation with random prompt templates increases the amount of task-specific data to generalize the LM to the target task domain. We explore 2 pre-training strategies: Masked language modeling (MLM) and Simple contrastive pre-training (SimSCE) and the impact of adding template augmentations with these pre-training strategies. Our system achieves an F1 score of 0.433 on the test set without using supplementary resources and medical dictionaries.

## introduction the goal of the shared task as part of the social media mining for health (smm4h) -2022 @xcite involves detecting tweets that have adverse drug effect (ade) mentions. organizers of smm4h task 1 provided datasets of english tweets with annotations ade and noade indicating the presence or absence of ade mentions in the tweet. recent work @xcite @xcite showed that downstream performance can be improved by further adapting a general pretrained model by continued pre-training on more relevant set of downstream tasks. the representations learned in the task adaptive pre-training (tapt) @xcite involves pretraining using an unsupervised objective on not just the similar domain but the actual end-task and dataset itself. it has shown to improve the performance of the model for the target task and is less computationally demanding. we explore task adaptive pre-training strategies together with fine-tuning for the classification problem. we further explore the use templates as augmentations to improve the task adaptive pre-training. ## data the dataset @xcite comprises of a training set (18, 000 tweets), validation set (915 tweets), and test set (10, 000 tweets). the dataset is very imbalanced with only around 7% of the tweets consisting of ade mentions. we use oversampling to deal with this class imbalance. as part of the pre-processing on the dataset, we removed url, retweets, mentions, extra space, non-ascii words and characters. further we lowercased and striped off white spaces at both ends. we inserted space between punctuation marks. ## model we first apply tapt @xcite on the pre-trained model to expose it to the task domain sentences and increase the overlap between the language model domain and the target task domain. the two pre-training strategies we explored in this system setup are: 1) simple contrastive pretraining (simsce) @xcite : the idea is to encode the same sentence twice. due to the dropout used in the transformer models, both sentence embeddings will be at slightly different positions in the vector space. the distance between these two embeddings will be minimized, while the distance to other embeddings of the other sentences in the same batch will be maximized which serve as negative examples. given that amount of task data is insufficient for tapt,we further augment the task data and generate multiple sentences with various prompt templates appended at the end of each sentence. the templates are randomly assigned from a set of predefined templates based on label information. examples of templates include: "it contains an adverse drug effect.","it contains an ade","ade is present","it mentions an ade". we thus expose the pre-trained language model (lm) to syntactically diverse template prompts to either cluster the similar representations (simsce_aug) or probe the pretrained lm using masked tokens (mlm_aug). we further fine-tune the continually task pre-trained model on the task data. the supervised tapt with augmentations phase helps the pre-trained lm encapsulate a broader range of task distribution. we use bertweet @xcite as the base encoder to compute the sentence representations. to tackle class imbalance, we experiment with oversampling. for oversampling approach, we randomly sampled positive examples with replacement until ade class contained 10,000 tweets. ## experiments for the classification task, each model is fine-tuned for 10 epochs with a learning rate of 5e -5 using adam optimizer @xcite . we set the batch size to 32 and the maximum sequence length to 128. we utilize pytorch @xcite and huggingface library 1 for training bert using cross-entropy loss. we save model_checkpoint every 200 steps against the validation set using the f1-score of the ade class for evaluation. for adaptive pre-training with mlm, we train for 10 epochs with masking probability set as 0.15. for adaptive contrastive pre-training, we train for 10 epochs with batch size of 64 and the maximum sequence length 100. ## discussion ## conclusion in this work, we explore and propose the use of template based tapt for improving the downstream task performance of detection ade entity mentions in english tweets. we demonstrate the significance of the template augmentation in comparison to traditional adaptive pre-training strategies. experiments have shown that our model with template augmentations with tapt has achieved an f1-score of 0.43, precision of 0.61, and recall of 0.33 on the test set. the future directions would be to use supplementary data and back-translation together with template augmentations.