zydhjh4593@ SMM 4 H ’22: A Generic Pre-trained BERT -based Framework for Social Media Health Text Classification (2022)

Abstract: This paper describes our proposed framework for the 10 text classification tasks of Task 1a, 2a, 2b, 3a, 4, 5, 6, 7, 8, and 9, in the Social Media Mining for Health (SMM4H) 2022. According to the pre-trained BERT-based models, various techniques, including regularized dropout, focal loss, exponential moving average, 5-fold cross-validation, ensemble prediction, and pseudo-labeling, are applied for further formulating and improving the generalization performance of our framework. In the evaluation, the proposed framework achieves the 1st place in Task 3a with a 7% higher F1-score than the median, and obtains a 4% higher averaged F1-score than the median in all participating tasks except Task 1a.

## introduction social media platforms such as twitter and reddit are full of various statements made by users, and there is abundant social media data that can be used for mining health-related information. natural language processing (nlp) plays an important role in social media data manipulation, which helps solve issues including informal expressions, misspelling of terms, noise, and multiple languages in the tweets. the intersection of social media data mining and health applications is concerned in the 7th social media mining for health applications (smm4h) workshop @xcite . our team participates in 10 classification tasks of the 7th smm4h shared tasks, which are task 1a @xcite , 2a, 2b @xcite , 3a, 4-9. in this paper, a generic framework is formulated to fine-tune pre-trained bidirectional encoder representations from transformers (bert)-based models on text classification datasets of smm4h 2022 shared tasks. for obtaining higher accuracy, we adopt 5-fold cross-validation (cv) to get 5 models and use an ensemble learning technique to obtain the result. furthermore, considering the evaluation performance on test datasets, we fine-tune our models using regularized dropout (r-drop) @xcite and exponential moving average (ema) to increase the models' generalization capability. to deal with data imbalance, focal loss @xcite ) is adopted to assign higher weights to samples with minority classes. additionally, pseudolabeling @xcite ) is adopted as an attempt for improvement. as a result of applying the above techniques, our framework obtains a 4% higher averaged f 1 -score than the median in all participating tasks except task 1a, and a 7% higher f 1 -score than the median in task 3a. ## framework a generic framework is formulated for all classification tasks, which is shown in figure 2 . before model training, simple data preprocessing is conducted. then, to encode tweet texts, bert-based classifiers are trained using 5-fold cv. during the inference phase, ensemble learning is applied for more accurate prediction over individual classifiers. in an attempt to further improve the performance, pseudo-labeling based on the inferred test dataset is used. ## result analysis through the results shown in in terms of training techniques, the 3rd and 5th rows of to verify the effectiveness of focal loss, we compare focal loss with oversampling, which randomly duplicates samples with minority classes until achieving data balance. as shown in the 3rd and 4th rows of additionally, ## conclusion in this work, a generic framework based on pretrained bert-based models is formulated for 10 text classification tasks, which are task 1a, 2a, 2b, 3a, 4-9 in the smm4h 2022. through 5-fold cv, 5 models are trained and fine-tuned with r-drop, ema, and focal loss. then, during the inference, ensemble prediction from 5 models is outputted. in addition, pseudo-labeling is applied for the better fitting capability of our models. as a result, our framework obtains a 4% higher averaged f 1 -score than the median in all participating tasks except task 1a, and achieves the 1st place in task 3a with a 7% higher f 1 -score than the median. for future work, in order to sufficiently utilize labeled data, contrastive learning @xcite will be studied between tweet texts and other information, such as claims in task 2.