Approaching SMM 4 H with auto-regressive language models and back-translation (2021)

Abstract: We describe our submissions to the 6th edition of the Social Media Mining for Health Applications (SMM4H) shared task. Our team (OGNLP) participated in the sub-task: Classification of tweets self-reporting potential cases of COVID-19 (Task 5). For our submissions, we employed systems based on auto-regressive transformer models (XLNet) and back-translation for balancing the dataset.

## introduction the social media mining for health applications (smm4h) shared task 2021 @xcite focuses on textual processing of noisy social media data in the health domain. our team (ognlp) participated in task 5, a binary classification task to identify tweets self-reporting potential cases of covid-19. tweets are labeled as positive (marked "1") if they self-report potential cases of covid-19, and negative (marked "0") otherwise. ## dataset the data provided by the organizers comprises tweets gathered from twitter. as shown in ## results and discussion for the second official submission, we used the xlnet system trained for four epochs without bt. however, this model achieved a significantly lower f-score with 0.69 on the official test set and 0.82 on the unofficial test set. we assume that pm did not lead to an improvement as we had to set the number of folds for crossvalidation very low (5) due to the limited gpu computing power available to us. furthermore, we can conclude that back-translation leads to a significant improvement, but the number of additional generated samples plays a decisive role.