CAISA @ SMM 4 H ’22: Robust Cross-Lingual Detection of Disease Mentions on Social Media with Adversarial Methods (2022)

Abstract: We propose adversarial methods for increasing the robustness of disease mention detection on social media. Our method applies adversarial data augmentation on the input and the embedding spaces to the English BioBERT model. We evaluate our method in the SocialDisNER challenge at SMM4H’22 on an annotated dataset of disease mentions in Spanish tweets. We find that both methods outperform a heuristic vocabulary-based baseline by a large margin. Additionally, utilizing the English BioBERT model shows a strong performance and outperforms the data augmentation methods even when applied to the Spanish dataset, which has a large amount of data, while augmentation methods show a significant advantage in a low-data setting.

## motivation social media are a rich source of discussion about diseases in various contexts, including newly emerging cases, experienced symptoms, drug effects, or social consequences. such content can be used to make more informed decisions regarding the societal perception of these diseases, as well as to augment existing medical knowledge. recently, pre-trained models such as bert @xcite in the general domain and biobert @xcite in the medical domain have become the go-to solutions for semantic annotation tasks, including previously held similar shared tasks @xcite . however, most participants have opted for a model trained on the same language (spanish) as the task language @xcite ; @xcite ; @xcite . our research questions in this work are: (1) can we use a domain-specific model pre-trained in one language (english) to produce reasonable performance in another? (2) can we still further improve the robustness of such pre-trained models with methods effective on small datasets? to address these questions, we explore the effects of adversarial and augmentation techniques, namely adversarial training @xcite and aeda @xcite , on the pre-trained english biobert model for detecting disease mentions in spanish tweets. ## system description our system is an ensemble of biobert, a bag-ofwords model, improved bert model, and adversarial data augmentation techniques: biobert @xcite ) is a bert-based @xcite model with the same architecture trained on the general domain (wikipedia and book corpus) and the medical domain (pubmed abstracts and pmc full-text articles). bow @xcite or bag-of-words model creates a dictionary from the training words and calculates a ratio of their occurrences as a disease word. based on this ratio (0.7 in our work), words in the test set are labeled as diseases. this is a fast and strong baseline. adversarial training (at) @xcite is a method which creates adversarial examples in the embedding space and uses them in the training process in order to make a neural network model more robust to small perturbations. psum @xcite or parallel aggregation is a module that is added on top of the bert model to improve its performance. it applies an extra bert layer to the last four layers of the biobert model and aggregates the losses. aeda @xcite is an adversarial data augmentation technique which inserts punctuation marks into a sentence randomly creating more data for training. in addition to each individual method above, we also pair biobert, adversarial training, and psum models with the aeda data augmentation technique. we acquire our final results by combining labels produced for each token in the sequence using majority voting. ## dataset statistics the dataset was provided as part of the shared task of detecting disease mentions (task 10) at the social media mining for health 2022 (smm4h'22) @xcite . the initial 5k tweets were later expanded by the organizers to make a 90k training set. the dev and test sets contained 2.5k and 23.5k tweets. covid19 was by far the most discussed disease. the 10 most frequent disease words include covid (13478), covid19 (11661), cáncer (7705), covid-19 (7394), ansiedad (4921), diabetes (4253), discapacidad (3284), enfermedades (2567), enfermedad (2455), and depresión (2229). ## results as can be seen, english biobert achieves the best recall. our qualitative analysis indicates that there are already many similar words in english and spanish in the medical entities domain. in addition, the large amount of spanish data used for fine-tuning injects a sufficient amount of extra spanish knowledge into the model. biobert also outperforms the data augmentation methods since the 90k training set seems to be large enough to neglect the augmentation benefits. however, these models seem to be more promising in low-data regimes as we can see from ## error analysis a confusion matrix of the two classes indicates that the number of false positives was almost triple that of false negatives @xcite . examining the top misclassified tokens ( ## conclusion we introduced and successfully evaluated adversarial and augmentation strategies, which proved to be significantly influential in a low-data regime, to be combined with the biobert model in order to address the problem of detecting disease mentions in spanish tweets. we also showed that the english biobert can have a strong performance on the same dataset; this behavior can be attributed to the dataset domain (medical entities), where the two languages share many similar words. we believe that this can be applied to other languages as well, as long as there are some similarities between them.