AILAB - U dine@ SMM 4 H ’22: Limits of Transformers and BERT Ensembles (2022)

Abstract: This paper describes the models developed by the AILAB-Udine team for the SMM4H’22 Shared Task. We explored the limits of Transformer based models on text classification, entity extraction and entity normalization, tackling Tasks 1, 2, 5, 6 and 10. The main takeaways we got from participating in different tasks are: the overwhelming positive effects of combining different architectures when using ensemble learning, and the great potential of generative models for term normalization.

## introduction transformer-based models are the backbone of state-of-the-art solutions for a lot of nlp tasks. the real strength of these models (like bert @xcite @xcite stands in the pre-training phase which permits them to have extensive language knowledge. this is particularly helpful in tasks where the amount of training data is restricted, like the ones addressed in this workshop @xcite . in this work we used a variety of pretrained transformers models for the tasks. we refer to ## multitask classification (task 2a+2b) task 2a and 2b @xcite consist in the classification of english tweets containing opinions about mandates during the covid-19 pandemic. the tweets can deal with three topics: face masks (m), stay at home orders (h) and school closures (s). task 2a is a ternary stance classification (against, none, favor), while task 2b is a binary premise classification (1, 0) to determine whether the tweet is argumentative or not. ## ade normalization (task 1c) task 1c consists in mapping ade mentions from english tweets to their corresponding meddra terms (formal medical terms). the dataset @xcite was developed with three sub-tasks in mind, so to perform task 1c it is necessary to complete the two preliminary tasks 1a (binary classification ade/noade) and 1b (ade extraction). ## conclusions we explored the use of simple transformerbased architectures for several tasks proposed by smm4h'22. the most noticeable phenomena we encountered were: the collaborative effect of different architectures (e.g., bert and roberta) when used in ensemble learning (task 2 and 5, as opposed to task 6); the efficacy of generative models for term normalization (task 1c); and the low transferability of methods developed for ade extraction to disease detection (task 10).