RACAI @ SMM 4 H ’22: Tweets Disease Mention Detection Using a Neural Lateral Inhibitory Mechanism (2022)

Abstract: This paper presents our system employed for the Social Media Mining for Health (SMM4H) 2022 competition Task 10 - SocialDisNER. The goal of the task was to improve the detection of diseases in tweets. Because the tweets were in Spanish, we approached this problem using a system that relies on a pre-trained multilingual model and is fine-tuned using the recently introduced lateral inhibition layer. We further experimented on this task by employing a conditional random field on top of the system and using a voting-based ensemble that contains various architectures. The evaluation results outlined that our best performing model obtained 83.7% F1-strict on the validation set and 82.1% F1-strict on the test set.

## motivation social media data (e.g. twitter, facebook) analysis for health informatics is an active area of research that aims to understand public opinion of health-related topics. the social media mining for health applications (#smm4h) workshop is a venue that brings together researchers that want to contribute to this area, including, but not limited to subjects such as: deriving health trends from social media, health-related message classification or disease monitoring from messages on social media. we chose to participate at the tenth task of the #smm4h competition -socialdisner -whose aim was to improve the disease detection in spanish tweets by identifying enfermedad entities in the given text @xcite . we approached this problem using the xlm-roberta multilingual pre-trained model @xcite ) on which we incorporated the recently introduced lateral inhibitory (li) layer (pȃis , , 2022) in the finetuning process, together with a conditional random field (crf) @xcite layer on top of the model and an ensemble system. the interest in this task came from our involvement within the curlicat project for the cef at action @xcite , where named entity recognition systems need to be developed for various domains (health-related included), in romanian language. ## system description we treat the socialdisner task as a sequence tagging problem, so we try to predict a label for each token given as input. the architecture of our system consists of using the xlm-roberta pre-trained model that produces an embedding for each token in the input sequence. then, we employ the li layer on each token embedding, followed by a dense layer to produce the output probabilities for each of the three possible classes: o, b-enfermedad or i-enfermedad foot_0 . we further use a crf layer on top of the predicted logits to it must be noted that we also experimented with models that do not include the li and/or the crf layers. in addition, we created a voting-based ensemble system that incorporates the predictions of all four architectures developed in this work: xlm-roberta, xlm-roberta+li, xlm-roberta+crf and xlm-roberta+li+crf. we fine-tune each model for 80 epochs using a batch size of 16 and a gradient accumulation of 8, resulting in a total batch size of 128. we use the adam optimizer (kingma and ba, 2014) with a learning rate of 2e-5 and the default values for the weight decay, epsilon and the betas in pytorch @xcite . during training, we save only the checkpoint that obtained the highest performance on the validation set. ## evaluation we evaluate and present the results of our models on the validation and test sets. the results of the xlm-roberta with different layers and of the ensemble system are listed in the results on the test set are outlined in ## conclusion analysing social media data remains an important area of research for health informatics, offering a better understanding of public opinion in healthrelated topics. this work describes our system used to participate at the smm4h competition socialdisner shared task. we treated the problem as a sequence tagging task and employed the xlm-roberta multilingual pre-trained model to process the spanish tweets. in addition, we analysed the capabilities of the li layer on this task in combination with a crf module. we sent to evaluation the xlm-roberta model with the li layer, as well as an ensemble system. the two systems obtained 82.1% and 82.0% f1-strict scores on the evaluation set, respectively. no additional resources were employed. in the future we plan to apply this experience also on romanian microblogging text @xcite .