PLN CMM at S ocial D is NER : Improving Detection of Disease Mentions in Tweets by Using Document-Level Features (2022)

Abstract: This paper describes our approaches used to solve the SocialDisNER task, which belongs to the Social Media Mining for Health Applications (SMM4H) shared task. This task aims to identify disease mentions in tweets written in Spanish. The proposed model is an architecture based on the FLERT approach. It consists of fine-tuning a language model that creates an input representation of a sentence based on its neighboring sentences, thus obtaining the document-level context. The best result was obtained using an ensemble of six language models using the FLERT approach. The system achieved an F1 score of 0.862, significantly surpassing the average performance among competitor models of 0.680 on the test partition.

## introduction understanding the social perception of diseases is crucial for quantifying the effectiveness of public policies on medicine. in particular, analyzing social network data allows us to study this issue in depth. in this context, the socialdisner shared task @xcite aims to better understand diseases' social perception by using named entity recognition (ner) to identify diseases mentioned in twitter data: from highly prevalent conditions such as cancer and diabetes to rare immunological and genetic diseases. in this paper, we discuss our approaches used to solve the socialdisner task. specifically, we use several language models trained on both general and domain-specific corpora, exploiting the use of document-level features. finally, we study the use of ensembles and their impact on performance compared to individual models. ## methodology this section describes the models used in our experiments and the chosen baseline. ## experiments the training process for the flert-based models was analogous to each language model. we searched for an optimal learning rate between 1e-5, 5e-5, 5e-6, and 1e-6. for the sake of brevity, we only present the best model, trained with a learning rate of 5e-6. we used the adam optimizer with linear decay and no warm-up steps. the models were trained for 20 epochs using a batch size of 16 sentences with a maximum length of 512 tokens. in contrast, the baseline model was trained with a learning rate of 2e-5 for 3 epochs with a weight decay of 0.01, which is the setting that gave us the best result. the rest of the hyperparameters are the same. the training of each model took approximately 3 hours using a tesla v100 gpu. as shown in