FRE at S ocial D is NER : Joint Learning of Language Models for Named Entity Recognition (2022)

Abstract: This paper describes our followed methodology for the automatic extraction of disease mentions from tweets in Spanish as part of the SocialDisNER challenge within the 2022 Social Media Mining for Health Applications (SMM4H) Shared Task. We followed a Joint Learning ensemble architecture for the fine-tuning of top performing pre-trained language models in biomedical domain for Named Entity Recognition tasks. We used text generation techniques to augment training data. During practice phase of the challenge our approach showed results of 0.87 F1-Score.

## introduction twitter is one of the main channels for communication at present. the high number of users and the daily activity worldwide on this platform, in addition to the nature of tweets, i.e., short informal text messages, make it as an ideal source of information for natural language processing (nlp) tasks @xcite . the combination of such amount of information with pre-trained transformer language models has changed the way we do nlp nowadays @xcite . these pre-trained language models have brought significant breakthroughs in deep learning, creating a revolutionary ecosystem for nlp techniques, such as named entity recognition, text classification, etc. in the 2022 socialdisner challenge, we focused on finetuning several pre-trained language models architectures to detect disease mentions in tweets. ## system description the summary of our system is the following: first, we search for the two best performing pre-trained language models for our task. then, we used data augmentation techniques to extend the initial training data and fine-tune the selected language models with the joint learning ensemble approach described in @xcite . finally, by qualitative analysis of results, we cleaned the model predictions with post-processing techniques. ## evaluation during the practice phase of the challenge, participants were given the opportunity to test their approach against the evaluation system from the organizers to assess the quality in an intermediate stage. during this phase we obtained strict f1-scores of 0.84 with our joint-embedding model fine-tuned with the augmented data and 0.87 with our post-processing techniques. two sets of predicted data were submitted for final evaluation. submission 1 consisted of the joint learning model predictions with our postprocessing cleaning and submission 2 consisted of the joint learning model predictions without any kind of post-processing. performances of each submission are presented in ## conclusion in this paper we presented our system focused on fine-tuning two roberta-based language models together with our joint learning approach for disease mentions detection task in spanish tweets. we extended the training data by generating text with a gpt-based model fine-tuned with biomedical and twitter data. combining the joint learning and augmented data, our system achieved 0.87 f1-score in the practice phase of the challenge. our final results in evaluation phase of the challenge achieved 0.73 and 0.74 f1-scores. in the future, we plan to apply transfer learning techniques to extend our system to the analysis of social media messages in other languages.