KU _ ED at S ocial D is NER : Extracting Disease Mentions in Tweets Written in S panish (2022)

Abstract: This paper describes our system developed for the Social Media Mining for Health (SMM4H) 2022 SocialDisNER task. We used several types of pre-trained language models, which are trained on Spanish biomedical literature or Spanish Tweets. We showed the difference in performance depending on the quality of the tokenization as well as introducing silver standard annotations when training the model. Our model obtained a strict F1 of 80.3% on the test set, which is an improvement of +12.8% F1 (24.6 std) over the average results across all submissions to the SocialDisNER challenge.

## introduction in this system description paper, we aim to detect disease mentions from tweets written in spanish as part of the social media mining for health (smm4h) 2022 socialdisner task @xcite . the organizers provided the participants with various sets of data, either labelled by healthcare experts or machine-generated by the organizers themselves. all the sets contain anonymous tweets written in spanish, each saved as a txt file, with a corresponding tsv file that contains the label for each tweet. since the data is user-generated text limited to 280 characters as per the limitation from twitter, it contains misspelled words, abbreviations, emojis, links, hashtags, and mentions to other users of the platform, making the task challenging. acknowledging the outstanding performance for named entity recognition presented in @xcite and from its adaptation to the biomedical domain presented in @xcite , we decided to use pre-trained biomedical language models for detection of disease mentions in the socialdisner task. we selected four publicly available pre-trained language models on spanish corpora @xcite @xcite , one pre-trained on english corpora @xcite 2019) to retrain one of the models for named entity recognition. we reported the results of our experiments that range from 62.4% to 82.9% strict f1 score on the validation set. we also needed to convert the data to the correct format. we decided to use the bio tagging format (begin, interior, outside) (ramshaw and marcus, 1999), where each token within a tweet was coupled with one of the three tags. ## results and analysis of the error we reported the results of our experiments in looking at the mentions from the model compared to the mentions from the healthcare expert it seems that extra rules in the post-processing step could have been implemented to reduce the gap between both performances. for example when the identified token is part of a word, which is not a hashtag, extract the word instead of the token. ## conclusion in this work, we studied the difference in performance of pre-trained language models depending on the quality of the labels and tokenization for detection of disease mentions in tweets. at the end, the model achieved 80.3% f1 score on the test set. for future work, one direction would be to use ensemble models for token classification, where we combine a twitter pre-trained language model with a biomedical pre-trained language model.