The S ocial D is NER shared task on detection of disease mentions in health-relevant content from social media: methods, evaluation, guidelines and corpora (2022)

Abstract: There is a pressing need to exploit health-related content from social media, a global source of data where key health information is posted directly by citizens, patients and other healthcare stakeholders. Use cases of disease related social media mining include disease outbreak/surveillance, mental health and pharmacovigilance. Current efforts address the exploitation of social media beyond English. The SocialDisNER task, organized as part of the SMM4H 2022 initiative, has applied the LINKAGE methodology to select and annotate a Gold Standard corpus of 9,500 tweets in Spanish enriched with disease mentions generated by patients and medical professionals. As a complementary resource for teams participating in the SocialDisNER track, we have also created a large-scale corpus of 85,000 tweets, where in addition to disease mentions, other medical entities of relevance (e.g., medications, symptoms and procedures, among others) have been automatically labelled. Using these large-scale datasets, co-mention networks or knowledge graphs were released for each entity pair type. Out of the 47 teams registered for the task, 17 teams uploaded a total of 32 runs. The top-performing team achieved a very competitive 0.891 f-score, with a system trained following a continue pre-training strategy. We anticipate that the corpus and systems resulting from the SocialDisNER track might further foster health related text mining of social media content in Spanish and inspire disease detection strategies in other languages.

## introduction with more than 4.2 billion users worldwide, social media have become the most widely used digital platform for interacting with peers as well as accessing information relevant to specific groups @xcite . specifically twitter, an online micro-blogging social network (osn), has been widely used to extract information about people: from opinions and effects of environmental pollution @xcite to biomedical aspects such as adverse drug reactions @xcite , public health @xcite , and the psychological effects of a pandemic on the population @xcite . most analysis and information extraction studies are carried out in english, the main language used by the platform's users. however, other major languages such as spanish have generated a large amount of potentially usable data for analysis @xcite , which increases the impact of nlp systems able to extract information in this language. to date, the resources for working with twitter data in spanish have focused on corpora to extract information related to emotions @xcite and professions @xcite . recently, language models trained with spanish tweets have been developed to improve the performance of nlp tasks applied to data obtained from this osn @xcite . however, there is a clear lack of corpora to train systems capable of extracting biomedical information from spanish content that could be used for real-time mining of health information posted by patients, which is of growing interest for different scenarios such as screening for rare diseases @xcite . one of the main entities to effectively recognise health-related content is diseases. shared-task such as distemist @xcite , which focused on the detection of disease mentions in clinical cases, fostered the creation of many tools for this purpose. these systems focus on extracting information from technical and scientific texts, but their performance is limited to the more lay language used in social networks. in the socialdisner shared task, we have applied the knowledge and experience from clinical cases acquired in distemist to detect disease mentions in spanish tweets written by patients and medical professionals. shared task setting. the track comprised a single challenge in which participants were required to train nlp systems capable of detecting disease mentions automatically. we conducted the task using codalab in a three-stage scenario. in the practice phase, the training and evaluation set were released so that participants could build their systems and evaluate their models on the validation set. for the evaluation phase, the test and background set were released without annotations for the participant teams to compute and submit their predictions. teams were only evaluated on the test set, using the background data to avoid possible manual corrections and to evaluate the scalability of the systems. in this phase, the participants were only allowed to upload 2 runs. in order to further develop disease mention recognition systems in tweets, the competition has been kept open in the post-evaluation phase, so that researchers continue to measure and compare the performance of their systems. evaluation metrics. teams were evaluated and ranked using micro-average f1 score. in addition to this metric, micro-averaged precision and recall were computed. the evaluation script is available on github 1 . we also compared the systems versus a baseline model following a lexical search approach (temu-bsc, 2022). ## results participation. socialdisner achieved a considerable impact in the scientific-technical community. a total of 47 teams were registered, out of which 17 submitted a total of 32 runs for evaluation. although most of the teams came from academic environments, a significant number (4) came from industry. interestingly, 10 teams came from non-spanish-speaking countries, which indicates the community interest in developing systems in languages other than english. results. error analysis. in common with similar biomedical entity recognition tasks, the longer the mentions, the more difficult is for models to predict them correctly @xcite . in so-cialdisner we have found a correlation of -0.24 between the prediction errors of the systems and the length of the mentions, with a tendency to error when the length of the mention increases. regarding specific issues of detection, we have detected 4 common detection problems in several participating systems: systems are able to detect a lowercase mention, but not its uppercase version if they have not seen it before in the training set. ## discussion socialdisner is the first task focused on extracting diseases from social media content written in spanish. the first gold standard corpus of tweets with diseases annotated by medical experts has been built specially for the task. the corpus documents were selected to contain first-person patient experiences and relevant biomedical information written by experts. we also published additional resources such a large-scale silver standard corpus annotated with additional biomedical entities that might be used to train systems to detect entities in spanish tweets that previously could not be detected due to lack of resources. this large-scale corpus made it possible to generate co-mention networks that can be used toward knowledge graph mining to replicate studies on adverse drug effects in spanish-speaking people @xcite . a knowledge graph similar to the one in figure 2 might allow descriptive analysis of disease co-morbidities, to detect new symptoms in rare diseases, to discover diseases associated with specific professions, and even to discover adverse effects of biomaterials and prosthetics. socialdisner has been very well received by the community. there have been participants from the academy and the industry, probably enticed by the methodology followed to collect content, which ensured that a significant percentage of the documents were of high relevance. nevertheless, the modularity with which the methodology has been defined enables its improvement for future shared-tasks and projects. for example, more sophisticated first-person detection systems such as those developed in al-garadi et al. (2020) could be used, based on manual annotation of previous data. ner systems could also be trained with the large-scale corpus to improve the data enrichment process. this data selection methodology to retrieve relevant information could also be easily transferred to other languages as it relies on the presence of patient associations on twitter, which is relatively common. when organizing the task, we contacted several patient associations with twitter accounts. these groups showed interest in socialdisner, its results and how the output of the task may benefit the patients. the interest shown by associations, which in many cases helped to disseminate the event, shows the importance of involving all relevant stakeholders during the development and organization phases in order to increase the impact and use cases of our work.