The P rof NER shared task on automatic recognition of occupation mentions in social media: systems, evaluation, guidelines, embeddings and corpora (2021)

Abstract: Detection of occupations in texts is relevant for a range of important application scenarios, like competitive intelligence, sociodemographic analysis, legal NLP or health-related occupational data mining. Despite the importance and heterogeneous data types that mention occupations, text mining efforts to recognize them have been limited. This is due to the lack of clear annotation guidelines and high-quality Gold Standard corpora. Social media data can be regarded as a relevant source of information for real-time monitoring of at-risk occupational groups in the context of pandemics like the COVID-19 one, facilitating intervention strategies for occupations in direct contact with infectious agents or affected by mental health issues. To evaluate current NLP methods and to generate resources, we have organized the ProfNER track at SMM4H 2021, providing ProfNER participants with a Gold Standard corpus of manually annotated tweets (human IAA of 0.919) following annotation guidelines available in Spanish and English, an occupation gazetteer, a machine-translated version of tweets, and FastText embeddings. Out of 35 registered teams, 11 submitted a total of 27 runs. Best-performing participants built systems based on recent NLP technologies (e.g. transformers) and achieved 0.93 F-score in Text Classification and 0.839 in Named Entity Recognition. Corpus: https://doi.org/10.5281/zenodo.4309356

## introduction the number of social media users and content is rapidly growing, with over 700 million tweets posted daily @xcite . the social media mining 4 health (smm4h) effort @xcite attempts to promote the development and evaluation of nlp and text mining resources to extract automatically relevant healthrelated information from social media data. as social media content is produced directly by users at a global scale, a variety of application scenarios of medical importance have been explored so far, including the use of social media for pharmacovigilance @xcite , medication adherence, @xcite or tracking the spread of infectious and viral diseases @xcite . the spanish-speaking community on twitter is large, exceeding 30 million @xcite , which motivated the implementation of text mining efforts for health-related applications, in particular on drug-related effects @xcite @xcite . one of the current challenges for health-related social media applications is to generate more actionable knowledge that can drive the design of intervention plans or policies to improve population health. this is particularly true for infectious disease outbreaks like the covid-19 pandemic, where certain occupational groups and subpopulations have been at higher risk, due to direct exposure to infected persons, a higher degree of social interaction, work-related travels to high-risk areas or mental-health problems associated to workinduced stress. early detection and characterization of at-risk professions is critical to design and prioritize preventive and therapeutic measures or even vaccination plans. to date, occupational text mining has been used in clinical narratives @xcite and to explore injuries in the construction sector @xcite , mainly in english. however, it has not yet been systematically used in social media and clinical content in spanish. to implement a central nlp component for occupational data mining, namely the automatic detection of occupation mentions in social media texts, we have organized the profner (smm4h 2021) shared task. in this article, occupation mentions are all those elements that indicate the employment information of a person. within occupations, we have identified three main labels: (i) "profession", occupations that provide a person with a wage or livelihood (e.g., nurse); (ii) "activity", unpaid occupations (e.g., activist); and (iii) "working status" (e.g., retired). resources released for the profner track included annotation guidelines in spanish and english, a consistency analysis to characterize the quality and difficulty of the track, a large collection of manually annotated occupation mentions in tweets, as well as fasttext word embeddings generated from a very large social media dataset. we foresee that the occupation mention recognition systems resulting from this track could serve as a key component for more advanced text mining tools integrating technologies related to opinion mining, sentiment analysis, gender-inequality analysis, hate speech or fake news detection. moreover, there is also a clear potential for exploring occupation recognition results for safety management, risk behavior detection and social services intervention strategies. ## results participation overview. since this is the first task on the detection of occupational entities in social media, profner has received considerable attention from the community. indeed, 35 teams registered for the task (31 for the subtrack a and 29 for the subtrack b), and there were 27 submissions ## discussion to the best of our knowledge, profner is the first occupational data mining effort in social media. it is also the first shared task on health applications of social media in spanish. specifically for the shared task, we have built a pioneer gold standard for named entity recognition (ner) of occupations in social media in spanish, the profner corpus. it was generated following the profner annotation guidelines that are shared with the community in spanish and in english. finally, we have trained and released the first word embeddings for our use case (twitter posts in spanish related to covid-19) and a gazetteer of relevant terms. in addition, the profner shared task can be used as template for future shared tasks on the recognition of occupations in social media. indeed, the english translation of the annotation guidelines eases this research possibility. profner has aroused interest from both academia and industry. interestingly, teams from non-spanish speakers have participated in this task. tweet classification systems reach high performances. however, the detection and classification of occupational data can still be improved. we propose the use of the whole ecosystem of occupational text mining resources generated in this shared task (corpus, systems, guidelines, etc.) for building and evaluating novel systems that allow subpopulation characterization in social media in the current pandemic. this system has also the potential to assist public health policy makers in the prevention and management of current and future epidemics. moreover, beyond classical evaluation scenarios focusing on traditional quality evaluation using metrics like precision and recall, there is also a need to propose shared tasks and community benchmark efforts that do take into account involvement of end users, as was the case of the biocreative interactive task @xcite or the technical evaluation on the robustness and implementation of named entity components @xcite , especially when considering the volume and rapid change of social media data.