Benchmarking for Public Health Surveillance tasks on Social Media with a Domain-Specific Pretrained Language Model (2022)

Abstract: A user-generated text on social media enables health workers to keep track of information, identify possible outbreaks, forecast disease trends, monitor emergency cases, and ascertain disease awareness and response to official health correspondence. This exchange of health information on social media has been regarded as an attempt to enhance public health surveillance (PHS). Despite its potential, the technology is still in its early stages and is not ready for widespread application. Advancements in pretrained language models (PLMs) have facilitated the development of several domain-specific PLMs and a variety of downstream applications. However, there are no PLMs for social media tasks involving PHS. We present and release PHS-BERT, a transformer-based PLM, to identify tasks related to public health surveillance on social media. We compared and benchmarked the performance of PHS-BERT on 25 datasets from different social medial platforms related to 7 different PHS tasks. Compared with existing PLMs that are mainly evaluated on limited tasks, PHS-BERT achieved state-of-the-art performance on all 25 tested datasets, showing that our PLM is robust and generalizable in the common PHS tasks. By making PHS-BERT available, we aim to facilitate the community to reduce the computational cost and introduce new baselines for future works across various PHS-related tasks.

## introduction public health surveillance (phs) is defined by the world health organization foot_1 as the ongoing, systematic collection, assessment, and understanding of health-related required information for the planning, implementation, and assessment of healthcare @xcite . phs aims to design and assist interventions; it acts as a primary warning system in health emergencies (epidemics, i.e., acute events), it reports and records public health interventions (i.e., monitoring health), and it observes and explains the epidemiology of health issues, allowing for the prioritization of necessary details for health policy formulation (i.e., targeting chronic events). traditional phs systems are often limited by the time required to collect data, restricting the quick or even instantaneous identification of outbreaks @xcite . social media is growingly being used for public health purposes and can disseminate disease risks and interventions and promote wellness and healthcare policy. social media data provides an abundant source of timely data that can be used for various public health applications, including surveillance, sentiment analysis, health communication, and analyzing the history of a disease, injury, or promote health. systematic reviews of studies that examine personal health experiences shared online reveal the breadth of application domains, which include infectious diseases and outbreaks @xcite , illicit drug use @xcite , and pharmacovigilance support @xcite . these applied health studies are motivated by their potential in supporting phs, augmenting adverse event reporting, and as the basis of public health interventions @xcite . the use of deep learning in natural language processing (nlp) has advanced the development of pretrained language models (plms) that can be used for a wide range of tasks in phs. however, directly applying the state-of-the-art (sota) plms such as bidirectional encoder representations from transformers (bert) @xcite , and its variants @xcite @xcite that are trained on general domain corpus (e.g., bookcorpus, wikipedia, etc.) may yield poor per-formances on domain-specific tasks. to address this limitation, several domain-specific plms have been presented. some of the well-known in the biomedical field include the following: biomedical bert (biobert) @xcite and biomedical a lite bert (bioalbert) @xcite . recently, other domain-specific lms such as bertweet @xcite ) for 3 downstream tasks, i.e., part-of-speech tagging, named-entity-recognition, and text classification and covid twitter bert (ct-bert) @xcite for 5 text classification tasks have been trained on datasets from twitter. despite the number of plms that have been released, none have been produced specifically for phs from online text. furthermore, all these lms were evaluated with the selected dataset, and therefore their generalizability is unproven. to benchmark and fill the gap, we present phs-bert, a new domain-specific contextual plm trained and fine-tuned to achieve benchmark performance on various phs tasks on social media. phs-bert is trained on a health-related corpus collected from user-generated content. our work is the first largescale study to train, release and test a domainspecific plm for phs tasks on social media. we demonstrated that phs-bert outperforms other sota plms on 25 datasets from different social media platforms related to 7 different phs tasks, showing that phs is robust and generalizable. ## method phs-bert has the same architecture as bert. fig. 1 illustrates an overview of pretraining, finetuning, and datasets used in this study. we describe bert and then the pretraining and fine-tuning process employed in phs-bert. ## conclusion we present phs-bert, a domain-specific plm trained on health-related social media data. our results demonstrate that using domain-specific corpora to train general domain lms improves per-formance on phs tasks. on all 25 datasets related to 7 different phs tasks, phs-bert outperforms previous state-of-the-art plms. we expect that the phs-bert plm will benefit the development of new applications based on phs nlp tasks. ethics and societal impact ethics: no additional ethics approval was sought for the analysis of data in this study because data were drawn from already published studies.