Health Sentinel: An AI Pipeline For Real-time Disease Outbreak Detection (2025)

Abstract: Early detection of disease outbreaks is crucial to ensure timely intervention by the health authorities. Due to the challenges associated with traditional indicator-based surveillance, monitoring informal sources such as online media has become increasingly popular. However, owing to the number of online articles getting published everyday, manual screening of the articles is impractical. To address this, we propose Health Sentinel. It is a multi-stage information extraction pipeline that uses a combination of ML and non-ML methods to extract events–structured information concerning disease outbreaks or other unusual health events–from online articles. The extracted events are made available to the Media Scanning and Verification Cell (MSVC) at the National Centre for Disease Control (NCDC), Delhi for analysis, interpretation and further dissemination to local agencies for timely intervention. From April 2022 till date, Health Sentinel has processed over 300 million news articles and identified over 95,000 unique health events across India of which over 3,500 events were shortlisted by the public health experts at NCDC as potential outbreaks.

## introduction disease surveillance is the continuous collection, analysis and interpretation of health data, particularly data concerning disease outbreaks and other unusual health events. it is essential that disease surveillance collects information in real time for timely interventions. further, continuous disease surveillance allows us to monitor disease spread patterns and allocate resources more effectively, directing attention and funding to areas where they are most needed. traditional surveillance approaches follow a bottom-up approach wherein information is collected from health care workers, public health fa-1 media scanning and verification cell (msvc) news article "two die of dengue in mizoram, 1 in manipur. meanwhile fortysix cases of chikungunya have been detected so far in assam taking the total number of infections to 70" health sentinel {'disease': 'dengue', 'location': 'mizoram', 'incident': 'death', 'incident type': 'new', 'number': '2'} {'disesae': 'dengue', 'location': 'manipur', 'incident': 'death', 'incident type': 'new', 'number': '1'} {'disease': 'chikungunya', 'location': 'assam', 'incident': 'case', 'incident type': 'total', 'number': '70'} {'disease': 'chikungunya', 'location': 'assam', 'incident': 'case', 'incident type': 'new', 'number': '46'} extracted events figure 1 : health sentinel extracts structured information from online articles reporting unusual health events. the given example shows how our pipeline extracts multiple events from a single news article. in contrast to the indicator-based surveillance, event-based surveillance looks at multiple sources of information either formal or informal such as print media reports, online articles, and social media posts. this approach is designed to detect unusual health events quickly, providing early warnings for potential outbreaks. owing to the nature of sources, data collected in this approach is likely to be noisy, redundant, and unstructured. consequently, information retrieval (ir) and natural language processing (nlp) techniques are increasingly being used in event-based surveillance systems to filter out irrelevant and redundant data to extract cohesive structured information. @xcite @xcite @xcite according to the 2011 census, india has a population exceeding 1.2 billion (of the registrar general and census commissioner of india). most of the neglected tropical diseases are prevalent in india. health threats triggered by climate change @xcite is another significant concern. health governance in india is decentralized where states hold primary responsibility for healthcare. traditional, indicator-based disease surveillance in such a setting demands a well coordinated system involving stakeholders belonging to different departments and different state governments. in this work, we present health sentinel, an information extraction pipeline, that feeds structured information concerning public health events to an event-based surveillance system in india. it extracts events related to 122 human and animal diseases which were prioritized based on inputs from public health experts. as shown in figure 2 , health sentinel follows a multi-stage process. it begins with data ingestion where articles are periodically crawled from the web. followed by this step, a binary text classifier filters out irrelevant articles. next, all articles are translated to english. once an article is identified to carry information on one or more unusual health events, we extract structured information from it. this is referred to as event extraction (ee) @xcite in information retrieval (ir) and natural language processing (nlp). in our case, an "event" comprises, i) disease-the specific disease or "others" if the disease is not among a predefined list of 122 diseases our health experts have curated, ii) location -the geographical area where the disease occurrence is reported, iii) incidentthe nature of the event, such as case or death , iv) incident type -whether the incident is new or total (cumulative), and v) number -the numerical value associated with the incident and its type (number of cases or number of fatalities). an example where 4 distinct events are extracted from a single article is shown in figure 1 . followed by event extraction, similar events are clustered together to isolate unique event occurrences. the unique events are finally passed on to an expert for further review. the highlights of health sentinel are listed as follows: ## related work most ml-based disease surveillance approaches employ rule-based techniques along with classical ml models @xcite . @xcite and promed @xcite are the two most popular disease surveillance systems. medisys uses patternmatching techniques to extract events from articles which leads to many false positives. since it is rulebased, extending it to other languages is non-trivial. on the other hand, in promed, filtering relevant information and further analysis is mostly performed by involving humans. in recent times, new tools like grits @xcite , eventepi @xcite and padi 3.0 @xcite have been developed for disease surveillance. although these systems use ml for tasks like classification and clustering, none of them exploit recent advances in event extraction using deep learning techniques and llms. ee, or extracting structured information from unstructured text, is a well studied problem in ir and nlp. deep learning-based ner and qa models have extensively been used for ee tasks @xcite . however, llms are increasingly being used for ee tasks in zero-shot, fewshot and finetuned settings @xcite . llms have demonstrated impressive results in information extraction tasks using few-shot approach without requiring task-specific fine-tuning @xcite . these results have further been validated by studies exploring use of openai models like gpt-3.5 and gpt-4 foot_0 for ee in various scenarios @xcite @xcite . similar to these works, we use llms for few-shot ee. dagdelen et al. demonstrate that gpt-3 and llama-2 (touvron et al., 2023) can be finetuned for extracting structured information from scientific text. we have not explored llm finetuning for ee for disease surveillance owing to the lack of training data. harrod et al. (harrod et al., 2024) use llms for extraction of structured epidemiological data from documents and geotagging each record. their work is similar to ours as they extract structured disease related information and use llms for the same. however, their work focuses on extracting information concerning rift valley fever (rvf) alone. secondly, the objective of their work is not disease surveillance but creation of a structured epidemiological dataset for rvf from past documents-pdfs of research articles and other documents concerning rvf. in contrast, our work uses llms for event extraction from web articles for real time surveillance of 100+ diseases. ## method in the following sections, we present details of each stage in the health sentinel system. the overall flow can be referred to in figure 2 . ## deployment and impact health sentinel was launched in april 2022 with support for english and hindi. over the two years, it has expanded the support to 11 additional languages. the system has undergone multiple upgrades, particularly in the event extraction module, to integrate the latest deep learning models and llms. to date, it has processed over 300 million articles from over 500,000 unique domains and has identified over 95,000 unusual unique health events. on a daily basis, health sentinel processes around 375,000 news articles and identify around 150 unusual unique health events. since deployment, over 3,500 events have been shortlisted by the health experts at ncdc. notably, only a small percentage of detected events were shortlisted by the human experts. this can be attributed to the following: i) duplicates that are not clustered correctly, ii) common outbreaks that occur during expected seasons are often not shortlisted by the health authority, and iii) if the disease is endemic in the location, such events are not shortlisted. to better understand the impact of health sentinel, we compare it with the human-based surveillance. we observed the following, i) number of published events saw a 150% increase compared to the previous years where only human-based surveillance existed ii) in 2024, 96% of the health events published by the surveillance system were extracted by the health sentinel (only 4% were found by manual scanning of the media), and iii) the number of media sources covered has grown exponentially because of automated media scanning and multilingual support. ## limitations and future work in this section, we discuss known limitations of our system. due to the lack of robust pre-trained models for indic languages for event extraction and text embeddings, we need to translate non-english articles to english following the article classification stage. we have observed that named entities such as disease names and location names are sometimes mistranslated, leading to a lower performance for articles sourced from these languages. we experimented with fine tuning indictrans2 @xcite framework with an emphasis on correctly translating or transliterating named entities as appropriate. while this improved translation of the entities, it led to a decline in overall performance of the model. another known limitation is the lack of full context for the event extraction step, since we read only title and description of an article. reading the full body of online articles is practically challenging since the body text is almost always clubbed with advertisements and other unrelated content and the format varies from page to page. currently we are developing custom html source parsers for selected websites so that full body of the articles from these websites can be read. ## conclusion in this paper, we presented health sentinel for automating media-based disease surveillance in a multilingual setting. health sentinel works by leveraging the capabilities of different sequentially connected machine learning models to maintain an optimal level of latency while maximizing the ability to identify unusual health events. health sentinel has demonstrated promising results across multiple evaluation metrics and has greatly increased the capability of disease surveillance in india.