FORCE : A Benchmark Dataset for Foodborne Disease Outbreak and Recall Event Extraction from News (2024)

Abstract: The escalating prevalence of food safety incidents within the food supply chain necessitates immediate action to protect consumers. These incidents encompass a spectrum of issues, including food product contamination and deliberate food and feed adulteration for economic gain leading to outbreaks and recalls. Understanding the origins and pathways of contamination is imperative for prevention and mitigation. In this paper, we introduce FORCE Foodborne disease Outbreak and ReCall Event extraction from openweb). Our proposed model leverages a multi-tasking sequence labeling architecture in conjunction with transformer-based document embeddings. We have compiled a substantial annotated corpus comprising relevant articles published between 2011 and 2023 to train and evaluate the model. The dataset will be publicly released with the paper. The event detection model demonstrates fair accuracy in identifying food-related incidents and outbreaks associated with organizations, as assessed through cross-validation techniques.

## introduction the escalating number of food safety concerns remains a source of significant apprehension @xcite @xcite @xcite . globally, foodborne diseases continue to plague populations and stand as leading contributors to both illness and mortality @xcite @xcite . recent estimates have identified norovirus and campylobacter as the most common reason behind foodborne illnesses, while fatalities have been associated with non-typhoidal salmonella enterica, salmonella typhi, taenia solium, hepatitis a virus, and aflatoxin @xcite @xcite . one of the repercussions of food safety issues is the necessity for food recalls, which pose substantial economic threats to both businesses and nations alike @xcite . this underscores the imperative of uncovering the root causes behind these incidents and the factors contributing to contamination @xcite . cross-contamination in food and beverages is a multifaceted issue that can transpire at various stages of the food processing chain, including external raw food contamination, transportation, cleaning processes, heating, food packaging, and even during food storage. contamination events resulting in outbreaks can manifest at any point before, during, or after food processing @xcite . consequently, the pivotal task of identifying the origins of contamination or the triggers for recalls is of paramount importance @xcite . it is essential to gain insights into potential sources and pathways of contamination leading to foodborne outbreaks and product recalls, and to devise effective measures for prevention @xcite @xcite . it is worth noting that there is a dearth of substantial work in the development of computational and/or analytical models addressing these concerns @xcite . this scarcity of research can largely be attributed to the limited availability of data concerning the contributory factors associated with food safety incidents. as such, there is a pressing need to develop an automated tool that can mine reported food safety incidents and recalls to bridge this knowledge gap. applications of language technologies and data science for food-borne risk assessment are gaining ground @xcite @xcite . data-intensive systems play important roles in tracking food-borne illness cases and agents @xcite @xcite @xcite . examples at the us federal level include pulsenet @xcite , the national antimicrobial resistance monitoring system (narms) @xcite , foodnet @xcite , and the national outbreak reporting system @xcite . implementation of whole-genome sequencing (wgs) in surveillance and outbreak investigation has fueled an explosion of publicly available foodborne pathogen genomes in new systems such as genome-trakr @xcite , enterobase @xcite , and the national center for biotechnology information's pathogen detection. these works are primarily focused on a) identifying the sentiment polarity of the documents, and b) identifying the occurrences of a set of predefined types of entities and events. however, none of the above works perform deep linguistic analysis of the textual contents to identify food-related incidents based only on the linguistic structure of the text. the model presented in this paper is distinct from the earlier approaches since it is capable of detecting novel and heretofore unknown incidents from reports based only on semantic and linguistic analysis of content. keeping in mind the above-mentioned requirements of end users, in this work we propose an event detection model that can identify food safetyrelated insights, recalls, and outbreaks mentioned in regulatory reports and social media platforms. the proposed model uses a multi-tasking sequence labeling architecture that works with transformerbased document embeddings. we have created a large annotated corpus containing relevant articles published by multiple regulatory agencies over twelve years @xcite for training and evaluating the model. the dataset will be publicly released with this paper. the model has been thereafter applied to recent publications. aggregate analysis of these extracted insights reveals interesting trends. ## dataset creation the dataset comprises regulatory news articles from two sources namely, a) a corpus of 6000 regulatory articles comprising around 121080 sentences, under outbreak(o) and recall(r) categories published between 2011 and 2023 by food safety news (fsn) foot_0 and b) a collection of around 2200 news articles from united states food and drug administration (fda) recall and outbreak announcements foot_1 . all together there are 8100 articles. all the news articles were manually annotated to mark the various target entities. the annotation was done by multiple annotators following a rigorous procedure to ensure acceptable inter-annotator agreement. the entities and events to be identified by the annotators are as follows: target organization (to): of the many organization names that may appear in a document and are detected by the ner earlier, the task during annotation is to identify and tag the organization whose product has been recalled. product name (p): the name of the product that has been recalled or caused the outbreak. infection name (i): the name of the bacterial infection mentioned in the report that causes the outbreak/recall. safety incident (si) -annotators have to tag phrases or sequences of words that collectively are indicative of the food safety incident. cause of incident (ci) -annotators have to tag phrases or sequences of words that indicate the primary cause of the food safety incident. number of people affected (n): annotators have to tag phrases or sequences of words that collectively are indicative of a number of people affected due to the outbreak. to help the annotators, each document is first processed using the stanford ner @xcite to obtain the organization names, locations, and currency values as named entities. this helps in quick localization of the first four elements, if present in the document. the annotators are domain experts who are knowledgeable about the domain. task-2: -this task had two components: (a) from among the named entities, the target organization, product name, infection, and locations were marked, if any, and (b) mark phrases in the text that indicate food safety incidents, and its cause. at the end of the annotation, each word in the document is assigned a label to, p, i, si, ci, n, or none. ## a multi-tasking neural model for food safety knowledge extraction the proposed model works on each sentence at a time to detect elements of interest that are defined in the incident knowledge schema. multi-task learning utilizes the correlation between related tasks to improve classification by learning tasks in parallel. in the present work, the two related tasks are task-1: classifying a sentence into either recall or outbreak classes as discussed earlier and task-2: labeling appropriate phrases in the text as per the incident knowledge schema. a cascaded cnn-bilstm layer for the combined tasks of sentence classification and sequence label prediction, using the fine-tuned bert for creating the sequence embeddings. to obtain the multi-tasking model for dual tasks of sequence classification and sequence labeling, the bert -cn n -bilst m layers have been trained with two separate loss functions l 1 and we define the joint loss function using a linear combination of the loss functions of the two tasks: where λ controls the contribution of losses of the individual tasks in the overall joint loss. i [@xmath0. ## evaluation and results the performance of the proposed model has been compared with a number of baseline models used for single-objective document classification and sequence labeling tasks as well as large language models like llama-2 7b and fined-tuned mistral 7b. the primary reason for the poor performance of llama-2 as well as mistral-7b can be attributed due to two reasons: a) lack of environmental domain knowledge due to which critical domain concepts like, salmonella, listeria monocytogenes and e.coli gets ignored. b) unable to identify phrase boundaries. we observe that despite in most of the cases llama-2 correctly identified the safety incident and cause phrases, but the span of the phrases are either too long or too short. as a results of which outputs of the model get penalized. similar observations were made for mistral 7b, however, since the mistral model is fine-tuned over the current dataset, problems related to domain concept mismatch were relatively less. however, the output word span detection for incident and causes still remains a challenge. apart from the classification and extraction of food safety incidents, it is equally important to perform some basic analytics on the dataset. figure 1 shows the distribution of the top five causes of food recall across different locations in the united states. in general, we have observed that infections such as salmonella, listeria monocytogenes and e.coli are the biggest causes of food recall in the united states. figure 2 depicts the top food products that contain the aforementioned germs. ## conclusion in this paper we present resource creation and extraction of critical information related to food safety and food-borne infection from regulatory reports and social media platforms. the proposed model, founded on a multi-tasking sequence labeling architecture integrated with transformer-based document embeddings, demonstrates its effectiveness in this task. to develop and evaluate our model, we meticulously curated an annotated corpus comprising pertinent articles. our initial analysis demonstrate the proposed multi-task model surpasses the performance of almost all the baseline models including llms such as llama-27b and finetuned mistral-7b.