S em E val-2025 Task 9: The Food Hazard Detection Challenge (2025)

Abstract: In this challenge, we explored text-based food hazard prediction with long tail distributed classes. The task was divided into two subtasks: (1) predicting whether a web text implies one of ten food-hazard categories and identifying the associated food category, and (2) providing a more fine-grained classification by assigning a specific label to both the hazard and the product. Our findings highlight that large language model-generated synthetic data can be highly effective for oversampling long-tail distributions. Furthermore, we find that fine-tuned encoder-only, encoder-decoder, and decoder-only systems achieve comparable maximum performance across both subtasks. During this challenge, we are gradually releasing (under CC BY-NC-SA 4.0) a novel set of 6,644 manually labeled food-incident reports.

## introduction the food hazard detection challenge at semeval 2025 evaluated classification systems for titles of food-incident reports collected from the world wide web. algorithms like these could, for example, be used to help automated crawlers find and extract food issues from publicly available sources like social media. since such systems could have a high economic impact (specific food items may need to be recalled, leading to financial damage for the producers), transparency is extremely important. human experts using data from these crawlers need to be well-informed about how the respective food issues are extracted. prior work has shown that a major challenge in food-hazard and food-product classification from text is the large number of possible classes, combined with a long-tail distribution @xcite . to address this, we define two subtasks: year e.g.: 2022 title + text e.g.: crave stevia brand chocolate products recalled due to undeclared milk -recalls, advisories and safety alerts -canada.ca features sub-task 1 (st1) hazard-category [main] product-category e.g.: allergens e.g.: cocoa and cocoa preparations, coffee and tea sub-task 2 (st2) hazard [main] • subtask 2 (st2) is a more fine-grained "vector" prediction task. a prior semeval challenge by @xcite ) framed a similar setup as an initial step toward explainability. while this interpretation may be somewhat broad, we recognize that a "vector" prediction task is particularly valuable for automated information extraction, as it provides more specific information. an overview of the semeval-task is shown in figure 1 . it includes two sub-tasks: (st1) text classification for food hazard prediction, predicting the type of hazard (hazard-category) and the type of product (product-category); (st2) food hazard and product "vector" detection, predicting the exact hazard (hazard) and product (product). the task was primarily concerned with detecting the hazard (more important than the product), hence a two-step scoring metric based the validation data was made available, and final submissions for both tasks were accepted on the test data to determine the final ranking. on the macro f 1 score was used, focusing on the respective hazard label per sub-task (see section 4). ## task organization the detailed timeline of the project is illustrated in figure 2 . participants were provided with training and validation data to develop, train, and evaluate their systems before the evaluation phase. the challenge was conducted on codalab 1 @xcite , adhering to the framework of previous competitions @xcite . the validation data was made available at the start of the challenge, enabling participants to submit to the leaderboards and compare their systems during the conception phase. however, these rankings did not influence the final results. the test set was released at the beginning of the challenge with labels concealed until its conclusion. during the evaluation phase, models could be trained on both the training and validation data but were evaluated exclusively on the test set to get the final ranking. after the evaluation phase, participants were required to submit a brief system description specifying the dataset features used, with this information made public alongside the final ranking. participants could submit up to five times per day and 100 times in total during the conception phase, whereas in the evaluation phase, each participant was limited to a single valid submission. additionally, participants were required to share their code (e.g., via github) along with their system description papers. ## dataset the dataset we used in the challenge is a subset of the data described in @xcite and publicly accessible on zenodo @xcite . it consists of 6,644 titles (length in characters: @xmath0, @xmath1, @xmath2, and full texts (length in characters: @xmath3, @xmath4, @xmath5 pljudwlrq irrgdgglwlyhvdqgiodyrxulqjv rujdqrohswlfdvshfwv sdfndjlqjghihfw rwkhukd]dug fkhplfdo iudxg iruhljqerglhv elrorjlfdo doohujhqv d kd]dugfdwhjru\ vdpsohv vfrw vj qd kn lh xn fd dx xv f frxqwu\ vdpsohv vxjduvdqgv\uxsv ihhgpdwhuldov irrgfrqwdfwpdwhuldov krqh\dqgur\domhoo\ irrgdgglwlyhvdqgiodyrxulqjv idwvdqgrlov shwihhg rwkhuirrgsurgxfwpl[hg khuevdqgvslfhv glhwhwlfirrgvirrgvxssohphqwviruwlilhgirrgv qrqdofrkrolfehyhudjhv frqihfwlrqhu\ dofrkrolfehyhudjhv frfrddqgfrfrdsuhsdudwlrqvfriihhdqgwhd lfhvdqgghvvhuwv qxwvqxwsurgxfwvdqgvhhgv vrxsveurwkvvdxfhvdqgfrqglphqwv vhdirrg suhsduhgglvkhvdqgvqdfnv iuxlwvdqgyhjhwdeohv fhuhdovdqgednhu\surgxfwv phdwhjjdqggdlu\surgxfwv e surgxfwfdwhjru\ vdpsohv g \hdu wudlq ydolg whvw english food recall announcements from the official websites of food agencies (e.g. the fda's website). in addition, the dataset contains meta information such as date of download and country of issue. these texts were primarily gathered between 2012 and 2022 from domains based in the united states, australia, canada, and the united kingdom (see figure 3 (c) and (d) ). the data was manually labeled with the reason for recall (hazard) and the recalled product. although neither title nor text individually is guaranteed to contain information about the product and hazard involved in a recall, their combination reliably provides the necessary details for classification. the distribution of this information between title and text varies across the dataset and largely depends on the issuing authority. each pair of title and text has been assessed by two experts on food science or food technology from agroknow foot_0 . some sample titles are shown in the data was stratified based on the more important hazard "vectors" (hazard) and divided into three subsets: 5,082 samples for training, 565 for validation, and 997 for evaluation. the training data, which was already published on zenodo @xcite , also contains additional non-english texts that could be used by participants to train their classifiers. nevertheless, our evaluation was only based on english texts. as the texts contain varying degrees of information on the haz-ard, we considered careful pre-processing of the data as part of the challenge. upon completion of the task, the complete dataset was made available under the creative commons by-nc-sa 4.0 license. one sample of the dataset is shown in figure 1 . as described above, the data includes the features year, month, day, country, text and ti-tle. participants performed their text analysis primarily on the title or text fields, while additional features were available if needed. the task was to predict the labels product-category and hazard-category, as well as the vectors product and hazard. the dataset comprises 1,256 different product values (e.g., "ice cream," "chicken based products," "cakes") sorted into 22 categories (e.g. "meat, egg and dairy products," "cereals and bakery products," "fruits and vegetables") with the help of ontologies. in addition, there are 261 distinct values for hazard (e.g., "salmonella," "listeria monocytogenes," "milk and products thereof ") , which are grouped (again using ontologies) into the following 10 values of the la-bel hazard-category: "allergens," "biological," "foreign bodies," "fraud," "chemical," "other hazard," "packaging defect," "organoleptic aspects," "food additives and flavourings," "migration." the class distribution in the data is heavily imbalanced with the above examples being ranked from the most to the least common in figure 3 . ## evaluation we computed the performance for st1 and st2 by calculating the macro f 1 -score on the participants' predicted labels ŷ using the annotated labels y as ground truth. this measure is the unweighted mean of per-class-f 1 -scores over the n classes. both ŷ and y are vectors of m samples: here @xmath6, y p ] is the 2 × m matrix with the hazard label y h and the product label y p as column vectors. the vector y p|h is defined as the entries of y p where y h is correctly predicted: the scalar y * j is the j-th element of y * . ŷ and ŷp|h are defined accordingly. with this measure we based our rankings predominantly on the predictions for the hazard classes. intuitively, this means that a submission with both y h and y p completely right would have scored 1.0, a submission with y h completely right and y p completely wrong would have scored 0.5, and any submission with y h completely wrong would have scored 0.0 independently of the value of y p . ## participant systems and results in total, our task attracted approximately 260 participants and received 99 valid submissions during the evaluation phase. among these, 27 system description papers were submitted for peer-review. these 27 systems form the basis of our analysis and the official ranking, as they are accompanied by detailed system descriptions, enabling a thorough evaluation. the full, unofficial ranking -including all submissions to codalab -is available on the task's website. 4 ## conclusion in conclusion, our task demonstrates that llmgenerated synthetic data can be highly effective for oversampling in long-tail distributions. a second, albeit expected, finding is that ensemble strategies significantly enhance classification performance. additionally, while combined approaches for vector and category classification can be beneficial in prompting scenarios, they do not generally lead to performance improvements. more notably, we do not observe a clear winner among transformer architectures: fine-tuned encoder-only, encoderdecoder, and decoder-only models achieve comparable maximum performance across both subtasks. future research on our dataset should prioritize the more challenging vector classification task. our analysis indicates that classification errors often stem from low class support and that food recall texts contain ambiguous instances, with semantically similar classes contributing to misclassification. we argue that debugging classifiers using explainability techniques may help improve performance. despite its potential to assist human validation and enable meta-learning approaches, such as clustering or pre-sorting examples, explainability in text-based food risk classification remains underexplored. however, explanations can vary significantly depending on the model and task. existing literature addresses both model-specific @xcite and modelagnostic @xcite explainability approaches, which should be further investigated in this domain.