OPI - DRO - HEL at S em E val-2025 Task 9: Integrating Transformer-Based Classification with LLM -Assisted Few-Shot Learning for Food Hazard Detection (2025)

Abstract: In this paper, we propose a hybrid approach for food hazard detection that combines a fine-tuned RoBERTa classifier with few-shot learning using an LLM model (GPT-3.5-turbo). We address challenges related to unstructured text and class imbalance by applying class weighting and keyword extraction (KeyBERT, YAKE, and Sentence-BERT). When RoBERTa’s confidence falls below a given threshold, a structured prompt which comprising the title, extracted keywords, and a few representative examples is used to re-evaluate the prediction with ChatGPT.

## introduction the semeval-2025 food hazard detection task @xcite focuses on automatically identifying mentions of food hazards, such as allergens or chemical contaminants, in textual data. this challenge is critical for public health, as rapid and accurate detection can help prevent outbreaks and protect consumers. our system employs a two-stage classification pipeline. first, we use a roberta-based classifier @xcite , fine-tuned on the training data, to predict whether a document includes a food hazard. if the model's confidence is below a given threshold, we move on to a secondary check with an llm model. in this step, we provide chatgpt with the document's title, a list of keywords, and a few examples of how to identify hazards -a few-shot learning setup @xcite @xcite . this additional step helps detect subtle or rare hazard mentions that roberta might overlook. despite these efforts, our system ranked 17th out of 27 teams. further analysis showed that our approach struggled with complex language describing hazards. in addition, the automatic ex-traction of keywords and the variability in chat-gpt's responses sometimes led to different outcomes. these results suggest that improved keyword selection, more precise threshold tuning, and better domain guidance for chatgpt could lead to higher performance. our complete codebase, including preprocessing and training scripts, is publicly available at: https://gitlab.com/ mspiewak/food-hazard-detection . ## background the semeval-2025 food hazard detection task provides a structured dataset for identifying food hazard information in text documents. each record in the dataset contains metadata (e.g., year, month, day, country), a title, and a full-length text field that describes a food recall event. the labels include both product and hazard each of which belongs to a higher-level category: product-category (22 possible categories) and hazard-category (10 possible categories). ## system overview the semeval-task combines two sub-tasks: (st1) text classification for food hazard prediction, predicting the type of hazard and product, and (st2) food hazard and product "vector" detection, predicting the exact hazard and product. this task prioritizes accurate hazard detection and employs a two-step evaluation metric based on the macro f1 score, emphasizing hazard classification in both sub-tasks. in this study, we focused exclusively on st1, developing a classification system tailored to the prediction of food hazards and associated product categories. ## experimental setup to ensure that our findings can be reproduced, we detail our experimental design below. the dataset is divided into 5,082 training records for model development and hyperparameter tuning, 565 validation records for model selection, and 997 test records for independent evaluation. our text preprocessing pipeline involves removing html tags, normalizing whitespace, converting all text to lowercase, and eliminating numeric elements and urls, with an optional step for stopword removal as needed. ## results we submitted the configuration using roberta with keybert. however, the gap between validation and test results for hazard-category classification suggests possible overfitting, while we do not observe this for product-category classification. notably, for product-category classification on the test set, methods using the llm component clearly improved performance, highlighting the benefit of few-shot learning. our experiments show that adding few-shot learning with targeted keyword extraction improves the basic roberta performance. using class balancing, especially for product categories, effectively addresses data imbalance. additionally, key-bert and sentence-bert work better than yake, providing more reliable support for the few-shot component. the main source of errors is that underrepresented classes tend to be misclassified with high confidence. in many instances, rare hazard and product categories are incorrectly predicted with strong probabilities, indicating that the model is overly biased toward majority classes. additionally, the few-shot learning component using chat-gpt struggles with these cases because the examples provided in the prompts are insufficient or weak, leading to more errors for rare classes. these observations suggest that both the primary model and the llm require improved strategies, such as improved prompt design and better handling of low-frequency classes, to effectively address these issues. ## conclusion we have developed a two-stage classification system for food hazard detection that combines the strengths of both roberta and chatgpt to tackle challenges posed by imbalanced data. our approach employs class weighting to reduce bias toward majority classes and leverages few-shot learning to improve predictions when the model's confidence is low. although our system ranked 17th out of 27 teams, indicating room for improvement, error analysis indicates that underrepresented classes are still frequently misclassified with high confidence. in future work, we plan to improve the prompt design and use active learning strategies to improve model adaptability. additionally, we will explore advanced data augmentation techniques to better capture the characteristics of rare hazard categories. these improvements should make the model more robust and generalizable, improving its effectiveness in real-world food safety applications.