CICL at S em E val-2025 Task 9: A Pilot Study on Different Machine Learning Models for Food Hazard Detection Challenge (2025)

Abstract: This paper describes our approaches to SemEval-2025 task 9, a multiclass classification task to detect food hazards and affected products, given food incident reports from web resources. The training data consists of the date of the incidents and the text of the incident reports, as well as the labels: “hazard-category” and “product-category” for task 1, “hazard” and “product” for task 2. We primarily focused on solving task 1 of this challenge. Our approach is in two directions: Firstly, we fine-tuned BERT-based models (BERT and ModernBERT); secondly, in addition to BERT-based models, linearSVC, random forest classifier, and LightGBM were also used to tackle the challenge. From the experiment, we have learned that BERT-based models outperformed the other models mentioned above, and applying focal loss to BERT-based models optimized their performance on imbalanced classification tasks.

## introduction food safety is one of the main concerns of consumers when making purchasing decisions. therefore, a system that detects possible hazardcontaining products and their corresponding hazards from past reports can help consumers identify certain possible hazards in food products more easily. semeval-2025 task 9 @xcite is a shared task focusing on food hazard detection, the participants are encouraged to design classification systems that detect hazards and the affected products from food safety incident reports (all texts were originally in or translated to english). 1 the challenge has two subtasks: our group focuses primarily on subtask 1 of the challenge. to solve the task effectively, we propose our two-direction approach: foot_1 approach with pre-trained language models: fine-tuning the pre-trained language models, namely bert @xcite and modernbert @xcite to adapt to pre-processed data. to minimize training cost, lightweight and free computational cost fine-tuning enhancements were used. approach without pre-trained language models: training and tuning the hyperparameters of traditional machine learning models, namely linearsvc, random forest, and more recent desicion-making model, lightgbm @xcite ; which are less time-consuming than fine-tuning bert-based models. this method serves as a comparison to the first approach. in addition to finding the most effective model, the way to perform data augmentation is also a challenging problem. first of all, the training data is heavily imbalanced. for example: class food additives and flavourings only has 24 entries while class allergens has 363 entries in label hazardcategory. therefore, for our approach with bertbased models, we applied back-translation via mar-ianmt @xcite framework to generate more training samples for long-tail data, improving generalization on minority classes. besides, we replaced the standard cross-entropy loss with focal loss @xcite , which dynamically reduced the influence of majority-class samples, allowing the model to learn better from underrepresented categories. in addition to the imbalance of classes, the data are also non-specific and lack context. therefore, we introduced keyword masking with contextual prompting, where key terms were masked and replaced with the [mask] token; as well as category-specific prompts to provide additional context, guiding the model's attention. in comparison to our system with bert-based models, we applied simpler methods to the system without bert-based models such as oversampling (random oversampling) to minimize the influence of the imbalanced classes in our training data, to clean up the data noise, we also applied a function to remove all punctuations and unnecessary whitespaces. through the approaches with bert-based models, we discovered that the integration of focal loss with bert effectively addresses class imbalance, which is consistent with the findings of previous research @xcite on the handling of class imbalance in dataset mention detection. while keyword masking and contextual prompting showed the potential to improve the results. in contrast, neither back-translation for data augmentation nor our pre-processing efforts: including noise removal and utilizing spacy @xcite for named entity recognition yield the expected improvements. for our approach without bert-based models, our attempts with oversampling methods did not achieve significant improvement. ## experimental setup data split during our training process, we split the training set into 80% training vs. 20% testing in cross-validation for all of our systems. data preprocessing for training bert-based models, we defined labels with fewer than 10 samples as minority classes, resulting in 34 underrepresented entries in total. back translation was applied to the underrepresented entries and 2 new entries were generated from each category and added to the original training data. for training the other models, we applied a function to eliminate punctuation and multiple whitespaces and all training data was weighted by tf-idf. training strategies firstly, we used the bert baseline provided by the task organizer as our baseline. then the three strategies: back translation, focal loss, as well as keyword masking and contextual prompting were applied separately to the bert model. moreover, we also tested the performance of bert with all three strategies. however, due to resource constraints, we were only able to create a baseline and an experiment with full strategies with modernbert. in addition, we created baselines with randomforestclassifier, linearsvc and lightbgm, oversampling method (random sampling) was applied to them. 7 all models were trained only with the feature title, which consists of the titles of food safety incident reports. ## results in this section, we present the performances of our systems with different settings in macro f1 score during our validation process. unfortunately, we were only able to upload our results from bert with focal loss (0.6006) and linearsvc (0.6079) to the organization leaderboard (rank #23). ## conclusion our results suggest that the bert-based models have better performance than other models, and we discovered that applying focal loss optimized the performance of bert-based models on imbalanced classification task. however, the combination of bert + focal loss has a lower score than linearsvc in the final evaluation. a possible reason is that our bert-based models lack generalizing ability while the test set may have a different class distribution and/or degree of noises than the training data. besides, according to our result, back translation and keyword masking/prompting also showed some benefits but rather limited. looking ahead, we see several promising directions for further research. one key improvement for model robustness could be the generation of higher-quality augmented data, ensuring that synthetic samples closely resemble real-world instances; in addition, if sufficient resources are provided, the ensemble method could be used to optimize the performance of multiple bert-based models. another potential avenue is the transition from a single-task learning framework to multi-task learning, which could help the model generalize better across related tasks.