SRCB at S em E val-2025 Task 9: LLM Finetuning Approach based on External Attention Mechanism in The Food Hazard Detection (2025)

Abstract: This paper reports on the performance of SRCB’s system in SemEval-2025 Task 9: The Food Hazard Detection Challenge. We develop a system in the form of a pipeline consisting of two parts: 1. Candidate Recall Module, which selects the most probable correct labels from a large number of labels based on BERT model; 2. LLM Prediction Module, which is used to generate the final prediction based on Large Language Models(LLM). Additionally, to address the issue of long prompts caused by an excessive number of labels, we propose a model architecture to reduce resource consumption and improve performance. Our submission achieves the macro-F1 score of 80.39 on Sub-Task 1 and the macro-F1 score of 54.73 on Sub-Task 2. Our system is released at https://github.com/Doraxgui/Document_Attention

## introduction semeval is a series of international research workshops aimed at advancing the field of natural language processing(nlp), with a particular focus on semantic analysis techniques and the creation of high-quality annotated datasets to address various complex challenges in natural language semantics. each year, the workshop organizes a series of shared tasks, offering a platform for the presentation and comparative evaluation of computational semantic analysis systems developed by different teams. the food hazard detection task @xcite comprises two sub-tasks: sub-task 1 focuses on classifying the type of hazard and product, while sub-task 2 aims to classify the exact hazard and product. the challenges of this task include: 1) a heavily imbalanced class distribution, and 2) a large number of labels: 10 types of hazards and 22 types of products in sub-task 1, and 128 exact hazards and 1,142 exact products in sub-task 2. constructing prompts with such a large number of labels for llm-based response generation leads to high resource consumption and degraded performance. for challenge 1, our proposed solution employs large language models(llms). as llms demonstrate strong performance on data augmentation @xcite , particularly for imbalanced data. therefore, we use llm, specifically qwen2.5-72b-instruct @xcite , to perform data augmentation and deal with the imbalanced class distribution. for labels with limited training samples, we prompt the llm to generate new samples based on the content of these samples, the semantic meaning of the label, and the text format of a randomly selected training sample. for challenge 2, we propose a novel model architecture named the external attention mechanism, which is used to combine input embeddings with label embeddings to inject label information into the input, as opposed to the conventional approach of concatenating label information in text form and relying on the self-attention mechanism. this innovative structure treats input and labels as two separate parts, reducing the length of prompts and leading to significant resource savings and performance improvements. our system is a pipeline consisting of two parts: candidate recall module and llm prediction module. the candidate recall module is mainly composed of a bert model with a classifier added at the top, which is used to filter all labels and keep those candidate labels which have high probability of correctness. the purpose of this module is to reduce the number of all candidate labels and help subsequent modules reduce error options and improve performance. the llm prediction module first obtains the hidden states of the input and the hidden states of all labels, and then aligns them through the external attention mechanism (alternatively called label attention module) to generate a new representation. finally, the system uses this new representation to predict hazards and products in the form of next token prediction task. the purpose of this module is to allow the hidden states of the input to calculate the attention mechanism with labels externally, rather than splicing labels directly into the input in the form of text and using self-attention mechanism of llm. in this way, the length of the llm's input prompts can be significantly reduced. moreover, in the external attention mechanism, all labels are treated equally, and the potential impact of the labels' order will not be introduced, as the hidden states of labels are calculated in parallel. ## background the food hazard detection corpus, introduced in @xcite , includes the professional manually labeled titles and full texts on food recall collected from official food agency websites, and the language is english. figure 1 shows an example of training data, our analysis reveals that relying solely on the title or text results in incomplete information extraction. therefore, we combine both features to enhance the performance of the models. @xcite states that optimizing the top-layer classifier is ineffective for imbalanced class distribution. @xcite introduces that the text generation mode based on autoregressive llm can be used as a better alternative method. @xcite claims that llm can adapt to different tasks without a large number of training samples due to their ability of understanding natural language instructions. based on these studies, we focus mainly on llm instruction tuning in our system, finetuning llm to predict hazards and products. ## system description for each classification task (including the classification of 1.type of hazard, 2.type of product, 3.exact hazard, 4.exact product), we use a unified pipeline consisting of two modules: candidate recall module and llm prediction module. ## conclusion in this work, we propose a system consisting of the candidate recall module and the label prediction module. we identify that the primary cause of high training time consumption is the long prompt. to address this, we design the candidate recall module to reduce the length of prompt in terms of system structure. and we design the label prediction module to further minimize the impact of the prompt length using the external attention mechanism in terms of algorithm. additionally, the label prediction module also addresses the problem of the option sequence. with the help of other techniques like data augmentation and pipeline optimization, our system achieves the highest score for sub-task 2, and a competitive score for sub-task 1. for future work, we plan to explore advanced architectures to optimize the llm prediction module.