Last Minute at S em E val-2025 Task 5: RAG System for Subject Tagging (2025)

Abstract: Last Minute at SemEval-2025 Task 5: RAG System for Subject TaggingZahra Sarlak, Ebrahim AnsariIn this study, we explore the LLMs4Subjects shared task, which focuses on leveraging retrieval-augmented generation (RAG) to enhance subject classification in technical records from the Leibniz University’s Technical Library (TIBKAT). The challenge requires participants to recommend appropriate subject headings from the GND taxonomy while processing bibliographic data in both German and English.

## introduction subject tagging is essential for organizing and retrieving information in vast collections of technical records. the leibniz information centre for science and technology (tib) manages tibkat, an open-access bibliographic database that encompasses a significant variety of scientific and technical metadata. to enhance user accessibility, tib aims to provide precise subject tagging based on the gnd (gemeinsame normdatei) taxonomy, which is widely used in german-speaking libraries. the task of manually tagging records is laborintensive due to the vast number of different subject tags available, leading to potential inconsistencies and inefficiencies. the llms4subjects shared task @xcite invites researchers to design models capable of processing bilingual technical documents-specifically, those written in german and english. by accurately tagging these documents, systems can significantly enhance the discoverability of information and improve research workflows. our study leverages the recent advances in natural language processing (nlp) made possible by retrieval-augmented generation (rag). this method integrates retrieval mechanisms with language generation, allowing for a more contextaware approach to tagging. despite much of the focus in nlp being on english and monolingual tasks, the demand for effective bilingual models remains high, especially in library settings. in our approach, we utilize rag to dynamically retrieve relevant subject headings from the gnd taxonomy based on the technical record's title and abstract foot_0 . we also employ contrastive learning to fine-tune embedding model for subject tags and incorporate a re-ranking mechanism to optimize our recommendations. by utilizing milvus @xcite for efficient vector storage, we aim to enhance both the accuracy and speed of the subject tagging process. ## related work the integration of automated tagging and structured vocabularies in digital libraries is becoming increasingly important for improving the discoverability of academic content. recent research has presented various strategies to enhance metadata management, which closely aligns with our use of retrieval-augmented generation (rag) for tagging in the tibkat database. a key contribution in this area is by @xcite , who focus on using a hierarchical learning taxonomy to automatically tag academic questions. their method addresses the challenges of understanding the relationships between terms in the taxonomy and the questions being tagged. simi-larly, @xcite propose a data-driven approach for analyzing biodiversity subject metadata, using named entity recognition and word embeddings to group related terms effectively. @xcite also highlight the importance of using artificial intelligence tools to improve metadata creation, particularly to fix inconsistencies and outdated descriptions in digital archives. these advances reflect a trend in natural language processing that emphasizes the need to combine retrieval methods with language models to enhance their performance on tasks that require knowledge. a key study in this area is by @xcite , who introduced rag. this approach shows that combining a pre-trained sequence-tosequence language model with a dense vector index allows for better access to external knowledge, leading to improved performance, especially in opendomain question answering. @xcite provided a comprehensive survey of rag methodologies, identifying the advantages of incorporating external databases into language generation. their work emphasizes that rag can reduce issues such as hallucination and outdated knowledge by continuously updating and integrating information from external sources. this approach not only enhances the accuracy of generated outputs but also facilitates a more transparent reasoning process, making it a robust framework for knowledge-intensive applications. in the framework presented by @xcite , the demon-strate-search-predict (dsp) model outlines a sophisticated method for combining retrieval and language models. by creating a pipeline for passing information between these models, the dsp framework seeks to leverage the strengths of both retrieval and generation techniques, achieving new state-of-the-art results in various knowledge-intensive settings. this work exemplifies the potential for improving the interaction between retrieval and language generation, leading to more reliable and coherent output. @xcite explored the comparison between retrieval-augmented generation and traditional unsupervised fine-tuning for knowledge injection in large language models. their findings suggest that while fine-tuning can provide benefits, rag consistently outperforms it, particularly in integrating new knowledge and enhancing the models' overall capabilities. this study underscores the limitations of conventional methods and highlights the growing importance of retrieval-based approaches in effectively updating language model knowledge. lastly, @xcite examined in-context retrieval-augmented language models (ralms), presenting a simple yet effective method for conditioning language models on relevant documents without altering their architecture. this approach focuses on maintaining the existing model while improving performance through external document incorporation, which can simplify deployment and usability. the findings suggest that leveraging retrieval mechanisms can lead to significant advantages in language modeling without necessitating extensive changes to the underlying model architecture. these studies shows the impact of retrievalaugmented methods in enhancing the capabilities of language models, particularly for tasks that require accurate knowledge retrieval and integration. ## methodology our approach to the llms4subjects shared task involved multiple key steps designed to enhance the accuracy and efficiency of subject tagging for technical records from the tibkat database. below, we outline the methodology employed: the embeddings for each subject tag were generated by concatenating name and classification name along with definition and related subjects when available. this comprehensive representation effectively captures the contextual meaning of each tag. given the following abstract of a technical document: finally, we extracted the top 50 tags recommended by the llm as the final output for each title and abstract. this multi-step methodology, combining embedding, tagging retrieval, re-ranking, and llm refinement, aimed to enhance the overall accuracy and effectiveness of subject classification for the technical records in the tibkat database. ## result the initial results of our tagging approach revealed several challenges in the subject tagging process, despite the potential of retrieval-augmented generation (rag). these challenges underscore the need for further refinement and optimization. however, they also provide valuable insights of subject classification in a multilingual and domain-specific context. in this section, we present the detailed evaluation results. the evaluation of our subject tagging system was carried out by the semeval team through both quantitative and qualitative assessments. the combined language and record-levels results is available at appendix a, with the separated evaluation result for different record types and languages. tables 4-6 each represented result at @xmath0,10,15 respectively. having better recall for more @k is natural, because the more subject retrieved increase the chance of more correct tag selection. ## challenges one of the main reasons for the low scores is that, despite fine-tuning the embedding model, the distance between related and unrelated subject embeddings in the embedding space is still not adequate. since extracting relevant subject tags heavily relies on cosine similarity, the distance between subject embeddings is crucial; however, achieving this distinction through fine-tuning the embedding model proves to be challenging. it is important to note that a subject may be related to one record while being unrelated to another, yet all of them may have near vectors in the vector space. fine-tuning the model may improve some similarities, but it can also distort others, further complicating the differentiation process. therefore, fine-tuning must be implemented with great care and involve iterative reviews of the results. additionally, the vast, highly imbalanced, and diverse set of subject tags, along with the need to embed and index this extensive set of tags in vector storage, makes evaluation challenging and limits our options for testing different embedding models. ## conclusion we examined the effectiveness of retrieval-augmented generation (rag) for tagging subjects in technical records from the tibkat database. by fine-tuning a lightweight language model and integrating a retrieval mechanism, we aimed to improve the accuracy and efficiency of the tagging process. the challenge of accurately tagging subjects is heightened by the vast number of subject tags-over 240,000-making it difficult to achieve effective results with simple similarity searches. for future work, we propose adopting a structured approach, such as using graph-based representations, to store subject tags and utilize their hierarchical taxonomy for more precise candidate selection. additionally, since our solution relies heavily on similarity search, evaluating its effectiveness poses challenges due to the large search space. to address this, we can implement heuristics that limit the search set for each abstract sample, ultimately improving both efficiency and accuracy in the tagging process.