S em E val-2025 Task 5: LLM s4 S ubjects - LLM -based Automated Subject Tagging for a National Technical Library’s Open-Access Catalog (2025)

Abstract: We present SemEval-2025 Task 5: LLMs4Subjects, a shared task on automated subject tagging for scientific and technical records in English and German using the GND taxonomy. Participants developed LLM-based systems to recommend top-k subjects, evaluated through quantitative metrics (precision, recall, F1-score) and qualitative assessments by subject specialists. Results highlight the effectiveness of LLM ensembles, synthetic data generation, and multilingual processing, offering insights into LLMs for digital library classification. The task attracted over 700 participants. We received final submissions from more than 200 teams and 93 system description papers. We report baseline results, as well as findings on the best-performing systems, the most common approaches, and the most effective methods across various tracks and languages. The datasets for this task are publicly available. The dataset is available at {href{https://github.com/emotion-analysis-project/SemEval2025-task11}{SemEval2024-task 11}}.

## introduction subject classification within library systems involves organizing books and resources based on their content and subject matter to facilitate easy retrieval and access. automated methods for classifying scientific texts include springer nature's cso classifier @xcite , which uses syntactic and semantic analysis to categorize papers based on the computer science ontology (cso) using abstracts, titles, and keywords. other approaches have utilized citation metadata for classification @xcite , while research like new mexico state university's application of topic modeling on digital news releases @xcite demonstrates the diversity of techniques. methods range from embeddings @xcite and deep learning @xcite to citation analysis @xcite and topic modeling @xcite . thus, while the use of nlp in digital library subject classification is well-established @xcite , the potential for leveraging large language models (llms) for their extensive knowledge representation remains largely unexplored. several open-source toolkits integrate machine learning (ml) and nlp for automated subject in-dexing, notably annif ( https://annif.org/ ), developed by the national library of finland, as a diy automated subject indexing tool supporting the training of multiple traditional machine learning algorithms @xcite . annif allows users to train models on a chosen subject taxonomy and metadata to generate subject headings for new documents. it has performed well on scientific papers and books but struggles with older or diverse materials like q&a pairs or finnish wikipedia. european national libraries, including sweden's national library, the leibniz information centre for economics, and the german national library, have adopted annif. supporting multiple languages and vocabularies, it offers command-line, web, and rest api interfaces, demonstrating the adaptability required for effective subject classification. with these insights, as a semeval 2025 shared task, we organized task 5 -llms4subjects -to explore the untapped potential of llms for subject classification and tagging. the task was defined on the catalog of the tib -leibniz information centre for science and technology, germany's national library for science and technology. its catalog, tibkat, holds 5.7 million records (as of march 2025), including bibliographic data and metadata from freely available electronic collections. a subset of around 100,000 records is available as open access, and the task focused on this subset. the collection includes various record types such as technical reports, publications, and books, primarily in english and german. regardless of full-text language, records are consistently annotated using subject terms from the gemeinsame normdatei (gnd), the integrated authority file and subject taxonomy used in the german library system. llms offer promising opportunities for subject classification through their ability to process natural language at scale and capture the nuances of complex, interdisciplinary topics. this can significantly improve the accuracy and efficiency of organizing large collections, enhancing the accessibility and discoverability of information. the solutions developed in this task serve as a benchmark for applying llms in digital library systems, fostering innovation and setting new standards in the field. moreover, the task aligns with the goals of the semeval series by evaluating a novel application of computational semantics essential for effective information organization and retrieval. while the shared task focused on practical systems development, it was also driven by the following four research questions. rq1 multilingual vs. monolingual models in subject tagging: how do multilingual pre-trained models compare to monolingual models in bilingual subject tagging tasks?, rq2 effect of training data size and diversity: how does the size and diversity of training data affect llm performance in subject tagging?, rq3 role of augmented generation: how impactful are rag approaches versus finetuning?, and rq4 efficiency of language models: how effective are small versus large lms in performing the task? a key observation from the systems submitted to this shared task is that the advantages of llms over traditional machine learning algorithms for subject indexing remain debatable @xcite . while the first iteration of this shared task brings this question to light, to further explore the possibilities offered by llms, we will reorganize the llms4subjects shared task a second time and this time with a theme to build solutions based on energy-and compute-efficient llms. the gnd taxonomy. the gemeinsame normdatei foot_0 (gnd, german for "integrated authority file") is an international authority file used primarily by german-speaking library systems to catalog and link information on topics, organizations, people, and works. it is publicly available for download foot_1 in various formats under a cc0 license. the gnd's records specifically cover entities such as persons, corporate bodies, conferences, geographic locations, subject headings, and works, relevant to cultural and scientific collections. for the llms4subjects shared task, only the gnd subject heading (sachbegriff) records are of interest. since accessing the gnd for the first time for new users can be overwhelming, for the convenience of our participants, we have created a how-to guide to download the latest gnd file. ## source dataset we queried the tibkat service to restrict its metadata to records containing abstracts and gnd subject indexing. the query is fully reproducible via this persistent search link. it returned 189,665 records at the time of dataset creation. the tib open-access catalog spans nine media types: book (136,434), thesis (31,859), conference (12,212), report (6,711), article (2,080), collection (188), audiovisualdocument (167), periodical (57), and chapter (11), detailed in appendix a. using the langdetect foot_2 python library, we identified 48 languages. the top five were german (108,637), english (76,735), french (1,741), indonesian (945), and spanish (311). 4 for the official shared task corpus, we retained only records in german and english and excluded the four least represented media types, resulting in a dataset of 123,589 records. the excluded data is available as supplementary data. the final shared task dataset is available at: https: //github.com/jd-coderepos/llms4subjects/ tree/main/shared-task-datasets. in libraries, content is typically described using controlled vocabularies. in germany, the gnd is used for cataloging literature. in addition to descriptive cataloging (e.g., author, title, year, publisher), subject cataloging is performed by subject librarians. based on the title, abstract, and full text, librarians assign appropriate gnd keywords to describe the content as precisely as possible. this collaborative work is carried out across various libraries and national library networks. with tib adding around 15,000 new titles each month, subject cataloging is a labor-intensive task. integrating ai-driven solutions-especially llms-can significantly boost efficiency, partially automate workflows, and improve usability, all while maintaining cataloging quality. such innovations are key to modernizing information management and supporting research at scale. ## shared task description and dataset the llms4subjects shared task challenged participants to develop llm-based systems for recommending relevant subjects from the gnd taxonomy to annotate tib technical records. given a record's title and abstract as input, systems were expected to generate a customizable top-k ranked list of relevant gnd subjects. since the dataset included records in both english and german, systems were required to support bilingual semantic processing. the task was defined over two dataset collections. ## task setup the shared task offered multiple communication channels: a dedicated website, foot_8 a google group for faqs, and direct email support (llms4subjects@gmail.com). the organizing team included two computer scientists and three tib subject specialists who supported participants throughout. the task timeline spanned four months, from october 2024 to january 2025. a declaration of interest for participation (dip) survey initially recorded 33 teams; of these, 14 teams submitted system outputs, and 12 also contributed system description papers to the semeval workshop. the next section summarizes their approaches. the participant systems are described with a focus on their key methodological contributions. teams are listed in alphabetical order of their names. an overview of the systems is provided in ## discussion to establish a reference point for the llms4subjects shared task, we developed a baseline system using openai's gpt-4o via the assistant api. foot_11 two assistants were configured-one for all-subjects and another for tib-core-each equipped with the respective gnd subject taxonomies stored in openai's vector stores. for each tibkat record, the assistants embedded the title and abstract and queried the vector store to retrieve 50 gnd subjects. both assistants followed an identical prompt that defined their role as a subject matter expert in a technical library using the gnd taxonomy. the prompt instructed them to select exactly 50 valid, semantically relevant subject tags based on the input title and abstract, and return them in a strict json format. it also enforced constraints such as avoiding duplicates and non-matching entries, and supported bilingual input in german and english. despite using a gpt llm and structured prompt, the assistant api showed reliability issues: some outputs broke schema constraints or included malformed gnd codes, requiring multiple runs. still in beta, its stability at scale is uncertain. this baseline ranked below all participant submissions on the leaderboard, underscoring that while the api enables quick prototyping, effective subject tagging demands more specialized, robust systems like those of the top teams. ## conclusion semeval-2025 task 5: llms4subjects presented the first evaluation of llms for gnd-based subject indexing in bilingual (german/english) technical library records, combining quantitative metrics with expert assessments. the task revealed four key takeaways: multilingual models and training data outperformed monolingual ones (rq1, rq2); synthetic data and retrieval-augmented pipelines improved performance, underscoring the value of data diversity and system design (rq3); and smaller, well-engineered systems often rivaled large instruction-tuned llms, highlighting tradeoffs between scale and specialization (rq4). all data, code, and evaluation resources are openly available at https://github.com/ jd-coderepos/llms4subjects . a second edition foot_12 of the task is planned, with an emphasis on energy-and compute-efficient llm systems.