DNB - AI -Project at S em E val-2025 Task 5: An LLM -Ensemble Approach for Automated Subject Indexing (2025)

Abstract: This paper presents our system developed for the SemEval-2025 Task 5: LLMs4Subjects: LLM-based Automated Subject Tagging for a National Technical Library’s Open-Access Catalog.Our system relies on prompting a selection of LLMs with varying examples of intellectually annotated records and asking the LLMs to similarly suggest keywords for new records. This few-shot prompting technique is combined with a series of post-processing steps that map the generated keywords to the target vocabulary, aggregate the resulting subject terms to an ensemble vote and, finally, rank them as to their relevance to the record.Our system is fourth in the quantitative ranking in the all-subjects track, but achieves the best result in the qualitative ranking conducted by subject indexing experts.

## introduction the llms4subject task @xcite aims at utilising large language models (llms) for the task of automated subject indexing on a dataset of open-access publications. automated subject indexing is a task that helps enabling access to user-relevant publications by identifying and recording their most important themes and topics in the tagged subject terms. the ever-growing number especially of digital publications requires reliable automated systems for this task, which has become infeasible to achieve manually. in our previous work on automated subject indexing on a similar dataset @xcite , we found that the performance of llms, while succesfully applied to a range of other tasks @xcite @xcite , was not yet on par with classical supervised machine learning methods. therefore, it is important to do further research on the capabilities of llms in this context. rather than fine-tuning models ourselves, the main strategy of our system is to leverage the existing capabilities of off-the-shelf foundational or instruction-tuned open-weight llms. in contrast to our previous work, the key contribution of this system is that it does not rely on only one llm, but a combination of different language models along with varying prompts to generate the subject terms. we found this ensemble approach to dramatically improve the performance of our system. to handle the challenge of the controlled vocabulary unknown to the llms, we first generate free keywords with generative llms and then map these onto the vocabulary with a smaller embedding model. the official quantitative results put us in fourth place, the qualitative results even in first place. we think that our approach provides valuable insights into the chances and bounds of the few-shot prompting approach, showing that competitive results are possible without fine-tuning and large training corpora, simply by combining several llms into an ensemble. our code is publicly available. foot_0 ## background outlining the field of automated subject indexing, @xcite presented important fundamentals, approaches and best practices for the task. referring to it as index term assignment, @xcite compared and combined two strategies to perform this task: multi-label classification (mlc) and keyword extraction. detecting separate strengths, their results aligned with @xcite , who also found the combination of approaches to be beneficial. regarding frameworks for automated subject indexing, the annif system @xcite is an important contribution. annif has established methods built in, like omikuji foot_1 , which is based on partitioned-label-tree-method bonsai (khandagale et al., 2020), or mllm foot_2 , a lexical approach building on @xcite 's maui. in earlier work @xcite , we presented experiments with a closed-source llm on automated subject indexing, but the two baseline methods implemented in annif mentioned above were found to be as good as or even outperform our llm-based method. llms have also been utilised for mlc @xcite @xcite and keyword extraction @xcite . recently, building ensembles or fusioning (the results of) llms has been addressed as a promising research direction. there are different works sharing the idea of exploiting the individual strengths and diminishing the weaknesses in different llms @xcite @xcite @xcite . exploring the goal of building ensembles, tekin et al. (2024) aimed at maximising diversity and efficiency, whereas @xcite targeted the reduction of inference cost. not only llms have been combined, but also prompts @xcite . combining both prompts and models on the task of phishing detection, @xcite contrasted prompt-based ensembles (with one prompt and several llms), model-based ensembles and an ensemble consisting of a mixture of prompts and models. ## system overview our system is an enhancement from our previous llm-based subject indexing approach, described in @xcite . in total, it consists of 5 stages, complete, map, summarise, rank and combine, as depicted in the overview in figure 1 . at its core, the system approaches the subject indexing task as a keyword generation problem which is solved by a few-shot prompting llm procedure. as these generated keywords are a priori not restricted to the target vocabulary, a mapping stage with a smaller word embedding model is needed as a supplementary step. in comparison to our previous approach, we have extended the system by combining multiple llms and prompts to an ensemble and by introducing an llm-powered ranking step as in d'oosterlinck et al. (2024). ## conclusion to sum up, we have demonstrated that our ensemble appoach is a promising way to combine the strengths of different models and prompts, achieving competitive results in the llms4subjects task. as we have covered a wide range of prompts and llms, we expect our system to provide a good estimate of the results possible by prompting llms even without fine-tuning. while our system comes with no extra training costs, a significant drawback is the high cost involved in prompting multiple llms at inference time. appendix a.9 demonstrates the costs of processing the documents with each of the llms used in our ensemble. particularly larger models use up an enourmous amount of gpu-ressources that may be infeasible in productive settings. in future work, we would like to further investigate techniques for automated prompt optimisation, such as dspy @xcite , or methods belonging to the family of parameter-efficient-fine-tuning (peft). also we would like to investigate more sophisticated methods for the ensemble combination.