Annif at S em E val-2025 Task 5: Traditional XMTC augmented by LLM s (2025)

Abstract: This paper presents the Annif system in SemEval-2025 Task 5 (LLMs4Subjects), which focussed on subject indexing using large language models (LLMs). The task required creating subject predictions for bibliographic records from the bilingual TIBKAT database using the GND subject vocabulary. Our approach combines traditional natural language processing and machine learning techniques implemented in the Annif toolkit with innovative LLM-based methods for translation and synthetic data generation, and merging predictions from monolingual models. The system ranked first in the all-subjects category and second in the tib-core-subjects category in the quantitative evaluation, and fourth in qualitative evaluations. These findings demonstrate the potential of combining traditional XMTC algorithms with modern LLM techniques to improve the accuracy and efficiency of subject indexing in multilingual contexts.

## introduction subject indexing is an important aspect of improving the discoverability of bibliographic databases and digital collections. systems for automating subject indexing have traditionally been based on natural language processing (nlp) and traditional machine learning (ml) methods. the rise of generative ai and large language models (llms) holds some promise to revolutionise many automation tasks, yet producing accurate subject predictions using llms remains elusive (e.g. (eric h. c. chow and li, 2024), @xcite ). the llms4subjects challenge @xcite invited teams to produce innovative solutions for llm-based subject indexing using a data set @xcite based on the bilingual bibliographic database tibkat of tib, the leibniz information centre for science and technology. we have been developing the annif foot_0 multilingual open source automated subject indexing toolkit since 2017 @xcite . our toolkit is mainly based on traditional nlp and ml methods. by participating in this task, we aim to provide a strong baseline using traditional methods augmented with some llm-based techniques. the novel aspects of our work are 1) translating the subject vocabulary and metadata records using llms, 2) generating synthetic training data using llms, 3) including xtransformer in an ensemble of algorithms, and 4) generating suggestions using separate monolingual prediction pipelines and merging their results. our system ranked 1 st in the all-subjects category and 2 nd in the tib-core-subjects category in the quantitative evaluation. it was ranked 4 th in qualitative evaluations. we discovered that our approach, based mainly on traditional nlp and ml, remains competitive against other systems that make heavier use of llms. we also found ways to efficiently translate bibliographic records and to produce additional synthetic training data using llms. our code, configuration files and customised data sets are available on github foot_1 . the models we trained are available on hugging face hub foot_2 . ## background the task involved developing llm-based systems that recommend the most relevant subjects from the gnd subject vocabulary to tag a given tibkat record based on its title and abstract, which is a type of extreme multilabel text classification (xmtc) problem. the organisers provided two variants of the gnd subject vocabulary: all-subjects (all 200,035 gnd subjects) and tib-core-subjects (a subset of 78,741 subjects especially important for tib). we used the skos versions of gnd provided by the german national library (dnb). the organisers also provided bibliographic records from the tibkat database as two data sets, corresponding to the two gnd variants, each of them divided into train, development, and test subsets. the number of records was 81937 / 13666 / 27986 for the all-subjects data set and 41923 / 7001 / 6119 for the tib-core-subjects data set. all these subsets were further split by document type (e.g. article or book) and language (german or english). we did not distinguish records by these two aspects foot_3 . known gnd subjects were included only for the train and development records. although the task description suggested using llms, we did not use one for the core task of choosing subjects but instead relied on the traditional xmtc algorithms in annif. however, we used llms to pre-process the data sets and to generate additional synthetic training data. ## system overview we based our system on the annif automated subject indexing toolkit. it provides a selection of xmtc algorithms as configurable backends which can be used for subject indexing by setting up projects that define the vocabulary and specific configuration settings of a backend. we chose three annif backends for this task: 1) omikuji foot_4 , an implementation of a family of efficient machine learning algorithms for multilabel classification based on the idea of partitioned label trees, including parabel @xcite and bonsai @xcite . we used the bonsai-style configuration. 2) mllm foot_5 (maui-like lexical matching), a lexical algorithm for matching words and expressions in document text to terms in a subject vocabulary. it is a reimplementation of the ideas behind maui @xcite , an earlier tool for automated subject indexing that uses heuristic features and a small machine learning model to select the best performing heuristics. 3) xtransformer, an xmtc and ranking algorithm based on fine-tuned bert-style transformer models that is part of the pecos framework @xcite . its annif integration is experimental and was refined in the process of this task. we used facebookai/xlm-roberta-base as the base model. combinations of xmtc algorithms, called ensembles, often outperform individual algorithms. we combined the base backends that return lists of suggested subjects along with numeric scores into two kinds of ensembles: simple ensembles that merge subjects suggestions from two or more backends by averaging their scores, and neural ensembles that, in addition to averaging scores, also involve training a neural network model that adjusts the subject scores, for example suppressing subjects that are frequently wrongly suggested (false positives). ## experimental setup we first set up, trained and evaluated the three kinds of base projects. we aimed to maximise their evaluation scores, measured against the development set, by exploring various approaches and settings. once satisfied with the performance of the base projects, we combined them into ensembles that were finally used to produce our system output. for evaluation during system development, we used two common xmtc metrics built in to the annif toolkit: f1@5 (f1 score calculated using the top 5 suggestions from the system) and ndcg@10 (normalised discounted cumulative gain @xcite , a ranking metric calculated using the top 10 suggestions from the system). ## results the final quantitative and qualitative evaluations were performed by the task organisers. ## conclusions in conclusion, our participation in the semeval-2025 task 5 (llms4subjects) provided valuable insights into the capabilities and performance of annif, particularly when augmented with large language models (llms) for data preparation. by leveraging traditional xmtc algorithms such as omikuji bonsai, mllm, and xtransformer, and enhancing them with llm-generated synthetic data and translations, we demonstrated competitive results across multiple categories. while this task focused only on the resulting quality of the subject indexing, we note that the computational requirements, energy consumption and processing latency of traditional ml approaches are modest in comparison to llms.