Jim at S em E val-2025 Task 5: Multilingual BERT Ensemble (2025)

Abstract: The SemEval-2025 Task 5 calls for the utilization of LLM capabilities to apply controlled subject labels to record descriptions in the multilingual library collection of the German National Library of Science and Technology. The multilingual BERT ensemble system described herein produces subject labels for various record types, including articles, books, conference papers, reports, and theses. Results indicate that for English language article records, bidirectional encoder-only LLMs can achieve high recall in automated subject assignment.

## introduction semeval-2025 task 5 utilizes large language model (llm) capabilities to assign controlled subject labels to multilingual german and english record descriptions @xcite . bert models, such as modernbert, are a natural fit for subject tagging @xcite . when trained as classifiers, such as in this submission, they are less prone to hallucination-a common challenge in generative ai models. the modern-bert model provides improvements of increased context sequences of 8192 tokens, over prior limits of 512 tokens in the original bert @xcite . another advance in modernbert is the use of flash attention @xcite . the emergent capabilities of llms are not fully explained by existing theories @xcite . bert utilizes a transformer architecture, but does not stack together transformers as in gpt models @xcite . the research community has been able to empirically inspect why bert works so effectively @xcite . similar "mechanistic interpretability" is underway for llms, but is not nearly as mature as the understanding of bert models @xcite . the bert ensemble developed for this task consists of four models: two multilingual bert models, one german-only bert model, and one english-only bert model. all models were finetuned with data from the tib technical library's open-access catalog. see for the average recall measures in the quantitative leaderboard, the bert ensemble ranked 7th out of 11 teams in the "all subjects" task group. in the qualitative results, this system's highest ranking was 5th out of 13 teams. the bert models do not mimic reasoning and cannot correct labels in the way current state-of-the-art reasoning models can, which puts purely bert ensembles at a disadvantage. future work will investigate combining bert outputs with reasoning over the labels using advances in chain of thought (cot). the code for training, testing, and inference is available on github ( https://github. com/jimfhahn/semeval-2025-task5 ). returning to the call to research and develop a system that could be used in practice, the system is fully reusable from the hugging face platform ( https://huggingface.co/ ). noteworthy for its open source hosting, the hugging face platform enables hosting of models and datasets and has useful inference capabilities for machine learning projects. ## background the task to assign a subject to a work requires a target vocabulary. in this case, the gnd vocabulary is paired with the title and abstract data from the tibkat collection @xcite . the language of the title and abstract was both english and german. to train the bert models that encompassed the ensemble powering the core of the inference stack all provided data from tibkat are processed into jsonl format and were modeled using title and abstract text along with corresponding model name url german bert https://huggingface.co/jimfhahn/bert-german-cased multilingual cased https://huggingface.co/jimfhahn/bert-multilingual-cased multilingual uncased https://huggingface.co/jimfhahn/bert-multilingual-uncased modernbert base https://huggingface.co/jimfhahn/modernbert-base-gnd while llms excel at a wide range of generative ai tasks, the specific task at hand is generating subjects, which falls under classification. therefore, bert models are well-suited to act as the core inference engine powering an llm-based subject indexing system. ## system overview the hugging face software package "autotrain advanced" was configured for training the component bert models @xcite . the input training data, sourced from the "all subjects" folder provided by the competition organizers, was incorporated into the dataset. additionally, the supplementary dataset, "dnb skos exports of the gnd," was subsequently incorporated to enrich the input data. a roughly 25/75 split was applied, allocating 78,800 rows to testing and 245,000 rows to training. this decision reflects the gnd technical staff's acknowledged expertise in curating high-quality resources. a departure point for prior work in semi-automated subject indexing is to reference existing professional skills while extending professional expertise @xcite . the combined dataset is multilingual mixing in both german language training with english language text. for the training, the software was installed in a compute environment at the university of illinois campus compute cluster where gpu hardware, nvidia a100 tensor core gpus are available to be scheduled. training time for the largest bert models, including modernbert, was completed within ten hours. the system employs an ensemble of bert models to generate classification results. refer to the inference folder of the github repository for the methods described herein ( https://github. com/jimfhahn/semeval-2025-task5/blob/ main/inference/inference.py ). during inference, the classify_text_batch function processes input texts in batches. each input is tokenized with truncation applied, followed by generating probability scores for all possible labels using the torch.softmax function. the system then identifies the top n labels (default is 50) and their associated confidence scores using torch.topk. while the models are trained for single-label classification, this approach enables the generation of multiple subject labels for each input. to aggregate classification results from individual models in the ensemble, the filter_and_aggregate function combines confidence scores for each label across models, summing them to produce a single combined score. the system then retains the top 50 labels based on the highest accumulated scores. the get_top_50_subjects function finalizes the process by extracting and validating these top 50 labels for each input. by leveraging the probabilistic confidence scores, this pipeline adapts single-label models to a multi-label context, effectively simulating a multi-label classification system through confidence score aggregation. ## experimental setup the training of bert models began with loading and processing the dataset from hugging face. refer to the train folder of the github repository for the methods described herein ( https:// github.com/jimfhahn/semeval-2025-task5/ blob/main/train/train.py). the processing code filtered out underrepresented labels, ensuring that each label had at least two examples. subsequently, the code split the dataset into training and validation sets for bert, ensuring that all classes were represented in both sets. the autotrain advanced software package included a default configuration to stop training if there was no improvement after 5 epochs, to prevent overfitting. the threshold for measuring the new optimum to continue training was set to 0.01 by default. it ensured that the training continued as long as the model's performance improved by at least 1% in each iteration. figure 2 shows the settings of autotrain advanced, as textclassificationparams that were utilized to train each of the models on the curated dataset. the autotrain advanced software used optuna for automated hyperparameter optimization @xcite . in practice, two iterations of fixed learning rate settings (@xmath0, with the latter yielding superior f1 scores during inference. similarly, the trial and error included two batch size iterations in training. the initial tests used a batch size of 8, while the final, better-scoring model training parameters were trained using a batch size of 16. the bert models were all trained as single-label classifiers where each input was assigned exactly one label. ## results recall@k was used as the central measure. specifically, the average of recall@k scores was used for the final leaderboard ranking of "average recall." according to the quantitative leaderboard, the multilingual bert ensemble ranked 7th out of 11 systems in the "all subjects" category. see the system's performance on english language articles is noteworthy, as it was a standout in subject recall; the details are considered in section 5.1. regarding the qualitative results, the system ranked 5th out of 13 systems in both case 1 in case 2. detailed qualitative results are discussed in more detail in section 5.3. ## conclusion the system showed good performance in recall for english language articles. evidence as to why these records are amenable to subject classification were considered. specifically, an analysis of the subject distributions by record type and the readability or reading complexity of the record metadata was conducted. the system is completely portable from the hugging face platform. the ensemble models are all easily extensible into library systems, allowing experimentation to be taken into production without requiring extensive coding for adapting to local environments.