NBF at S em E val-2025 Task 5: Light-Burst Attention Enhanced System for Multilingual Subject Recommendation (2025)

Abstract: This paper presents a system for automated subject tagging in a bilingual academic setting. Our approach leverages a novel burst attention mechanism to enhance the alignment between article and subject embeddings, derived from a large cross-lingual subject corpus. By employing a margin-based loss with negative sampling, our resource-efficient model achieves competitive performance in both quantitative and qualitative evaluations. Experimental results demonstrate average recall rates of 32.24% on the full test set, along with robust performance on specialized subsets, making our system well-suited for large-scale subject recommendation tasks.

## introduction automated subject classification of scholarly articles is of growing importance for digital repositories, allowing more efficient data retrieval, recommendation, and systematic reviews @xcite . at semeval-2025 task 5 @xcite , participants must predict subject codes for articles in two languages (english and german). previous semeval tasks on concept or subject classification have used both classical machine learning and deep approaches @xcite . large pre-trained models (e.g., gpt-based @xcite or other generative approaches) could be powerful but are often computationally heavy. we instead adopt a dimension-as-token approach, refining base embeddings from sentence transformers @xcite via an ultralight attention transform. by participating in semeval-2025 task 5, we discovered that our parameter-efficient, dimensionas-token self-attention approach can effectively handle both english and german data, achieving moderate recall rates of 32. 24% in the overall quantitative setting (all subjects), 43. 16% in case 1 and 31. 53% in case 2 of overall qualitative evaluation methods with minimal gpu usage. despite challenges such as near-synonymous subject labels and sparse domain terms, our system still secured the 9th position in both quantitative (all subjects) and qualitative evaluations. these findings highlight the potential of cross-lingual pipelines and public code for fragmented training and inference, even under resource-constrained conditions. ## task description the llms4subjects shared task @xcite challenges participants to develop llmbased systems for automated subject tagging in a national technical library. in this task, systems must assign one or more subject codes to each article, drawing from an extensive gnd (german national library, 2025) subjects taxonomy. participants are provided with a human-readable version of the taxonomy along with a bilingual data set (english / german) from tibkat (tib -leibniz information centre for science and technology, 2025) that includes various record types such as article, book, conference, report, and thesis. ## proposed system our method has two well-defined stages. first, we create fixed embeddings for every data. second, we train a small neural module that makes the embeddings of the articles match the embeddings of the gnd subjects. all subject labels (originally in german) are translated into english. we then use pre-trained sentence-transformer models to encode both subjects and articles. for articles, we join the title and abstract before encoding. each subject and each article are now represented by a single 768dimensional vector. these embeddings serve as input to our high-capacity transformation model, which is designed to enhance the semantic alignment between article embeddings and subject embeddings. together, these components transform initial embeddings into a space where articles are more closely aligned with their relevant subjects, thus improving the effectiveness of subject retrieval. training procedure. during training only article vectors go through the alignment module(transformation module). for each article, we also retrieve its gold subject vectors and average them. the loss function moves the aligned article vector closer to this average and further away from a few randomly chosen, incorrect subject vectors. the subject vectors themselves remain fixed. inference procedure. at test time the alignment module is frozen. a new article is embedded, passed through the module, and compared using cosine similarity to the stored subject vectors. the k subjects with the smallest distance are returned as recommendations. design motivation. leaving the large pretrained transformer models untouched avoids costly finetuning and keeps subject and article vectors consistent. the tiny alignment module is enough to correct systematic differences between the two sets of vectors while adding very little computation. figure 1 illustrates the complete system architecture. it begins by extracting embeddings for both the subject corpus and article texts, followed by a training module that applies the transformation block and computes a margin-based loss. after training, the resulting model is saved for inference, where articles are matched against subject embeddings to retrieve the most relevant subjects. ## training and inference during training, we learn only the small transformation block, keeping the sentence-transformer weights and the subject matrix s fixed. for each article j, with its raw embedding m j we apply the learned transform to obtain the anchor embedding a @xmath0. let the averaged gold-subject embedding be p j , and let n j1 , . . . , n jk denote k negative subject embeddings sampled from s. using cosine distance d(x, y) = 1 -x•y ∥x∥ ∥y∥ , we define the marginbased loss for article j as we train for 20 epochs with batch size 4, using the adamw optimizer and a cosine annealing scheduler. in each batch, only the embeddings of the article, their positives and the sampled negatives are passed through transform(•); the embeddings of the subject remain frozen. inference at inference time, we first transform and cache all gnd subject embeddings as s ′ = transform(s), so that each row s ′ is the aligned vector for one subject. for a new article (title and abstract), we compute its embedding @xmath1, then apply the trained mapping to obtain @xmath2. next, we measure the cosinedistance between a and each precomputed subject vector finally, we sort all subjects by increasing distance and return the k codes with the smallest values. ## experimental setup our experiments were carried out on a bilingual data set partitioned into training, development and test sets for both english and german articles. the training set was used for margin-based optimization, the development set for intermediate performance monitoring, and the test set was reserved for official semeval evaluation. the embedding dimension is set to 768, and our model is configured with a hidden dimension (model_dim) of 16, 2 attention heads, a single burst attention layer, an mlp hidden dimension of 256, and a dropout rate of 0.03. the training employs a batch size of 4 and utilizes 15 negative samples per anchor-positive pair over 20 epochs. we apply a margin-based loss with a margin of 0.2, optimizing the model using the adamw optimizer with a learning rate of 1 × 10 -4 (without weight decay) and a cosine annealing learning rate scheduler (t @xmath3, η @xmath4. all experiments were executed on google colab l4 gpus under python 3.9, using pytorch 1.13.1, sentencetransformers 2.2.2, pandas 1.5.3, numpy 1.23.5, and tqdm 4.64.1. evaluation measures included precision@k, recall@k, and f1@k across multiple cutoffs, along with average recall, as reported in section 6. ## results we report the performance of our system on the official test segment as well as on two qualitative evaluation subsets. for evaluation, we use precision @ k (p @ k), recall @ k (r @ k), and f1 @ k. precision @ k is defined as the number of relevant items in the top k predictions divided by total number of items in k, recall @ k is defined as the number of relevant items in the top k predictions divided by the total number of relevant items. the fβ-score is computed as quantitative results evaluation 1 of the shared task compared systems in a fully automatic setting in the blind tibkat test corpus. participants submitted the top-50 gnd subject codes for every english or german record, and the organisers calculated p @ k, r @ k, and f1 @ k for each @xmath5, 10, . . . , 50}. scores were released at three granularities: (i) language level (en vs. de), (ii) record-type level (five technical record types), and (iii) the combined language-record level. systems were ranked by the average recall @ k over all cut-offs, a criterion in which our nbf run placed 9th. qualitative results evaluation 2 complemented the automatic leaderboard with a manual assessment by subject specialists from tib. a stratified sample of 140 records-ten from each of fourteen discipline codes (arc, che, elt, fer, his, inf, lin, lit, mat, oek, phy, sow, tec, ver)-was drawn. for every record, librarians inspected the top-20 subjects proposed by each system and labelled them y (correct), i (irrelevant but technically admissible), or n/blank (incorrect). two leaderboards were produced: case 1, counting both y and i as correct, and case 2, counting only y. these settings correspond exactly to tables 2 and 3 . ## conclusion and future work we presented our system, which employs a novel dimension-as-token self-attention mechanism (burst attention) on top of sentence transformers. our experiments demonstrate that, even with an ultra-light hidden dimension (16) and a single attention layer, the approach is competitive in terms of recall, achieving average recall rates of 32.24%, 43.16% and 31.53% in the overall quantitative, qualitative case 1, and qualitative case 2 evaluations, respectively. these results indicate that our model is effective at capturing relevant subject information under resource constraints, although there is still room for improvement. in future work, we plan to explore deeper stacking of burst attention layers, more advanced negative sampling strategies, and the incorporation of hierarchical subject ontologies for improved synonym disambiguation. additionally, we intend to investigate the integration of multimodal features to further enhance system performance. these enhancements aim to further improve both the accuracy and robustness of our subject classification system.