T artu NLP at S em E val-2025 Task 5: Subject Tagging as Two-Stage Information Retrieval (2025)

Abstract: We present our submission to the Task 5 of SemEval-2025. We frame the task as an information retrieval problem, where the document content is used to retrieve subject tags from a large subject taxonomy. We leverage two types of encoder models to build a two-stage information retrieval system—a bi-encoder for coarse-grained candidate extraction at the first stage, and a cross-encoder for fine-grained re-ranking at the second stage.

## introduction semeval-2025 task 5 aims to produce a technical solution to annotate a large collection of documents with relevant subject tags @xcite . subject tags come from the gnd (gemeinsame normdatei in german or integrated authority file in english) subject taxonomy. for example, an article dealing with seismic resistance of industrial buildings may have subject tags such as "industrial plant technology (general)" and "earthquake safety construction engineering". systematic and precise metadata annotation, subject tagging in particular, is essential for digital collections of documents to be usable and useful for the end-users as information retrieval and knowledge discovery sources. however, producing such metadata, especially at scale, requires an immense amount of specialist time and effort. for instance, a recent prototype at the national library of estonia, kratt, found that even with machine assistance catalogers still needed to validate many tags, highlighting the labor bottleneck @xcite . the core idea behind our system is that given a tag definition and a document to tag, it should be possible to predict whether the given tag is suitable for the document by measuring how similar their representations are using a pre-trained language model. this can be thought of as an information retrieval problem. bi-encoder and cross-encoder models are two common ways to approach this problem @xcite @xcite , both with their strengths and weaknesses. a bi-encoder processes documents and queries independently to produce their vector representations. then, cosine similarity is measured between the document and query vectors to score relevance. bi-encoder allows for efficient retrieval in large document collections as the document representations can be pre-computed. the downside, however, is that it is impossible to capture token-level similarities between documents and queries with this approach. meanwhile, a cross-encoder processes document and query pairs simultaneously, capturing such similarities and producing more fine-grained similarity scores. the computational cost of using a cross-encoder is significantly larger compared to that of a bi-encoder, making it generally unsuitable for large document collections. accordingly, we combine both in a single two-stage system to produce the optimal result. our system scored closer to the middle of the quantitative leaderboard ( ## background in the shared task, the participants were given a collection of english and german documents containing a title and an abstract. the goal was to recommend the top n subjects most relevant from a predefined set of subjects. each subject was represented by an alphanumeric code, a name in german, and an optional definition in german. training and validation data provide ground truth tags for each document. two subtasks were offered to the participants: a complete collection of subjects (tibkat) and a smaller subset of the collection (tibkat-core). the former also contained a larger number of documents. in our experiments, we used the smaller tibkat-core dataset. however, we submitted the final results for both subtasks. ## system overview our system is based on the two-stage approach to information retrieval. we consider the available texts of the documents (usually, just the title and the abstract) to be the queries, while the subject definitions act as the documents to be retrieved. in the first stage, we employ an approximate nearest neighbors (ann) search algorithm @xcite over pre-computed subject embeddings. in the second stage, these n retrieved subject descriptions are re-ranked using a crossencoder model @xcite . ## experimental setup the shared task data underwent minimal transformations to be suitable for our approach. for each document, the title and the abstract were concatenated. similarly, the name and definition (when available) were concatenated for each subject. thus, each document and each subject was represented by a single string, which served as input to our models. our solution primarily relied on the sentence-transformers foot_1 library for both stages of the system. we employed multilingual-e5-largeinstruct foot_2 @xcite as the first stage model to produce document and subject representations. we customized the prompt used to encode the documents to include the following: "instruct: given the following title and abstract for the document, for the second stage, we fine-tuned a crossencoder based on mdeberta-v3-base foot_4 @xcite with default parameters using sentence-transformers. we trained on positive examples from document-subject pairs in the training set and constructed negative examples by pairing documents with randomly sampled subjects not explicitly linked to them. the model achieved high performance quickly during training; we stopped after one epoch as the f-score plateaued near 0.97. the resulting model was used to make predictions on the validation split. the model is available on huggingface foot_5 . ## results with our submission to the shared task, we aimed to build a hackathon-like proof-of-concept system to test the feasibility and measure the benefits of applying a two-stage information retrieval approach to match document contents with a structured knowledge resource. more specifically, our interest lied in the improvements attained by the second stage model at the cost of added complexity and computational requirements compared to using only bi-encoder representations for retrieval. with that purpose in mind, we submitted two runs for final evaluation: bi-encoder retrieval and two-stage retrieval. the results in however, the enhanced performance comes with computational implications. both stages involve parameters that directly impact scalability: upon examination of the results of the qualitative evaluation, we discover that the errors made by our system are primarily related to subjects with similar names and no definitions. the incorrectly assigned subject seems to be generally vaguely related to the document. however, they define a different subfield of a relevant subject. for example, for an article titled "model-based engineering of an automotive adaptive exterior lighting system: realistic example specifications of behavioral requirements and functional design", some of the tags that are considered correct are "automotive engineering, vehicle construction, conveyor technology, aerospace technology" and "system planning it, data processing". meanwhile, "lighting technology electrical engineering, electrical power engineering: calculation, design and construction of lighting systems" and "outdoor lighting electrical engineering, electrical power engineering" are only technically correct. finally, "adaptive process model measurement, control and regulation technology" and "model-based testing computer science, data processing" are assigned incorrectly to this document by our system. the main limitation of our approach is the reliance only on text representations. using additional information such as related subjects and tag co-occurrence could have improved the performance of our system. ## conclusion this paper described our solution to semeval-2025 task 5 based on a two-stage information retrieval system, where we used the documents to annotate as queries to retrieve candidate subject tags from a large collection of subject tags. for both stages we view sentence similarity between document texts and subject tag descriptions as the relevance score. our system demonstrates moderate performance. however, we confirmed our hypothesis that a second-stage re-ranker substantially improves the performance of the system compared to using a bi-encoder as the only stage.