T übingen- CL at S em E val-2024 Task 1: Ensemble Learning for Semantic Relatedness Estimation (2024)

Abstract: The paper introduces our system for SemEval-2024 Task 1, which aims to predict the relatedness of sentence pairs. Operating under the hypothesis that semantic relatedness is a broader concept that extends beyond mere similarity of sentences, our approach seeks to identify useful features for relatedness estimation. We employ an ensemble approach integrating various systems, including statistical textual features and outputs of deep learning models to predict relatedness scores. The findings suggest that semantic relatedness can be inferred from various sources and ensemble models outperform many individual systems in estimating semantic relatedness.

## introduction identifying semantic relatedness is a 'related' task to many well-studied tasks of semantic similarity. according to @xcite , two sentences are considered similar if they are paraphrases or share a relation of entailment. semantic relatedness, however, is a broader concept than semantic similarity. two expressions are considered related if they share any semantic association. for instance, 'teacher' and 'student' are related because they frequently occur within the same context or domain. similarly, 'tasty' and 'unpalatable' are related, as both terms are used to describe food, albeit with opposite meanings. semeval-2024 task 1 @xcite ) is designed to estimate the relatedness of sentence pairs. the task is based on a multilingual dataset of 14 languages and offers supervised, unsupervised and cross-lingual tracks. our team participated in two tracks, and a subset of available languages: track a (supervised learning) for english, and track b (unsupervised learning) for english, spanish, and hindi. we posit that semantic relatedness can be inferred from a multitude of sources and therefore propose an ensemble approach that integrates outcomes from diverse systems to estimate semantic relatedness. our study explores features from textual statistical analysis, general large language models, word embedding models, and models trained on semantic labeled datasets, question-answering pairs, or title-passage pairs in estimating semantic relatedness, and we conducted ensemble experiments with these features. ## related work semeval in previous years has introduced tasks focusing on semantic textual similarity to evaluate the degree of similarity between sentence pairs @xcite @xcite . there tasks provided datasets with human labeled similarity scores, which have been extensively utilized for training sentence embedding models and conducting semantic evaluations @xcite @xcite . ## methodology in this study, we hypothesize that semantic relatedness covers a broader spectrum than semantic similarity in theory. consequently, the integration of various systems and features should achieve superior results compared to individual systems. ## results and analysis the shared task evaluates the participating systems based on spearman's correlation (r) between the human-annotated scores, which ranges from 0 to 1. in results presented in the ensemble approach for english and spanish unsupervised tasks also achieved relatively high scores, despite the absence of similarity or relatedness scores in learning. it suggests that semantic relatedness can be estimated without necessarily relying on human-annotated scores of semantic similarity or semantic relatedness. other sources like question-answering pairs or statistical features of texts also play a role in relatedness estimation. thus, the ensemble of statistical text features, word embedding models, and models trained on question-answer pairs can achieve good results. although the results for hindi did not match the superior outcomes of other supervised models, such as mpnet-base and labse, which were trained with semantic labels or similarity scores, the ensemble system's performance still surpasses that of the multilingual bert embedding model and the multi-qa model, both of which were utilized for ensemble learning as base models. ## conclusion our system employs an ensemble approach to estimate semantic relatedness, integrating results from multiple systems: textual statistical features, word embedding models, and sentence representation models. our findings suggest that semantic relatedness can be deduced from a variety of sources. although some features (e.g., lexical overlap ratio) may not perform as strongly as models specifically designed to obtain sentence representations, the results demonstrate that these features, when used in a combined manner, can outperform many individual systems and collaboratively achieve a better correlation with human judgment on semantic relatedness.