UA lberta at S em E val-2024 Task 1: A Potpourri of Methods for Quantifying Multilingual Semantic Textual Relatedness and Similarity (2024)

Abstract: We describe our systems for SemEval-2024 Task 1: Semantic Textual Relatedness. We investigate the correlation between semantic relatedness and semantic similarity. Specifically, we test two hypotheses: (1) similarity is a special case of relatedness, and (2) semantic relatedness is preserved under translation. We experiment with a variety of approaches which are based on explicit semantics, downstream applications, contextual embeddings, large language models (LLMs), as well as ensembles of methods. We find empirical support for our theoretical insights. In addition, our best ensemble system yields highly competitive results in a number of diverse categories. Our code and data are available on GitHub.

## introduction in this paper, we describe our submission for semeval-2024 task 1: semantic textual relatedness (str) @xcite , which is based on the semrel2024 dataset @xcite . each instance consists of a pair of sentences in the same language, annotated with a score that quantifies their semantic relatedness. semrel2024 was annotated by native speakers of the dataset's 14 languages, which span five language families. an example english instance consists of the sentence pair "the story is gripping and interesting" and "it's a brilliant, compelling, and heartfelt story", which is annotated with a relatedness score of 0.64. we participated in all three tracks (supervised, unsupervised, and crosslingual) on all 14 languages. semantic relatedness is distinct from semantic similarity. sentences that express opposite propositions, such as "it is raining" and "it is not raining", exhibit low similarity but high relatedness. the impact of relations such as antonymy and meronymy @xcite ) make semantic similarity a more specific task: similarity implies re-latedness, but not vice versa. nevertheless, many traditional algorithms make no attempt to distinguish between the two tasks @xcite . for example, the word overlap baseline in this shared task could also be applied to measure semantic similarity. the extent to which semantic similarity and relatedness correlate in practice remains an important open question. in this paper, we test the hypothesis that similarity is a special case of relatedness @xcite through implementing an array of methods that are designed to measure similarity, and applying them to the task of measuring relatedness. we experiment with several different approaches: our experimental results provide support for our theoretical insights. the experiments on the supplementary datasets demonstrate a high correlation between the str and sts tasks. out of 51 competing teams, we rank among the top three entries in 16 of the language/track settings. in particular, our best-performing supervised ensemble system achieves the highest score in the english track a among the teams that submitted a system description paper @xcite . taken together, these results support the idea of using similarity as a proxy for relatedness, as predicted by our hypotheses. ## methods we investigate ten different methods, divided into four types. each method takes as input a pair of sentences, and produces a scalar value, which, possibly after some normalization to place it within the range specified for this shared task, is used as a measure of str. thus, each individual method is a complete, functional str method; our principal innovation is the ensembling of these methods into a single str system. ## cross-lingual textual relatedness in this section, we discuss our experiments on new cross-lingual datasets which we created from the shared task data. the purpose of these experiments is to test our hypothesis that semantic relatedness is the wordovlap baseline performs poorly when applied to the eng-esp dataset because orthographic forms rarely match across languages even if they have the same meaning. in contrast, conceptovlap performs much better, as it is entirely multi-lingual and independent of orthography and script. however, both methods obtain similar results on eng-esp*, where the spanish text is translated into english. our amr, nli, and xgb-4ms systems cannot be applied to cross-lingual pairs, but when spanish is translated into english, their performance on eng-esp* is comparable to what is observed on the english test set. the most interesting findings emerge from the results of prompt and fusion. both are applicable directly to cross-lingual data, without translating the spanish sentences into english. surprisingly, for both methods, we observe only small differences between the relatively high numbers in the three columns. this finding supports our hypothesis that translation does not affect the degree of relatedness between a pair of sentences. ## semantic textual similarity another hypothesis that we investigate is that similarity is a special case of relatedness. therefore, we expect a strong correlation between the two concepts: sentences that are highly similar should also be considered highly related. in this section, we test this hypothesis by applying our methods to sts datasets (i.e., track 5 and 4a) from semeval 2017 task 1 @xcite . since both str and sts tasks output numerical scores on pairs of sentences, our methods can be directly applied to sts without modification. for supervised methods, we apply the models to sts in the same way as to str, without any additional training or fine-tuning. this approach can be viewed as transfer learning: models trained on str datasets are tested on the sts task. for crosslingual datasets, we again experiment with both direct application to different languages (eng-esp) and pre-translation into english (eng-esp*). a more detailed analysis of the individual methods reveals several additional insights. among the explicit methods, wordovlap and amr are both outperformed by conceptovlap on sts, which is likely due to their lack of robustness in crosslingual settings. among the extrinsic methods, pi works better than nli, exhibiting remarkable stability across tasks, likely due to the training of our pi model on diverse benchmark datasets, including adversarial examples from paws qqp and paws wiki. among the distributional methods, while embed-b consistently outperforms embed-r, both experience a decline in cross-lingual performance, revealing sensitivity of sentence embeddings to translation noise. among llms-based methods, prompt excels on cross-lingual sts benchmarks, possibly because of data leakage in training chat-gpt, as well as its multilingual design. our ensemble system xgb-4ms generally delivers the best results. ## conclusion we have investigated a wide array of methods on two of sentence-level semantic tasks in both mono-lingual and cross-lingual settings. in the process, we assembled a comprehensive benchmark of datasets for future explorations in this domain. the experiments furnish evidence for two hypotheses: (1) semantic similarity is a special case of semantic relatedness, and (2) both similarity and relatedness are preserved under translation. in practical terms, the evaluation results indicate that ensembling llms with diverse architectural designs yields the most robust and effective performance across languages and tasks. notably, our strongest system is at the top of the ranked teams in english track a, the setting with the highest number of participants.