IITK at S em E val-2024 Task 1: Contrastive Learning and Autoencoders for Semantic Textual Relatedness in Multilingual Texts (2024)

Abstract: This paper describes our system developed for the SemEval-2024 Task 1: Semantic Textual Relatedness. The challenge is focused on automatically detecting the degree of relatedness between pairs of sentences for 14 languages including both high and low-resource Asian and African languages. Our team participated in two subtasks consisting of Track A: supervised and Track B: unsupervised. This paper focuses on a BERT-based contrastive learning and similarity metric based approach primarily for the supervised track while exploring autoencoders for the unsupervised track. It also aims on the creation of a bigram relatedness corpus using negative sampling strategy, thereby producing refined word embeddings.

## introduction the semantic relatedness between texts in a language is fundamental to understanding meaning @xcite . automatically detecting relatedness plays an essential role in evaluating sentence representations, question answering, and summarization @xcite . the fundamental difference between semantic similarity and relatedness is that semantic similarity only considers paraphrase or entailment relationships. in contrast, relatedness accounts for all commonalities between two sentences, e.g., topical, temporal, thematic, contextual, syntactic, etc. @xcite . as highlighted in in track a (task 1) of the semantic textual relatedness (str) task @xcite , we are expected to calculate the degree of semantic relatedness between pairs of sentences in 14 different languages covering both african and asian ## background there have been several attempts to define and distinguish semantic relatedness from semantic simi-larity. the basic metric used in these experiments is spearman rank correlation. the correlation coefficient is calculated between the correctly annotated scores for the set of pairs of sentences and the scores returned by the models. essentially, this removes the absolute values of the scores and focuses on the relative values and, hence, the relative relatedness between the pairs of sentences. initial experiments explored frequency measures such as lexical overlap @xcite , related words, and related subjects and objects, leading to high spearman correlations of 0.82 and 0.83 for bertbase (mean) and roberta-base (mean) on the complex dataset @xcite . despite marginal improvements over a lexical overlap baseline, unsupervised models offer limited enhancement. normalized google distance(ngd) has been used as a novel metric for measuring semantic relatedness between words or concepts (lopes and moura, 2019), utilizing google search result counts to quantify relatedness. ngd (cilibrasi and vitanyi, 2007) normalized counts considering the overall corpus size and the co-occurrence of terms in web pages. various approaches have been proposed for the arabic language (al sulaiman et al., 2022) like automatic machine translation to translate english semantic textual similarity (sts) data into arabic, interleaving english sts data with arabic bert models, and employing knowledge distillationbased models to fine-tune them using a translated dataset. multilingual knowledge distillation (reimers and gurevych, 2019) techniques have been proposed where a student model, m , learns from a teacher model, m , on source language sentences and their translations by minimizing the mean-squared loss function. focusing on lowresource indian languages, a range of sbert models has been introduced for ten popular indian languages. indicsbert (deode et al., 2023) utilized a two-step training method, fine-tuning models using the nli dataset followed by semantic textual similarity benchmarking (stsb), resulting in substantial improvements in embedding similarity scores and cross-lingual performance. ## dataset description the dataset @xcite ) consists of a total of 14 languages, namely afrikaans, algerian arabic, amharic, english, hausa, indone-sian, hindi, kinyarwanda, marathi, modern standard arabic, moroccan arabic, punjabi, spanish, and telugu. every language consists of pairs of sentences with scores representing the degree of semantic textual relatedness between 0 and 1. the scores have been assigned to sentence pairs through a comparative annotation process @xcite . at the preliminary level, dataset length is the only non-semantic variable in these datasets. to assess semantic relatedness, it is crucial to mitigate these biases. correlation coefficients between sentence lengths and scores fall in the range -0.13 < ρ < 0.15; hence, there is no discernible correlation between sentence lengths and scores, suggesting a well-distributed dataset suitable for training. ## system overview our baseline system of scoring a pair of sentences uses jaccard similarity, a lexical metric that calculates the number of token intersections over total tokens in both sentences. our approach involves using contrastive learning for the supervised part and auto-encoders for the unsupervised part. all these methods are discussed below, and our model supervised architecture is shown in figure 1 . ## results the results for the supervised and unsupervised are mentioned in ## error analysis after the evaluation phase, we were provided with the labels for the evaluation data. on experimenting with the semantic relatedness scores for some languages, mainly hausa and kinyarwanda, we found out that our system was not performing well enough on these languages even after subsequent training and hyperparameter optimizations. the issue would be primarily attributed to the complex lexical variations and grammar rules of these languages. as for the unsupervised track, generating a bigram corpus for the case of amharic seemed difficult due to its language structure. ## conclusion by utilizing various approaches like contrastive learning, autoencoders, a custom relatedness metric incorporating all of the available lexical similarity metrics, we have developed a system capable of evaluating the degree of semantic relatedness between pairs of sentences in diverse high and low resource languages. in future, we will study the properties of each low resource language to find out where the models are performing poorly than relying too much on pre-trained models. this would give much clearer insights into semantics of each language thus improving the overall efficiency and performance of our system.