sca LAR S em E val-2024 Task 1: Semantic Textual Relatednes for E nglish (2024)

Abstract: This study investigates Semantic TextualRelated- ness (STR) within Natural LanguageProcessing (NLP) through experiments conducted on a dataset from the SemEval-2024STR task. The dataset comprises train instances with three features (PairID, Text, andScore) and test instances with two features(PairID and Text), where sentence pairs areseparated by '/n’ in the Text column. UsingBERT(sentence transformers pipeline), we explore two approaches: one with fine-tuning(Track A: Supervised) and another without finetuning (Track B: UnSupervised). Fine-tuningthe BERT pipeline yielded a Spearman correlation coefficient of 0.803, while without finetuning, a coefficient of 0.693 was attained usingcosine similarity. The study concludes by emphasizing the significance of STR in NLP tasks,highlighting the role of pre-trained languagemodels like BERT and Sentence Transformersin enhancing semantic relatedness assessments.

## introduction semantic textual relatedness (str) is a crucial concept in natural language processing (nlp), focusing on determining the degree of similarity between linguistic units like words or sentences based on their meaning. this measure plays a vital role in evaluating the effectiveness of large language models (llms) and aids in various nlp tasks. at its core, str delves into understanding the closeness in meaning between two pieces of text. it examines different dimensions of relatedness, including sharing the same viewpoint, originating from the same context, or complementing each other's content. for instance, if two sentences convey similar ideas through paraphrasing or entailment, they might be considered semantically similar. however, relatedness en-compasses all possible commonalities between them. in nlp, researchers and practitioners leverage str to enhance textual coherence, refine narrative structures, and tackle diverse language understanding challenges. by quantifying semantic relatedness, nlp systems can better comprehend and generate human-like responses, ultimately advancing the capabilities of language models. the concept of semantic relatedness between language units has been recognized as foundational in understanding meaning. the automatic determination of relatedness has found numerous applications, including the evaluation of sentence representation methods, question answering, and summarization. semantically similar sentences are those that exhibit either a paraphrasal or entailment relationship. in contrast, relatedness encompasses a broader spectrum of commonalities between two sentences. this includes considerations such as whether they pertain to the same topic, convey the same perspective, emerge from the same temporal context, or if one sentence elaborates on or logically follows from the other. despite the significance of relatedness, much of the prior work in natural language processing has predominantly fo-cused on semantic similarity, particularly within the context of english. we explored sbert. sentence-bert @xcite builds upon the architecture of bert (bidirectional encoder representations from transformers) @xcite , leveraging transformer-based models to encode contextual information from input sentences. unlike bert, which focuses on token-level representations, sentence-bert aims to generate fixed-size representations for entire sentences. to achieve this, sentence-bert employs siamese or triplet network architectures, which are trained on sentence pairs or triplets with similar or dissimilar semantic meanings. through contrastive loss functions, sentence-bert learns to map semantically similar sentences closer together in the embedding space while pushing dissimilar sentences farther 1 902 apart. ## background the exploration of semantic relatedness in language finds its roots in seminal works by @xcite and @xcite , which laid early foundations for understanding the subtleties of meaning in text. initially, these efforts primarily focused on semantic similarity, assessing the likeness between linguistic units through techniques like paraphrasing or entailment. however, as research progressed, scholars began recognizing the necessity of considering a broader array of connections between text segments, thereby giving rise to the concept of semantic relatedness. semantic similarity denotes the extent of resemblance in meaning between two linguistic units, while semantic relatedness encompasses a wider spectrum of connections, encompassing elements such as topical relevance, viewpoint alignment, temporal coherence, and logical sequence. while semantic similarity often relies on paraphrasing or entailment, relatedness factors in various nuances contributing to the overall coherence and cohesion of text. traditional methodologies for measuring semantic relatedness relied on lexical and syntactic features, including word overlap, syntactic parse trees, and semantic networks. however, the emergence of deep learning techniques has ushered in a paradigm shift towards leveraging neural embeddings and transformer-based models to capture richer semantic representations. these modern approaches have demonstrated superior performance across various semantic textual relatedness (str) tasks, such as semantic similarity estimation and semantic textual entailment. in the field of natural language processing (nlp), assessing the relatedness between pairs of sentences is a fundamental task. the paper @xcite addresses this challenge by proposing an innovative approach that combines two key techniques. first, the authors leverage embedding similarity techniques, utilizing seven different transformers to generate sentence vectors. these vectors capture the semantic content of sentences, allowing for more accurate relatedness assessment. second, a classical machine learning regressor is trained on these sentence vectors. by integrating these methods, the study achieved impressive results on the sick dataset. specifically, the mean square error is reduced to 0.0481, and high pearson's and spearman's correlations of 0.978 and 0.9696, respectively, demonstrate the effectiveness of this approach. overall, this research highlights the potential of combining embedding similarity techniques with machine learning for improving relatedness score assessment and advancing nlp algorithms. the zero-shot text classification (0shot-tc) has garnered significant attention. this task involves detecting classes that the model has never encountered during training. the emergence of pretrained language models has transformed 0shot-tc into a binary classification problem, akin to textual entailment. specifically, the model learns whether there is an entailment-relatedness (yes/no) between a given sentence (premise) and each category (hypothesis). however, existing approaches struggle with fully expressing the category space using labels or label descriptions. in contrast, humans can effortlessly extend a set of words to describe the categories to be classified. to bridge this gap, the paper @xcite introduces a novel method called semantically extended textual entailment (sete). inspired by human knowledge extension, sete enriches category representations using a combination of static knowledge (e.g., expert knowledge, knowledge graphs) and dynamic knowledge (e.g., language models). early methods, such as bag of words (bow) and term frequency-inverse document frequency (tf-idf), fell short in capturing nuanced word meanings and context. to address this, the paper @xcite surveys the evolution of semantic similarity techniques, categorizing them into knowledge-based, corpus-based, deep neural network-based, and hybrid approaches. by examining the strengths and limitations of each method, the survey provides a comprehensive overview for researchers navigating the complex landscape of semantic similarity research. understanding the degree of semantic relatedness between two language units is fundamental. however, prior research has primarily focused on semantic similarity, a subset of relatedness, due to the scarcity of relatedness datasets. to address this gap, the authors @xcite introduce the semantic textual relatedness (str-2022) dataset, comprising 5,500 english sentence pairs manually annotated using a comparative annotation framework. human intuition regarding sentence relatedness proves highly reliable, with a repeat annotation correlation of 0.84. the dataset not only 2 903 facilitates exploration of what makes sentences semantically related but also serves as a valuable resource for evaluating automatic sentence representation methods and various downstream nlp tasks. ## system overview a model in the sentence-transformers library is the bert-base-nli-mean-tokens collection. it is especially made for the purpose of semantic textual similarity (sts). sentences and paragraphs are mapped to a 768-dimensional dense vector space by this paradigm. the pre-training phase bert-base-nli-mean-tokens are pre-trained using the conventional bert architecture. this model is pretrained on the tasks of modeling masked languages (mlm): tokens in the training data are masked with a unique token [mask] or randomly substituted with a small percentage, bidirectional contextualization: by analyzing both left and right context, bert generates bidirectional contextualized embeddings as it learns to forecast masked tokens and next sentence prediction (nsp): in the original text, bert further forecasts if two sentences will come after one another. in learning sentence relationships, this aids the model. fig - 1 shows the visualization of sentence embeddings from this model. ## conclusion in order to address task 1 at semeval-2024, this paper presents the use of a bert-base model embedding. for our submission, we chose a supervised (finetuning) and unsupervised (not finetuning) approach, utilizing pre-trained transformers that are already tailored to the domain. based on this strategy, we used the contextual embeddings generated by the sentence transformer and used cosine similarity to measure the similarity between pairs of sentences, thereby quantifying the similarity between them. although our method was successful, there is still room for improvement, as evi-4 905 denced by the final ranking. possible alternate approaches include utilizing the zeroshot capabilities of models like gpt , increasing the training data size by adding more datasets. there is some space for improvement in our straightforward method when compared to the top-performing models. it is noteworthy, nonetheless, that the assignment could be completed with a reasonable computational cost and no further pre-training thanks to google colab's free online tools.