Pinealai at S em E val-2024 Task 1: Exploring Semantic Relatedness Prediction using Syntactic, TF - IDF , and Distance-Based Features. (2024)

Abstract: The central aim of this experiment is to establish a system proficient in predicting semantic relatedness between pairs of English texts. Additionally, the study seeks to delve into diverse features capable of enhancing the ability of models to identify semantic relatedness within given sentences. Several strategies have been used that combine TF-IDF, syntactic features, and similarity measures to train machine learning to predict semantic relatedness between pairs of sentences. The results obtained were above the baseline with an approximate Spearman score of 0.84.

## introduction the prediction of semantic relatedness between texts is a crucial task with applications in various natural language processing domains. in this study, our focus is on creating a system capable of predicting semantic relatedness across languages while investigating the features that contribute to this prediction. the development of such a system is not only beneficial for understanding semantic relationships within texts but also holds promise for enhancing deep learning models in tasks such as assessing sentence representation methods, question answering, and text summarization @xcite . despite the advancements in word representation techniques, especially using embeddings, the complexity of human languages presents persistent challenges in accurately capturing semantic relatedness. our experiment primarily concentrates on the english language, acknowledging its significance as a widely used language in various applications. the inherent difficulty in identifying and quantifying the shared elements between two texts necessitates a thoughtful exploration of diverse features and methodologies. in the subsequent sections, we describe the dataset used for this experiment, outline the methodology employed for predicting semantic relatedness, discuss the results obtained, and conclude with insights into the implications and potential future directions of this research. the research will delve into exploring features that enhance the prediction of semantic textual relatedness by developing several strategies concerning feature extractions and model training. ## literature review the complexity of machine-based human language modeling involves a nuanced understanding of various linguistic aspects, notably pragmatics and semantics @xcite . this research specifically emphasizes semantic modeling, with a focus on semantic relatedness, as opposed to the more commonly studied word similarity @xcite @xcite . traditionally, approaches like bag of words have been explored @xcite , but they often fall short in achieving high performance for semantic relatedness tasks. word-nets models, while prioritized, face limitations in language coverage and comprehensive embedding of semantic relationships @xcite . a notable contribution by (gomaa, 2019) introduced a model utilizing multiple similarity features, including cosine similarity and jaccard. their multi-layer architecture demonstrated that employing various similarity features collectively yields significantly better results than applying each measure in isolation. however, the approach did not add or consider syntactic features for the enhancement of the semantic prediction on textual data. recent advancements in deep learning models exhibit superior semantic similarity and relatedness performance. however, there remains a scarcity of research focusing on the distinctive features between semantic textual similarity (sts) and se-mantic textual relatedness (str), and how models can better capture the nuances of semantic relatedness between words and sentences @xcite . ## task description the primary objective of this experiment is to create a system that can predict the semantic relatedness between pairs of texts across various languages but also explore the different features that could help models identify semantic relatedness between given sentences. although the current experiment is focused on english, the development of such a system holds the potential to enhance deep learning models for various tasks, including assessing sentence representation methods, question answering, and text summarization @xcite . ## data description semeval 2024 track 1 utilized data provided by organizers, featuring sentence pairs in training, development, and test sets. each instance is annotated with a score indicating semantic textual relatedness, which ranges from 0 (unrelated) to 1 (related). instance score it that happens, just pull the plug. the two little girls jump on the bed. 0.5 a little girl is jumping down a sandy hill. you're taking a sweater in a shop. 0.03 to taking the life of conor greenleaf. ## methodology in this study, our objective is to predict semantic textual relatedness between two texts. we made two key assumptions: firstly, we refrained from preprocessing the corpus to preserve sentence structure, essential for information retrieval and semantic identification @xcite . secondly, we intentionally excluded large language models (llms) from experiments, anticipating challenges in interpreting specific features contributing to semantic identification due to their contextual abilities and complexity @xcite . to extract diverse features, we employed various distance measures, including jaccard distance, cosine similarity, levenshtein distance, and word mover's distance (wmd) @xcite @xcite . these measures compute the similarity between text pairs based on common words, term vectors, and the minimum distance embedded words need to travel between documents. feature extraction utilized the bag-of-words (bow) technique, specifically term-frequency inverse document frequency (tf-idf) @xcite , and hidden vectors from a pre-trained sentencebert model to compute cosine similarity @xcite . additionally, syntactic features were extracted, parsing each sentence pair to identify words with the same dependency role. this approach resulted in three features: probability of exact word matching, probability of unique words, and probability of related words. the rationale was to explore the impact of words with common dependency roles on predicting semantic relatedness and assess the effect of their absence on the english corpus. the reason behind using traditional models such as gradient boost is that this type of model can easily handle non-linear relationships in texts, which is crucial while capturing semantic relatedness but also can deal with imbalanced datasets @xcite . a diagram of our methodology as well as the source code is freely available on github. the link can be found in the appendix section. ## results the performance metrics of the different models submitted for semantic relatedness are presented. the evaluation metrics include solely the spearman score. even though the models submitted were fasttext and naive bayes, during the training phase linear regression, xgradient boost, random forest, and an ensemble of two traditional models were trained. ## discussions in the methodology section, our primary objective is to identify key features that enhance the capability of models in discerning semantic relatedness within textual data. we pursued two distinct approaches. first, we trained various models by employing tfidf, jaccard, or extracting specific syntactic features from the texts. this process does not take into account the internal structures of the texts which could give more insights about their meaning. when exclusively utilizing syntactic features for model training, we achieved a maximum spearman score of 0.32. training models solely that used tf-idf features to compute a cosine similarity and used the metric to predict yielded a separate score of 0.533. incorporating features extracted from sbert on top of the syntactic features resulted in a notable score increase of 2, reaching approximately 0.70. finally, combining all these strategies during the training phase produced a score of 0.85, with a corresponding score of 0.83 during the testing phase. ## conclusion in conclusion, the study aimed to predict semantic relatedness in english, exploring diverse features and strategies. limitations included the absence of word sense disambiguation algorithms, the exclusion of transformer models for explainability, and the decision not to merge training and development sets. despite these constraints, the models exhibited competitive performance, particularly the gradient boost model, which achieved a spearman score of 0.8371. however, the experiment conducted can not help us derive a conclusion that the model can make a strict difference between semantic relatedness (str) and semantic textual similarity (sts) in texts. the methodology highlighted the impact of syntactic features, tf-idf representations, and sentencebert embeddings. moving forward, addressing limitations, incorporating advanced algorithms, and leveraging diverse datasets but also developing approaches that help models distinguish between str and sts will contribute to a deeper understanding of semantic relations in textual data and further improvements in predictive capabilities.