V erba N ex AI Lab at S em E val-2024 Task 1: A Multilayer Artificial Intelligence Model for Semantic Relationship Detection (2024)

Abstract: This paper presents an artificial intelligence model designed to detect semantic relationships in natural language, addressing the challenges of SemEval 2024 Task 1. Our goal is to advance machine understanding of the subtleties of human language through semantic analysis. Using a novel combination of convolutional neural networks (CNNs), long short-term memory (LSTM) networks, and an attention mechanism, our model is trained on the STR-2022 dataset. This approach enhances its ability to detect semantic nuances in different texts. The model achieved an 81.92% effectiveness rate and ranked 24th in SemEval 2024 Task 1. These results demonstrate its robustness and adaptability in detecting semantic relationships and validate its performance in diverse linguistic contexts. Our work contributes to natural language processing by providing insights into semantic textual relatedness. It sets a benchmark for future research and promises to inspire innovations that could transform digital language processing and interaction.

## introduction the analysis of semantic relationships in natural language is considered an essential pillar for understanding the inherent complexity of textual communication @xcite . with the increasing application of artificial intelligence (ai) models in natural language processing, the ability to discern semantic similarity between text fragments has become a fundamental challenge due to the complexity of natural language and the diversity of meanings that words and phrases can have in different contexts @xcite . in this context, semantic textual relatedness (str) is a crucial element in natural language understanding, gaining increasing significance with integrating artificial intelligence (ai) models in language processing. this article aligns with the objectives set by se-meval 2024 task 1, a pivotal challenge centered on predicting semantic textual relationships between sentence pairs in the english language. the task's importance lies in its profound impact on advancing contextual language understanding, a cornerstone for ai applications across diverse domains. our approach to tackle this challenge involves a four-layer feature extraction process. the first layer focuses on extracting lexical similarity, providing a foundation for understanding semantic connections based on word usage. subsequently, the second layer delves into capturing knowledge-oriented similarity and incorporating domain-specific insights into the model. the third layer concentrates on corpus-oriented features, considering the contextual influence of larger text corpora. finally, in the fourth layer, we employ an embedding approach. here, we train a long short-term memory (lstm) model, extracting sentence features from phoneme embeddings and a sentence transformer model, thereby capturing nuanced semantic nuances. throughout the semeval 2024 task 1 competition @xcite , our system secured the 24th position out of 36 teams, achieving a competitive score of 0.8192. notably, our system was designed and optimized for the english language. to foster transparency and collaboration, we have released our code, accessible at https://github.com/verbanexai/semeval2024 . ## related work the analysis of semantic relations is considered fundamental for understanding the connection of meanings between words, phrases, and sentences in a text. various relationships, such as synonymy, antonymy, hyperonymy, meronymy, and cohyponymy, can manifest in this context. two main approaches have addressed this field: rulebased and machine learning-based approaches. rule-based approaches use ontologies, struc-tures that define concepts and their relationships, and semantic networks, which are graphical representations of these relationships. lexical patterns, which are rules that describe semantic relationships based on the structure of words, have also been used. on the other hand, approaches based on machine learning have gained relevance, utilizing technologies such as convolutional neural networks (cnn), recurrent neural networks (rnn), attention models, and word embeddings. in the literature, we can find several methods for semantic relation detection. in "learning short-text semantic similarity with word embeddings and external knowledge sources" @xcite , authors propose an approach that uses word embeddings and external knowledge to measure semantic similarity between short texts, managing to outperform traditional methods on diverse datasets. another significant work is "a multi-layer system for semantic relatedness evaluation" gomaa (2019), which presents a multi-layer system for semantic relatedness evaluation between sentences, combining various similarity features and achieving promising accuracy on the sick dataset. in addition, "a new methodology for computing semantic relatedness: modified latent semantic analysis by fuzzy formal concept analysis" @xcite proposes a hybrid methodology that combines latent semantic analysis and fuzzy formal concept analysis to compute the semantic relatedness between words and sentences, obtaining improved results compared to other baseline measures on a specific corpus. in the field of language-specific semantic relatedness detection, "sentence embedding and convolutional neural network for semantic textual similarity detection in arabic language" @xcite proposes a deep learning-based approach to detect paraphrases in arabic, using word2vec and a convolutional neural network to overcome traditional methods and other stylometric feature-based approaches. finally, "attention-based model for predicting question relatedness on stack overflow" @xcite introduces a deep learning model called asim, which uses the attention mechanism to predict the semantic relationship between questions in programming question and answer websites. this model outperforms previous models in terms of performance and generalization in detecting duplicate questions and predicting the relationship between knowledge units. despite these advances, knowledge gaps persist. most models focus on semantic relationships at the word or phrase level. however, we need more research in sentence-and paragraph-level relationship detection. in addition, we require more robust models to adapt to different domains and text types, ensuring a more complete and accurate understanding of semantic relations in natural language processing. ## system overview this section outlines our proposed model for tackling the task presented in semeval 2024, track a, which involves assessing the semantic relationship between pairs of sentences. initially, the text data undergoes preprocessing, including separating sentence pairs, followed by training a long short-term memory (lstm) model on the training dataset. subsequently, we extract text features based on a four-layer architecture proposed by @xcite , as illustrated in figure 1 . these layers include word embedding, syntactic relationships, corpus topics, and contextual information. additionally, we incorporate novel features to enhance our model's performance. these include: senticnet: utilized to extract the polarity of sentences in the knowledge-oriented layer. latent semantic indexing (lsi) is employed for the corpusoriented layer to gain insights into the underlying structure of the text corpus. phoneme extraction: a novel approach to capture phonetic information from the sentences. furthermore, we integrate an attention mechanism inspired by @xcite to effectively capture intricate dependencies within sequences. leveraging insights from recent advancements, our model incorporates a part-of-speech (pos)-aware and layer ensemble transformer, further enhancing its ability to discern semantic relationships. by drawing from diverse studies on data augmentation, ensemble learning, and transformer-based profiling, our model aims to provide a robust solution for semantic relationship detection. it showcases a comprehensive understanding of attention mechanisms and their integration with state-of-theart techniques. ## experimental setup the system model utilized sentences without lemmatization and stopword removal, preserving the original form of the sentences to capture a broader range of semantic nuances. phoneme extraction, as proposed by del castillo, was incorporated into the system, utilizing specific components of the provided code. this inclusion aims to enhance the model's sensitivity to phonetic features, contributing to a more comprehensive understanding of textual relationships. we employ the shufflesplit method for crossvalidation to assess the model's training. the dataset was split into training and validation sets using a test size 25%, and we utilized ten pairs (@xmath0. we set the random state to 42 (@xmath1. the details of the library versions used in the implementation are provided in ## results we evaluated the semantic relation detection model using the training and test datasets, and the results are in while the results obtained are below the established baseline, we recognize opportunities for improvement. abstraction analysis could reveal the specific contributions of each system component to this performance and guide future improvements. it is also crucial to consider the nature of the baseline and the inherent complexity of the task at hand. ## limitations while presenting our approach for evaluating semantic relations between sentences, it's crucial to acknowledge certain limitations that may impact the interpretation and applicability of our proposed model. we outline these limitations below: ## ethical considerations we linked the text similarity field to the detection of paraphrasing @xcite , which can pose an ethical problem when using an author's work without proper citation. our solution addresses the bias by extracting various text features, from word information to context and vector representation. this way, we can avoid some limitations from training the model with insufficient features. ## conclusions this paper presented a comprehensive approach to evaluating semantic relations between sentences, addressing the challenges posed by semeval 2024 task 1. our model employs a sophisticated fourlayered feature extraction technique, encompassing lexical similarity, knowledge orientation, corpus orientation, and embedding layers. despite achieving a no while the spearman correlation of 0.8192 places our model slightly below the established baseline of 0.83, this outcome provides an invaluable learning experience. moving forward, we plan to conduct ablation studies to dissect the impact of individual components, explore alternative models and preprocessing strategies, and conduct a detailed error analysis to address specific shortcomings. ultimately, this work contributes to a deeper understanding of semantic relations and provides a competitive model for semeval 2024. we are committed to advancing semantic understanding and improving ai systems for natural language processing. the journey from this competition is a stepping stone toward more refined and practical solutions in semantic relationship detection.