NLP _ STR _team S at S em E val-2024 Task1: Semantic Textual Relatedness based on MASK Prediction and BERT Model (2024)

Abstract: This paper describes our participation in the SemEval-2024 Task 1, “Semantic Textual Relatedness for African and Asian Languages.” This task detects the degree of semantic relatedness between pairs of sentences. Our approach is to take out the sentence pairs of each instance to construct a new sentence as the prompt template, use MASK to predict the correlation between the two sentences, use the BERT pre-training model to process and calculate the text sequence, and use the synonym replacement method in text data augmentation to expand the size of the data set. We participate in English in track A, which uses a supervised approach, and the Spearman Correlation on the test set is 0.809.

## introduction we participated in the english language of track a in task 1, "semantic textual relatedness for african and asian languages." track a uses a supervised approach where systems are trained on labeled training datasets. this task detects the degree of semantic relatedness between pairs of sentences for african and asian languages @xcite . semantic textual relatedness (str) is an important measure of the relationship between texts. it is considered to be the basis for understanding meaning @xcite and is crucial for many natural language processing tasks. by computing semantic textual relatedness, we can perform applications such as text matching @xcite , information retrieval @xcite ), text categorization @xcite , and question answering systems @xcite . however, previous nlp work has focused on semantic similarity (a small subset of semantic relatedness), in large part due to the lack of datasets on relatedness. for example, semeval-2015 task1 is paraphrase and semantic similarity in twitter @xcite . and semeval-2016 task1 is semantic textual similarity, monolingual and cross-lingual evaluation @xcite . semantic relatedness and semantic similarity are two ways to explore the closeness of meaning. two terms are considered semantically similar if there is a synonym, contextual, or modal relation relationship between them. two terms are considered semantically related if there is any lexical semantic relation between them. thus, all similar pairs are also related, but not all related pairs are similar @xcite . in semantic textual relatedness, we focus on the meaning and semantic information of the text, not just the surface word or sentence structure. thus, the semantic relatedness between two texts can relate to their themes, intentions, emotions, etc. the semantic relatedness of texts can be computed using the content and links of hypertext encyclopedias @xcite . semantic relatedness between texts can also be measured by calculating the similarity between text representations using a pre-trained language model. in the following, we describe in detail the methods we used and give the evaluation results and conclusions. ## background in this section, we present important details about the task setup. each instance in the train set, dev set, and test set is a sentence pair, and these two sentences are separated by a newline character. the instance is labeled with a score representing the degree of semantic textual relatedness between the two sentences @xcite . as shown in the scores can range from 0 (maximally unrelated) to 1 (maximally related), which are obtained using a comparative annotation framework. the sentence1 sentence2 str score a girl is communicating with sign language. a young girl is using sign language. 0.83 you should have respect for your mother. even if this is your own mother! 0.41 ## system overview in this section, we present our approach applied to the task of predicting str. we use the bert pre-training model @xcite ) for text sequence processing and computation, and also employ text data augmentation to improve the training results. we adopted prompt tuning @xcite to construct a new sentence, "the correlation of the next two sentences sent0 and sent1 is @xcite .", and used this constructed new sentence as a prompt template, where [mask] is used to predict the correlation between the two sentences. ## experimental setup the data set is given in csv file format by the semeval 2024 shared task organizer. it has three columns: pairid, text, and score, where text is a sentence pair. we take out the two sentences in the sentence pair and use these two sentences to construct a new sentence: "the correlation of the next two sentences sent0 and sent1 is @xcite .". this new sentence is then fed into the model for processing and training. when performing text data augmentation, we replace the two sentences with synonyms and then insert the newline character in the middle of the replaced sentence pairs to ensure that the data format is consistent with the original data set. we use the bert pre-training model to process and calculate text sequences. the text data augmentation method is synonym replacement. since this task is a regression task, we use the mean squared error(mse) loss function: the batch size is set to 64, the number of training iterations is 6, and the learning rate is 2e-2. at the same time, in order to help the model converge better and achieve better performance, we set up a learning rate scheduler. ## conclusion this paper describes our participation in the se-meval 2024 competition in the semantic textual relatedness for african and asian languages task. we participated in the english task in track a. our approach is to use the bert pre-training model for text sequence processing and computation, employing text data augmentation to enlarge the size of the train set, and adopting prompt tuning to construct a prompt template "the correlation of the next two sentences sent0 and sent1 is @xcite .", where [mask] is used to predict the correlation between two sentences. the final spearman correlation obtained on the test set was 0.809. in the future, we will use methods such as context awareness and manual intervention to address errors caused by text data augmentation to ensure their accuracy and rationality. at the same time, we will expand the size of the dev set, reduce the size difference between data sets, and try to use other more powerful pre-trained models.