HW - TSC 2024 Submission for the S em E val-2024 Task 1: Semantic Textual Relatedness ( STR ) (2024)

Abstract: The degree of semantic relatedness of two units of language has long been considered fundamental to understanding meaning. In this paper, we present the system of Huawei Translation Services Center (HW-TSC) for Task 1 of SemEval 2024, which aims to automatically measure the semantic relatedness of sentence pairs in African and Asian languages. The task dataset for this task covers about 14 different languages, These languages originate from five distinct language families and are predominantly spoken in Africa and Asia. For this shared task, we describe our proposed solutions, including ideas and the implementation steps of the task, as well as the outcomes of each experiment on the development dataset. To enhance the performance, we leverage these experimental outcomes and construct an ensemble one. Our results demonstrate that our system achieves impressive performance on test datasets in unsupervised track B and ranked first place for the Punjabi language pair.

## introduction the semantic relatedness of two units of language is the degree to which they are close in terms of their meaning @xcite . the linguistic units can be words, phrases, sentences, etc. though our intuition of semantic relatedness is dependent on many factors such as the context of assessment, age, and socioeconomic status @xcite , it is argued that a consensus can usually be reached for many pairs @xcite . in the semeval 2024 shared task 1 @xcite , there are three sub-tracks -track a: supervised, track b: unsupervised, and track c: cross-lingual and each track involves several language pairs. our team -huawei translation services center (hw-tsc) -participated in the track b: unsupervised one which covers most african and asian language pairs and has to be developed without the use of any labeled data for semantic relatedness. in this paper, we describe hw-tsc's system for unsupervised semantic relatedness tasks, which leverages multiple pre-trained multilingual language models to capture the semantic relatedness of different language pairs. the main features of our system are as follows: the first one is xlm-roberta @xcite , a large unsupervised cross-lingual model that extends facebook's roberta model with more languages and data. the second one is multilingual-bert @xcite , a transformers model that is pre-trained on a large multilingual corpus using self-supervised objectives. to measure the similarity between two sentences, we use their n-gram dictionaries as features and compute a similarity score based on them. we use xglm @xcite , a large-scale auto-regressive language model, as the backbone of this method. xglm is a pre-trained language model that can handle multiple languages and domains. by leveraging the powerful large language model, we can efficiently 1 1634 obtain the token logits and perform calculations with them. in this paper, we analyze the characteristics of the shared task and describe our solutions, which include the ideas and implementation processes. we use sentence-bert @xcite as our baseline for the experiment. we conduct various experiments with the base model, large language models, etc. our model achieves the best performance for the punjabi language pairs in the unsupervised track b. the results are encouraging for semantic relatedness, although there is still scope for improvement. ## related work our track is unsupervised, meaning that the systems submitted by participants do not rely on any labeled data for measuring semantic relatedness or similarity between text units longer than two words in any language. consequently, any pretrained language models that are further fine-tuned with text similarity data, using methods such as instruct-tuning, classification, or a similarity objective, are disqualified from our track. for our baseline score, we used sentence-bert @xcite (sbert), a variant of the pre-trained bert network that employs siamese and triplet architectures to generate sentence embeddings that are semantically meaningful and comparable by cosine similarity. sbert has been fine-tuned on natural language inference (nli) data, resulting in sentence embeddings that surpass other state-ofthe-art methods. hence, we selected sbert as our baseline model and obtained our baseline score. we introduce bertscore @xcite ) secondary, which is an automatic evaluation metric for text generation. analogously to common metrics, bertscore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. what is more, different from other matches, it computes token similarity using contextual embeddings. bertscore correlates better with human judgments and provides stronger model selection performance than existing metrics. ## experiments results in the beginning, we applied the following three methods to the english development dataset: ngram chars method with xlmr and mbert, bertscore method, and pre-trained large language model method with xglm to calculate the sum, mean, and half of the logits. see at last, we make other experiments to ensemble the results of ngram-xlmr and bertscore methods to find out if this way can bring us better performance. we make ngram-xlmr with ratio a, and bertscore method with ratio (1-a). see ## conclusion this paper describes hw-tsc's unsupervised system for semantic textual relatednes shared task held in semeval 2024 task 1 and also presents the design, the data, and the results. the participants of the shared task were provided with a collection of unsupervised datasets in multiple languages. the shared task is challenging, partly due to the unsupervised development data, and can not use models that have fine-tuned with text similarity data whether through instruct-tuning (e.g., bloomz @xcite ), classification, or a similarity objective (like sbert). our system uses three base models with the dataset and carries out comprehensive experiments with different pre-trained models and methods. finally, our system achieved the 1st best performance in the punjabi language. for some of the problems reflected in this task, there is still a lot of research space. in the future, we will investigate the transfer method to transfer the knowledge of one language to multiple languages to improve efficiency and we plan to leverage other multiple languages model's skills. devlin, jacob, chang, ming-wei, lee, kenton, toutanova, and kristina. 2018. bert: pre-training of deep bidirectional transformers for language understanding. arxiv preprint arxiv:1810.04805.