USTCCTSU at S em E val-2024 Task 1: Reducing Anisotropy for Cross-lingual Semantic Textual Relatedness Task (2024)

Abstract: Cross-lingual semantic textual relatedness task is an important research task that addresses challenges in cross-lingual communication and text understanding. It helps establish semantic connections between different languages, crucial for downstream tasks like machine translation, multilingual information retrieval, and cross-lingual text understanding.Based on extensive comparative experiments, we choose the XLM-R-base as our base model and use pre-trained sentence representations based on whitening to reduce anisotropy.Additionally, for the given training data, we design a delicate data filtering method to alleviate the curse of multilingualism. With our approach, we achieve a 2nd score in Spanish, a 3rd in Indonesian, and multiple entries in the top ten results in the competition’s track C. We further do a comprehensive analysis to inspire future research aimed at improving performance on cross-lingual tasks.

## introduction semantic textual relatedness (str) encompasses a broader concept that takes into account various commonalities between two sentences. this includes factors such as being on the same topic, expressing the same viewpoint, originating from the same period, one sentence elaborating on or following from the other, and more. semeval is an international workshop on semantic evaluation. in track c of semeval-2024 task 1: cross-lingual @xcite , participants are to submit systems,which are developed without the use of any labeled semantic similarity or semantic relatedness datasets in the target language and with the use of labeled datasets @xcite ) from at least one other language. various methods were proposed to address the task of textual relatedness. one common approach is based on feature engineering, where the syntactic, semantic, and structural features of text, such as word frequency, tf-idf, and word embeddings, are extracted. machine learning algorithms are then employed for relatedness determination. another popular approach is based on deep learning methods, such as convolutional neural networks @xcite , recurrent neural networks @xcite , and selfattention mechanisms @xcite . these methods can capture semantic relationships and contextual information within the text, and they are trained on large-scale datasets to enhance model performance and generalization ability. however, there are two challenges in track c of semeval-2024 task 1: in this paper, we used whitening techniques @xcite , which maps vectors to standard orthogonal bases, to transform the word vector representations from anisotropic to isotropic, and surprisingly, we found that whitening significantly improves the accuracy of judging semantic similarity. given the absence of labeled data in the target language, it is difficult to determine which other language would yield better prediction results when used as training data. therefore, we proposed that removing certain language categories from the training data for a specific target language contributed to improving performance. we conducted extensive experiments to demonstrate the effectiveness of the method we employed. as a result, our submitted outcomes achieved a 2nd score in spanish and a 3rd score in indonesian in track c of semeval-2024's task 1. additionally, we obtained multiple top-ten rankings in the competition. ## background the task of semantic text relatedness covers several specific subtasks, including semantic similarity, semantic matching, textual entailment, semantic relation classification, and text pair ranking. previous work has proposed various methods for these specific tasks, such as: lexical and syntactic-based methods @xcite : these methods rely on lexical and syntactic rules, such as word vector matching, lexical overlap, and syntactic tree matching. however, these methods often fail to capture higher-level semantic relationships. feature engineering-based machine learning methods @xcite : these methods involve using manually designed features, such as bag-of-words models @xcite , tf-idf weights, and syntactic features, followed by using machine learning algorithms like support vector machines and random forests for prediction. while these methods have improved performance to some extent, they still have limitations in capturing complex semantic relationships. neural network-based models: these models use neural networks to learn representations of text and capture semantic relationships between texts through training data. this includes methods that fine-tune pre-trained language models (e.g., bert (kenton and toutanova, 2019) and gpt2 @xcite etc.), as well as approaches that employ siamese networks, lstm, cnn, and other architectures for text encoding and matching. transfer learning and multi-task learning @xcite : these methods leverage knowledge from pre-trained models on related tasks to improve the performance of semantic textual relatedness tasks through transfer learning @xcite . multi-task learning combines multiple related tasks in training to enhance the model's generalization ability and effectiveness. application of external knowledge resources: researchers have also attempted to incorporate external knowledge resources such as word embeddings, semantic knowledge graphs, and multilingual data to enhance the model's understanding of semantic relationships. for cross-lingual semantic similarity tasks, mapping texts from different languages into a shared semantic space for similarity calculation is necessary. to address this, researchers have proposed various cross-lingual representation learning methods. among them, unsupervised alignment methods like unsupervised machine translation @xcite and cross-lingual pre-training models @xcite can learn the correspondences between multiple languages and map texts to a shared vector space. however, @xcite and @xcite mentioned that vector representations based on the transformer models exhibit anisotropy, which means that the vectors are unevenly distributed and clustered in a narrow coneshaped space. therefore, both bert-flow @xcite and bert-whitening @xcite aim to address the same issue, which is the anisotropy and uneven distribution of sentence embeddings. ## experimental setup we use the 12 labeled training data from @xcite as training data and the test data from track c as test data. we observe that the amount of data for each language is concentrated around 1,000, so we take 1,000 as the boundary, use oversampling to make up for less than 1,000, and use randomization to take out 1,000 for more than 1,000 to ensure that sentence pairs of different similarities are involved. in finding the training set combinations for the target languages, we compute the µ and w of whitening using the text data of the target languages in track c. we predict the training data one by one for each language, and compute the spearman coefficients using the gold labels and the predicted labels of the training data, and compare the results with the data without any whitening (i.e., the prediction result of the base model) to evaluate whether the target language enhances a certain language in the training data or not, and if it does not, it is excluded from the train data. eventually, the remaining language data is used as a training set to predict the target language. the hyperparameters are set as follows: we choose to freeze the pretrained model xlm -r base while setting the topk parameter of whitening to 256. the rubric we used was the spearman coefficient, calculated using the methodology provided by the competition officials. ## results the official competition used the spearman coefficients to evaluate the results, and as can be seen from baseline is given by @xcite . in order to further verify whether whitening works, we counted the cosine similarity distribution statistics of the data without whitening processing and after whitening. figure 2 gives two cosine similarity statistics. the left side is the cosine similarity statistics without whitening. the cosine similarity of all utterance pairs is concentrated between 0.9 and 1.0, indicating that the vector space is anisotropic. in contrast, after adding whitening, the whole distribution tends to be normal, which indicates that whitening plays a role in mapping the vectors to an isotropic space, amplifying the differences between statements. ## conclusion we use xlm -r base with whitening and propose a dataset filtering method that exploits the positive correlation of linguistic interactions, achieving good rankings in semeval-2024 task 1 track c. we verifies that whitening performs well on utterance characterization as well as str task. besides, the proposed dataset filtering method is more efficient and can alleviate the multilingual curse problem in cross-language problems to some extent. in the future, we will further study this positive correlation of language interactions, and we hope that this correlation can become more detailed, not only in terms of inter-language correlations but also in terms of the domain of the text. we also hope that this correlation can be better utilized in dataset preprocessing, not only to eliminate poorly performing languages but to further improve the combination of datasets that can be directly selected to correspond to the optimal solution.