PEAR at S em E val-2024 Task 1: Pair Encoding with Augmented Re-sampling for Semantic Textual Relatedness (2024)

Abstract: This paper describes a system submitted to the supervised track (Track A) at SemEval-24: Semantic Textual Relatedness for African and Asian Languages. Challenged with datasets of varying sizes, some as small as 800 samples, we observe that the PEAR system, using smaller pre-trained masked language models to process sentence pairs (Pair Encoding), results in models that efficiently adapt to the task.In addition to the simplistic modeling approach, we experiment with hyperparameter optimization and data expansion from the provided training sets using multilingual bi-encoders, sampling a dynamic number of nearest neighbors (Augmented Re-sampling). The final models are lightweight, allowing fast experimentation and integration of new languages.

## introduction the overall aim of the semantic textual relatedness (str) shared task @xcite is to correctly predict the relatedness between a given sentence pair on a scale from 0 to 1, described as closeness in meaning @xcite , exemplified by expressing the same views and one elaborating on the other. this shared task covers a broader aspect of the well-established semantic textual similarity (sts) field, which fails to address the intuitive relatedness between two sentences. from available sts data, such as from the semeval-2012 task on similarity @xcite , the sentences "a man is peeling a banana" and "a woman is peeling a potato" receive a normalized similarity of 0.3. in contrast, the two descriptions have a higher degree of relatedness, where something is being peeled. relatedness tends to focus less on equivalence and paraphrasing and more on the broader case of entailment and the cause-effect relationship between two sentences. the task consists of three tracks: a (supervised), b (unsupervised), and c (cross-lingual). the system described here will only consider track a, allowing the use of any training data. refer to @xcite for more details. the system and constraints this paper proposes a system for any language with an available pre-trained masked language model (mlm), such as bert or roberta, used to process pairs of sentences with full cross-attention. the constraint of using limited-size mlms was set early in the project to study their performance compared to the impressive baselines observed through existing multilingual bi-encoders. however, following ideas of @xcite , the addition of weakly supervised labels from such bi-encoders was added as an optional step to inspect its impact on smaller datasets. being unfamiliar with most of the involved languages and thus being unable to verify the results, no language-specific rules were implemented. consequently, no text manipulation (such as paraphrasing and replacing words), back-translation, or normalization steps were applied. while the task organizers permitted the use of any available data for the supervised track, in addition to large language models to a limited extent, the presented approach only uses the supplied training dataset per language. while performance suffers in some cases, we hope that the aforementioned constraints help to support as many future languages as possible with little to no modification. continuing the idea of supporting lower-resourced languages, this system only uses base size transformer mlms, ranging from 110m to 125m parameters. all code is available on github. foot_0 ## data the full dataset for semrel consists of 14 languages. however, only 9 of the 14 languages are included for track a and have labeled relatedness scores between 0 and 1. ## related work semantic textual relatedness (str), in the context of language modeling and prediction, has consid-2 an roc-auc score of 0.5 indicates that a model cannot differ between samples in the provided data sources. erably less research connected to it than semantic textual similarity (sts), which has several datasets and evaluation benchmarks openly available @xcite , many of which tied to the sts task within semeval @xcite . the included data is mainly monolingual english, but more recent additions added limited multilingual and cross-lingual tasks @xcite . the first str dataset was introduced by @xcite , including a monolingual dataset of 5,500 english sentence pairs. new for this task is the inclusion of several low-resource languages not yet studied at the sentence level. the field of natural language processing has drastically changed since the release of the majority of the datasets and shared tasks for semantic textual similarity, where the top-scoring methods typically included a significant amount of feature engineering based on methods like n-gram overlaps, edit distance, and longest common substrings, word alignments, and more, applied to both regression and deep learning models @xcite . additionally, knowledge-informed systems included semantic information with word-net and word frequency corpora @xcite . applying the same efforts to new languages would require significant work, such as collecting new corpora. sentence embeddings modeling similarity between sentences is commonly associated with sentence embedding models, some of which include 2 more than a billion gathered training pairs @xcite . while applicable across many languages and domains, with models initialized from the xlm-roberta models @xcite , the included languages do not cover many of which are part of semrel 2024. encoding sentence pairs this paper focuses on sentence-pair modeling, encoding the sentences with existing pre-trained language models to create a simpler model that allows fast and easy implementation for any language. this modeling scheme, referred to as cross-encoders, indicating full (cross) self-attention over the entire context, is well explained in previous work by @xcite @xcite . furthermore, cross-encoders have succeeded in supervised and unsupervised applications @xcite . in addition to the sentence scoring, we follow the work by @xcite to augment data with a bi-encoder, although on much smaller datasets, where the original work was carried out on data up to millions of samples. for details on bi-encoders and sentence embedding models, refer to the excellent implementations by @xcite @xcite . the baseline provided by the task organizers is labse, a dual-encoder bert-based sentence embedding model @xcite . ## system overview after restructuring the provided datasets into sentence pairs with their respective labels, they are passed to a mlm with an added regression head; using a sigmoid layer on top of the pooled output, the model is trained using a single-class binary cross-entropy loss, with mean reduction: the models ( augmented re-sampling in a separate module, a bi-encoder (multilingual-e5-base) is employed to find the closest non-existing sentence pairs in the data by creating sentence embeddings and searching for nearest k neighbors with cosine similarity. before initializing the bi-encoder, the cross-encoder is trained for e weak epochs before predicting weak labels for the augmented pairs (s i , s j , pred i,j ), which are added to the training data. k determines the number of nearest neighbors to retrieve for each source sentence. a figure outlining the weak supervision pipeline is found in figure 3 . figure 3 : the weak supervision pipeline modularity a big focus in the development was to keep it as modular as possible. models, parameters, data selection, and more are easily controlled through passed arguments. furthermore, the crossencoder is provided as a standalone module with varying levels of abstraction, e.g., calling fit directly or through a provided training pipeline, including optional weak supervision labeling. ## experimental setup all experiments and evaluations use the official train/dev/test data splits where applicable, and scores are presented by the spearman rank correlation coefficient multiplied by 100. in the development phase, we studied the effect of combining or using only per-language data, working as an initial baseline before dev-and test labels were released. this was done by 5-fold validation. as stated in section 1, no text manipulation or preprocessing was done to keep evaluations fair across languages. moreover, upon manual inspection, the data seemed sufficiently preprocessed. the following definitions will be used to differ between model configurations: 3 1407 ## results results from k-fold validation to quantify the differences between combining training sets vs. training per language show that combining data has a clear benefit. see ## conclusion after testing the capabilities of commonly used models for masked language modeling and sentence embeddings, we find mlms efficient at distinguishing relatedness with little training data. although attempts at optimizing parameters for indistribution data resulted in little to no performance gains, there are likely better-suited augmentation strategies for further improving performance with as little source data as possible. as a closing remark, we hope that the provided system may serve as a valuable tool for future developments in semantic textual relatedness.