S em R el2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages (2024)

Abstract: Exploring and quantifying semantic relatedness is central to representing language and holds significant implications across various NLP tasks. While earlier NLP research primarily focused on semantic similarity, often within the English language context, we instead investigate the broader phenomenon of semantic relatedness. In this paper, we present SemRel , a new semantic relatedness dataset collection annotated by native speakers across 13 languages: Afrikaans, Algerian Arabic, Amharic, English, Hausa, Hindi, Indonesian, Kinyarwanda, Marathi, Moroccan Arabic, Modern Standard Arabic, Spanish, and Telugu . These languages originate from five distinct language families and are predominantly spoken in Africa and Asia – regions characterised by a relatively limited availability of NLP resources. Each instance in the SemRel datasets is a sentence pair associated with a score that represents the degree of semantic textual relatedness between the two sentences. The scores are obtained using a comparative annotation framework. We describe the data collection and annotation processes, challenges when building the datasets, baseline experiments, and their impact and utility in NLP.

## introduction characterising the relationship between two units of text is an important component of constructing text representations. within this context, semantic textual relatedness (str) aims to capture the degree to which two linguistic units (e.g., words or sentences, etc.) are close in meaning @xcite semantic relatedness dataset language families 2008; @xcite . two units may be related in a variety of different ways (e.g., by expressing the same view, originating from the same time period, elaborating on each other, etc.). on the other hand, semantic textual similarity (sts) considers only a narrow view of the relationship that may exist between texts (such as equivalence or paraphrase) which does not incorporate other dimensions of relatedness such as entailment, topic or view similarity, or temporal relations @xcite . for example, 'i caught a cold.' and 'i hope you feel better soon.' would receive a low similarity score, despite the two being very related. in this work, we investigate the broader concept of semantic textual relatedness. str is central to understanding meaning in text @xcite @xcite and its automation can benefit various downstream tasks such as evaluating sentence representation methods, question answering, and summarisation @xcite . the datasets are composed of sentence pairs, each assigned a relatedness score between 0 (completely unrelated) and 1 (maximally related). with the aim of curating diverse str datasets, the pairs of sentences were first selected from pre-existing datasets covering various topics and formality levels, e.g., news data, wikipedia, and conversational data. additionally, we selected pairs with a large range of expected relatedness values by considering lexical overlap, contiguity, topic coverage, and random pairings. to generate the relatedness scores, the sentence pairs were then annotated by native speakers who performed comparisons between different pairs of sentences using best-worst scaling (bws) @xcite . bws is known to avoid common limitations of traditional rating scale annotation methods @xcite . the annotation process led to the high reliability of the final relatedness rankings in the different semrel datasets. our main contributions are as follows: to promote research in the field of semantic relatedness, we publicly released the semrel2024 datasets as part of a shared task that attracted a large number of participants interested in low-resource languages. foot_0 ## related work the field of semantic relatedness in natural language processing covers a variety of approaches and techniques designed to measure the closeness in meaning between units, specifically words @xcite , or sentences @xcite . most prior work focuses on sts, a narrower subset of str, and often only covers high-resource languages such as english @xcite @xcite @xcite , arabic, german, spanish, turkish @xcite , and italian @xcite with the only exception being finnish, slovene, croatian @xcite and farsi @xcite . to overcome the scarcity of available resources, @xcite proposed sentence-level, encoder-based methods leveraging english data to create arabic, spanish, thai, and indonesian datasets, whereas @xcite use traditional data augmentation methods to create bangla data. by comparison, this work is focused on the creation of resources for sentence-level str in multiple low-resource languages. here, the few works which exist for non-high-resource languages are at the word level (e.g., @xcite for korean). to our knowledge, the only corpora specially designed for semantic textual relatedness between pairs of sentences was created by @xcite for english. @xcite curated a dataset of 5,500 english sentence pairs annotated using a comparative annotation framework. their dataset has since been used to evaluate embedding approaches @xcite and other methods @xcite . the core of @xcite approach serves as the model for data annotations in this project. however, our work additionally explores new ways of data collection-curation, and several challenges had to be addressed when working with less-resourced languages. ## conclusion we presented semrel, a new collection of semantic textual relatedness datasets in 13 languages with the majority predominantly spoken in africa and asia and considered low-resource. the sentence pairs contained in the datasets are annotated by native speakers and are associated with fine-grained relatedness scores. we reported the details related to the data curation and emphasised the challenges faced when dealing with low-resource languages. we publicly release the datasets as well as other resources, such as the annotation guidelines and full labels for the research community interested in semantic relatedness, low-resource languages, and disagreements.