S em E val Task 1: Semantic Textual Relatedness for A frican and A sian Languages (2024)

Abstract: We present the first shared task on Semantic Textual Relatedness (STR). While earlier shared tasks primarily focused on semantic similarity, we instead investigate the broader phenomenon of semantic relatedness across 14 languages: Afrikaans, Algerian Arabic, Amharic, English, Hausa, Hindi, Indonesian, Kinyarwanda, Marathi, Moroccan Arabic, Modern Standard Arabic, Punjabi, Spanish, and Telugu. These languages originate from five distinct language families and are predominantly spoken in Africa and Asia – regions characterised by the relatively limited availability of NLP resources. Each instance in the datasets is a sentence pair associated with a score that represents the degree of semantic textual relatedness between the two sentences. Participating systems were asked to rank sentence pairs by their closeness in meaning (i.e., their degree of semantic relatedness) in the 14 languages in three main tracks: (a) supervised, (b) unsupervised, and (c) crosslingual. The task attracted 163 participants. We received 70 submissions in total (across all tasks) from 51 different teams, and 38 system description papers. We report on the best-performing systems as well as the most common and the most effective approaches for the three different tracks.

## introduction defining the relationship between two units of text is an important component of constructing text representations. within this context, semantic textual relatedness (str) aims to capture the degree to which two linguistic units (e.g., words or sentences, etc.) are close in meaning @xcite . two units may be related in a variety of different ways (e.g., by expressing the same view, originating from the same time period, elaborating on each other, etc.). on the other hand, semantic textual similarity (sts) considers only a narrow view of the relationship that may exist between texts (such as equivalence or paraphrase) which does not incorporate other dimensions of relatedness such as entailment, topic or view similarity, or temporal relations @xcite . for example, 'i am feeling sick.' and 'get well soon!' would receive a low similarity score, despite the two being very related. in this shared task, we investigate the broader concept of semantic textual relatedness. str is central to understanding meaning in text @xcite @xcite and its automation can benefit various downstream tasks such as evaluating sentence representation methods, question answering, and summarisation @xcite . prior shared tasks @xcite @xcite @xcite have mainly focused on textual similarity. in this work, we provide participants with semrel @xcite , a collection of 14 newly curated monolingual str datasets for afrikaans (afr), amharic (amh), modern standard arabic (arb), algerian arabic (arq), moroccan arabic (ary), english (eng), spanish (esp), hausa (hau), hindi (hin), indonesian (ind), kinyarwanda (kin), marathi (mar), punjabi (pun) and telugu (tel). the datasets are composed of sentence pairs, each assigned a relatedness score between 0 (completely unrelated) and 1 (maximally related) with a large range of expected relatedness values. the pairs of sentences were first selected from pre-existing datasets covering various topics and formality levels, e.g., news data, wikipedia, and conversational data. to generate the relatedness scores, the sentence pairs were then annotated by native speakers who performed comparisons between different pairs of sentences using best-worst scaling (bws) @xcite . the shared task included three main tracks: (1) supervised, (2) unsupervised, and (3) cross-lingual. each team could provide submissions for one, two, or all of the tracks in one or more languages. our official evaluation metric was the spearman rank correlation coefficient, which captures how well the system-predicted rankings of test instances aligned with human judgments. our task attracted 163 participants, received 70 final submissions from 51 different teams, and 38 teams submitted system description papers. track a (supervised) received the largest number of submissions: 40, followed by 18 submissions for track b (unsupervised) and 12 for track c (crosslingual). most teams participated in multiple languages (more than eight on average). all of the task details and resources are available on the task website. 1 ## related work the field of semantic textual relatedness in natural language processing covers a variety of approaches and techniques designed to measure the 1 https://semantic-textual-relatedness.github.io closeness in meaning between units of text, specifically words @xcite or sentences @xcite . most prior shared tasks focus on semantic textual similarity, a narrower subset of relatedness, and often only cover high-resource languages such as english @xcite @xcite @xcite , arabic, german, spanish, and turkish @xcite with few exceptions such as armendariz et al. (2020) who also included slovene, finnish, and croatian. by comparison, this shared task focuses on sentence-level str in various low-resource languages. to our knowledge, the only corpora specially designed for semantic textual relatedness between pairs of sentences was created by @xcite for english. the core of @xcite approach served as the model for data annotations added to new ways of data collectioncuration for several less-resourced languages. ## task description in this task, we aim to predict the semantic textual relatedness (str) of sentence pairs. participants had to rank sentence pairs by their degree of semantic relatedness which varies between 0 (unrelated) and 1 (closely related). each team could provide submissions for one, two, or all of the tracks presented below. ## discussion we observe that in general, teams opt out of pretrained models, and in most cases, the methods do not perform equally well across languages. hence, for a given track, performing well in a language does not mean performing equally well in another language. further, the results show that good scores are not only related to low vs. high-resourcedness. for instance, in tracks b and c, results for modern standard arabic (arb), which is considered high resource, are sometimes worse than those for low resource languages such as amharic (amh) and kinyarwanda (kin). interestingly, although the participating teams rarely use language-specific features, such approaches lead to good and interpretable results, as reported by e.g., team mainlp, who leveraged information about language families in track c. we also note that for track c, using a simple labse baseline can achieve results that are better or comparable to more sophisticated techniques (see @xcite for language-specific baseline results). ## conclusion we presented the first shared task on semantic relatedness, covering three tracks and 14 languages in total. the submitted systems were ranked based on the ranking of their predicted relatedness scores compared to the gold labels. we summarised the reported results, the bestperforming methods, and the most effective, promising, and original ones. overall, our findings on sentence representation techniques vary across the different languages and show that determining semantic textual relatedness is not a trivial task.