What Makes Sentences Semantically Related? A Textual Relatedness Dataset and Empirical Study (2023)

Abstract: The degree of semantic relatedness of two units of language has long been considered fundamental to understanding meaning. Additionally, automatically determining relatedness has many applications such as question answering and summarization. However, prior NLP work has largely focused on semantic similarity, a subset of relatedness, because of a lack of relatedness datasets. In this paper, we introduce a dataset for Semantic Textual Relatedness, STR-2022, that has 5,500 English sentence pairs manually annotated using a comparative annotation framework, resulting in fine-grained scores. We show that human intuition regarding relatedness of sentence pairs is highly reliable, with a repeat annotation correlation of 0.84. We use the dataset to explore questions on what makes sentences semantically related. We also show the utility of STR-2022 for evaluating automatic methods of sentence representation and for various downstream NLP tasks. Our dataset, data statement, and annotation questionnaire can be found at: https://doi.org/10.5281/zenodo.7599667 .

## introduction the semantic relatedness of two units of language is the degree to which they are close in terms of their meaning @xcite . the linguistic units can be words, phrases, sentences, etc. though our intuition of semantic relatedness is dependent on many factors such as the context of assessment, age, and socioeconomic status @xcite , it is argued that a consensus can usually be reached for many pairs @xcite . consider the two sentence pairs in the semantic relatedness of two units of language has long been considered fundamental to understanding meaning @xcite ; given how difficult it has been to define meaning, a natural approach to get at the meaning of a unit is to determine how close it is to other units. thus, unsurprisingly, automatically determining relatedness has many applications such as question answering, text generation, and summarization (more discussion in §7). however, prior nlp work has focused on semantic similarity (a small subset of semantic relatedness), largely because of a dearth of datasets on relatedness. the few relatedness datasets that exist are only for word pairs @xcite or phrase pairs @xcite . further, most existing datasets were annotated, one item at a time, using coarse rating labels such as integer values between 1 and 5 representing coarse degrees of closeness. it is well documented that such approaches suffer from inter-and intra-annotator inconsistency, scale region bias, and issues arising due to the fixed granularity @xcite . further, the notions of related and unrelated have fuzzy boundaries. different people may have different intuitions of where such a boundary exists. finally, for some tasks, it is more appropriate to train on a dataset of relatedness than similarity. ( §2.1 discusses how relatedness and similarity are different.) in this paper, we present the first manually annotated dataset of sentence-sentence semantic re-latedness. it includes fine-grained scores of relatedness from 0 (least related) to 1 (most related) for 5,500 english sentence pairs. the sentences are taken from diverse sources and thus also have diverse sentence structures, varying amounts of lexical overlap, and varying formality. the relatedness scores were obtained using a comparative annotation schema: two (or more) items are presented together and the annotator has to determine which is greater with respect to the metric of interest. since annotators are making relative judgments, the limitations discussed earlier for rating scales are greatly mitigated. importantly, such annotations do not rely on arbitrary boundaries between arbitrary categories such as "strongly related" and "somewhat related". we use the relatedness dataset to explore: 1. to what extent do speakers of english intuitively agree on the relatedness of sentences? ( §4) 2. what makes two sentences more related? ( §5) 3. how well do existing approaches of sentence representation capture semantic relatedness (by placing related sentence pairs closer to each other in vector space)? ( §6) 4. how can an improved annotation schema to capture relatedness benefit other nlp tasks? ( §7) we refer to our dataset as str-2022, and the task of predicting relatedness between sentences as the semantic textual relatedness (str) task. data, data statement, and annotation questionnaire are made available foot_0 . the three subsections below discuss key ideas from past work on annotating relatedness and similarity, existing datasets, and comparative annotation, respectively. notably, each of these subsections also discusses how relevant past work has influenced our approach to data annotation. ## creating str-2022 dataset creation included several steps: curating sentence pairs for annotation, designing the questionnaire, crowdsourcing annotations, and aggregating the annotations to obtain relatedness scores. ## reliability of annotations for annotations producing real-valued scores, a commonly used measure of quality and reliability is split-half reliability (shr) @xcite . shr is a measure of the degree to which repeating the annotations would result in similar relative rankings of the items. to measure shr, annotations for each 4-tuple are split into two bins. the annotations for each bin are used to produce two different independent relatedness scores. next, the spearman correlation between the two sets of scores is calculated-a measure of the closeness of the two rankings. if the annotations are reliable then there should be a high correlation. this process is repeated 1000 times and the correlation scores are averaged. as shown in it also validates our approach as a way to produce high-quality relatedness datasets; which, in turn, can be used to study the mechanisms underpinning relatedness (as we explore in the next section). ## what makes sentences more semantically related? the availability of a dataset with human notions of semantic relatedness allows one to explore fundamental aspects of meaning: for example, what makes two sentences more related? in this section, we examine some basic questions. on average, to what extent is the semantic relatedness of a sentence pair impacted by presence of: ## evaluating sentence representation models using str-2022 since str-2022 captures a wide range of finegrained relations that exist between sentences, it is a valuable asset in evaluating sentence representation and embedding models. essentially, predicting semantic relatedness is treated as a regression task, where first, using various unsupervised and supervised approaches described in the two sub-sections below, we represent each sentence as a vector. we use the cosine similarity between the vectors as a prediction of their semantic relatedness. we use the spearman correlation between the prediction and gold relatedness scores to measure the goodness of the relatedness predictions (and in turn of the sentence representation). the experiments below (unless otherwise specified) all involve 5-fold cross-validation (cv) on str-2022. we report the average of the spearman correlations across the folds. note that even for models that do not require training (e.g., dice score), to enable direct comparisons with trained methods, we evaluate their performance on each test fold independently and report the average of the correlations across folds. we first explore unsupervised approaches to sentence representation where the embedding of a sentence is derived from that of its constituent tokens. the token embedding can be of two types: we now evaluate the performance of bert-based models on str-2022 when formulated as a supervised regression task. we use the s-bert cross-encoder framework of @xcite , and apply mean-pooling on top of the token embeddings of the final layer to obtain sentence embeddings. the model is trained using a cosine-similarity loss-the cosine between the embeddings of a sentence pair is compared to the gold semantic relatedness scores to obtain the mean squared error (mse) loss for each datapoint. ## conclusion we created str-2022, the first dataset of english sentence pairs annotated with fine-grained relatedness scores. we used a comparative annotation method that produced a split-half reliability of 0.84. thus, we showed that speakers of a language can reliably judge semantic relatedness. we used the dataset to explore several research questions pertaining to what makes two sentences more related. finally, we used str-2022 to evaluate the ability of sentence representation methods to embed sentences in vector spaces such that those that are closer to each other in meaning are also closer in the vector space. the dataset is made freely available; facilitating further research in semantic relatedness and sentence representation.