C - STS : Conditional Semantic Textual Similarity (2023)

Abstract: Semantic textual similarity (STS) has been a cornerstone task in NLP that measures the degree of similarity between a pair of sentences, with applications in information retrieval, question answering, and embedding methods. However, it is an inherently ambiguous task, with the sentence similarity depending on the specific aspect of interest. We resolve this ambiguity by proposing a novel task called conditional STS (C-STS) which measures similarity conditioned on an aspect elucidated in natural language (hereon, condition). As an example, the similarity between the sentences “The NBA player shoots a three-pointer.” and “A man throws a tennis ball into the air to serve.” is higher for the condition “The motion of the ball.” (both upward) and lower for “The size of the ball.” (one large and one small). C-STS’s advantages are two-fold: (1) it reduces the subjectivity and ambiguity of STS, and (2) enables fine-grained similarity evaluation using diverse conditions. C-STS contains almost 20,000 instances from diverse domains and we evaluate several state-of-the-art models to demonstrate that even the most performant fine-tuning and in-context learning models (GPT-4, Flan, SimCSE) find it challenging, with Spearman correlation scores of <50. We encourage the community to evaluate their models on C-STS to provide a more holistic view of semantic similarity and natural language understanding.

## introduction over the years, natural language processing (nlp) has progressed through the co-evolution of model design (e.g. architectures, training methods) and evaluation methods for language tasks @xcite @xcite . a common task used to evaluate nlp models has been semantic textual similarity (sts) @xcite , which evaluates the models' ability to predict the similar a windsurfer skims the water with his outstretched hand. the surfer is riding a wave with a mountain in the background. the base of the object: ## methodology the c-sts task requires sentence pairs, conditions which probe different aspects of similarity, and the similarity label for a given sentence pair and condition. this section describes the technical details involved in creating the dataset. ## dataset analysis dataset statistics to ensure high-quality, faithful, and diverse annotations, we collect a total of 20,000 instances and perform quality assurance (section 5.3) resulting in a total of 18,908 instances as part of the c-sts-2023 dataset. following standard practice, we create train, validation, and test splits in a 60 : 15 : 25 ratio. we present the distribution of similarity scores, which are discrete numbers between @xcite , in figure 4 . we also measure the inter-annotator agreement on a random sample of 100 examples with three independent annotations and find fleiss' kappa score @xcite ) to be 0.61 which implies substantial interannotator agreement. average length of sentences and conditions is 12.6 and 5.3 words. qualitative analysis c-sts allows us to evaluate the generally fuzzy notion of sentence similarity with greater fidelity. we illustrate this in our diverse set of conditions provides broad support over the distribution of conditions and enables a holistic and multi-faceted evaluation of sentence similarity. for example, the conditions for the sentences on tennis in ## baselines we evaluate our dataset on several baselines which can be categorized into (1) fine-tuning baselines, which are pre-trained models finetuned on the c-sts training split, and (2) large language models (llms) baselines, which are evaluated using instructions and in-context examples. ## related work historical perspectives of semantic similarities measuring semantic similarities is a long-standing problem spanning cognitive science @xcite to psychology @xcite where early attempts are made to quantify the subjective similarity judgements with information theoretical concepts. more recently, interest in semantic similarity has gained popularity in the context of machine learning, with works in computer vision recognizing that the notion of similarity between images varies with conditions @xcite and can therefore be ambiguous @xcite . textual similarity tasks capturing textual similarity is also considered a fundamental problem in natural language processing. works such as @xcite define the textual semantic similarity tasks (sts), which is widely used in common benchmarks such as glue @xcite . extensions to the sts setting have been proposed such as making the task broader with multilinguality @xcite or incorporating relatedness @xcite . however, the loose definition of similarity has not been acknowledged as an issue explicitly. in contrast, our work tackles the ambiguity problem by collecting conditions and hence reduce subjectivity. to alleviate ambiguity, explanations play an important role in identifying the differences between the two sentences either in their syntactical structure @xcite or in natural language @xcite , but the post-hoc nature of explanations prevents it from being used prior to the similarity judgement, rendering it a supplemental component as opposed to a paradigm change in the task setup. beyond sts, works that leverage conditioning to enhance sentence representations obtain improved performance for retrieval @xcite and embedding qualities @xcite @xcite , which corroborates the observation that conditioning as a form of disambiguation benefits similarity measures. ## conclusion in this work, we propose conditional semantic textual similarity (c-sts), a novel semantic similarity assessment task that resolves the inherent ambiguity in sts. given the importance of sts and its importance in sentence representation evaluation we believe that c-sts is a timely and necessary addition to the language model evaluation landscape. rather than testing unconditional semantic similarity, the diversity of conditions in our dataset allows fine-grained evaluation. the same sentence pairs can be tested on a variety of different aspects represented by conditions, with similarities often varying significantly. c-sts poses a challenging hurdle to both encoder-only and state-of-the-art generative language models which struggle to capture the high-dimensional manifold of similarity. we believe that a combination of improved modeling and fine-tuning strategies are required to push the boundaries on c-sts and we hope that c-sts can enable innovative future work in language understanding and representation learning.