Linguistically Conditioned Semantic Textual Similarity (2024)

Abstract: Semantic textual similarity (STS) is a fundamental NLP task that measures the semantic similarity between a pair of sentences. In order to reduce the inherent ambiguity posed from the sentences, a recent work called Conditional STS (C-STS) has been proposed to measure the sentences’ similarity conditioned on a certain aspect. Despite the popularity of C-STS, we find that the current C-STS dataset suffers from various issues that could impede proper evaluation on this task. In this paper, we reannotate the C-STS validation set and observe an annotator discrepancy on 55% of the instances resulting from the annotation errors in the original label, ill-defined conditions, and the lack of clarity in the task definition. After a thorough dataset analysis, we improve the C-STS task by leveraging the models’ capability to understand the conditions under a QA task setting. With the generated answers, we present an automatic error identification pipeline that is able to identify annotation errors from the C-STS data with over 80% F1 score. We also propose a new method that largely improves the performance over baselines on the C-STS data by training the models with the answers. Finally we discuss the conditionality annotation based on the typed-feature structure (TFS) of entity types. We show in examples that the TFS is able to provide a linguistic foundation for constructing C-STS data with new conditions.

## introduction semantic textual similarity (sts) is an essential nlp task that measures the semantic similarity between two sentences @xcite . it is also a popular benchmark for developing tasks such as text embedding learning @xcite @xcite and language understanding @xcite . while the sts datasets have been developed and improved over the past years @xcite @xcite @xcite , the task itself still suffers from sentence ambiguity and subjectivity to judgment @xcite . a new task called conditional sts (c-sts) has been proposed to resolve those issues @xcite . it is designed to disambiguate the similarity between two sentences by measuring the similarity on a given condition. an accompanying dataset was also proposed to test models on the c-sts task. despite the popularity of c-sts, we observe certain limitations in the c-sts dataset that could hinder the understanding and proper evaluation of models on this task. as illustrated in figure 1 , these limitations primarily revolve around annotation errors, ill-defined conditions, and a general lack of clarity in task definition. taking into account the significance of these issues, we intend to improve the c-sts dataset by addressing the existing problems that we observed. we start by reannotating the c-sts validation set. by identifying an apparent annotation error rate of 55% foot_0 in their validation set, we analyze the provenance of the errors and discrepancies between the original and relabeled datasets. to further explore the utility of the condition and how it is understood by language models, we treat it as a question answering (qa) task and leverage large language models (llms) to generate the answer to the question that is constructed from the condition. we find that the llm-generated answers can better capture the similarity between two sentences and fit closely to our reannotated labels by having a higher spearman's correlation. based on this finding, we propose an approach to identify potential annotation errors from the c-sts dataset utilizing the llm-generated answers, achieving over 80% f1 score on the validation set. we also propose a new method to improve the c-sts task by training the models with the answers. we show that both supervised and generative models can efficiently and effectively learn the condition information encoded in the answers, improving the performance over baselines by a large margin. finally, we discuss a new annotation specification of the conditionality that aims to improve the formulation of the conditions with a more concrete semantic base. we use the entity type identified from the sentence pair as the surface condition text that is described by its underlying typed-feature structure (tfs) @xcite @xcite . we exemplify that tfs-based conditions can be successfully adopted to sentence pairs from the current c-sts dataset. we summarize the main contributions of this paper as threefold. we reannotate the c-sts validation set and propose an error identification pipeline that can be applied to the whole dataset to identify potential annotation errors and ambiguities; we propose a qa-facilitated method that largely improves the model performance on the c-sts task; we discuss using tfs as a new annotation specification to improve the conditionality in c-sts dataset with a more concrete semantic base. we make the source code and dataset publicly available. foot_1 2 related work and background semantic similarity tasks the semantic similarity between texts has long been a key issue in understanding natural language. @xcite proposed the first sts task that measures the similarity between a sentence pair. following this, @xcite proposed the second sts task that covered more text genres in the dataset. there are also sts tasks @xcite @xcite with focuses on measuring sentence similarity under a multilingual and cross-lingual conditional sts more related to our work, the c-sts task @xcite introduced conditions on top of the traditional sts tasks, and it measured the sentence similarity regarding the given condition. the c-sts dataset includes 18,908 instances. each instance contains a sentence pair, a condition, and a scalar for the similarity score on the 5-point likert scale @xcite . in this paper, we conduct the annotation and experiments on the c-sts validation set that consists of 2,834 instances. @xcite evaluated the c-sts dataset on different baselines such as simcse @xcite and gpt models @xcite by training or prompting with the sentence pairs and the conditions directly. however, our qa-based method uses the generated answers as the model input. qa-facilitated tasks question answering tasks are useful for extracting and inferring information from the texts that is relevant to the question. recent work utilized qa to improve other nlp tasks. @xcite and @xcite applied qa as an automatic evaluation metric for the summarization. @xcite used qa to improve the summarization directly. other works involved the application of qa for data augmentation @xcite and question generation @xcite . in this paper, we apply qa to generate condition-based answers for error identification and to improve models on the c-sts task. figure 2 : distribution of top 10 frequent features and entities from the conditions in the c-sts dataset. for the singleton with no explicit mention of the feature, we default the condition features from this group to type. ## reannotating c-sts we analyze the dataset and describe the annotation process for relabeling the c-sts validation set. 3 ## qa for c-sts with the consideration on scaling the relabeling task to the full c-sts dataset, we explore effective approaches to identifying potential mislabeled instances automatically. we apply qa as a pre-task for identifying information from the sentences that is relevant to the condition, and leverage llms to generate answers from condition-transformed questions. we show that the generated answers correlate better to the reannotated labels, and can be used as a reliable intermediate resource for identifying potential annotation errors from the original c-sts dataset. ## experiments we propose a new method that can improve the c-sts task by utilizing the llm-generated answers. instead of using the sentence pairs and the conditions directly as the model input, our method decomposes c-sts into two subtasks: generating answers that encode the essential semantic information about the condition, and learning the similarity score between the answer pair. we evaluate our method and compare with baseline models @xcite under both fine-tuning and prompting settings. we randomly select 70% instances from the reannotated c-sts validation set for training and the remaining 30% for testing. ## discussion: improving conditionality although the idea of c-sts is highly appreciated, we notice that the current c-sts dataset has certain issues ranging from the errors in the annotation to the lack of rigor in the condition definition. in this section, we discuss a new annotation specification of conditionality based on a word's lexical attribute value features, or typed feature structure (tfs) @xcite , and exemplify the annotation of the new conditionality on several sentence pairs from the c-sts dataset. ## conclusion in this paper, we made a comprehensive analysis and improvement to the c-sts task. with the reannotation effort on the original c-sts data, we identified and resolved annotation errors and discrepancies that could hinder the evaluation of the task. we showed that c-sts can be naturally treated as a two-step reasoning task. we applied qa to accomplish the first reasoning step by producing high-quality and correlated answers, and showed that the generated answers can be used effectively to automatically identify annotation errors and improve the c-sts task under both supervised and prompting model settings. finally, we proposed to use the typed-feature structure in c-sts to construct more semantically informed conditions. we believe that our work has led to a better execution of the c-sts task. we hope that our analysis and improvement on the c-sts can facilitate further developments by future researchers.