SEAVER : Attention Reallocation for Mitigating Distractions in Language Models for Conditional Semantic Textual Similarity Measurement (2024)

Abstract: Conditional Semantic Textual Similarity (C-STS) introduces specific limiting conditions to the traditional Semantic Textual Similarity (STS) task, posing challenges for STS models. Language models employing cross-encoding demonstrate satisfactory performance in STS, yet their effectiveness significantly diminishes in C-STS. In this work, we argue that the failure is due to the fact that the redundant information in the text distracts language models from the required condition-relevant information. To alleviate this, we propose Self-Augmentation via Self-Reweighting (SEAVER), which, based solely on models’ internal attention and without the need for external auxiliary information, adaptively reallocates the model’s attention weights by emphasizing the importance of condition-relevant tokens. On the C-STS-2023 test set, SEAVER consistently improves performance of all million-scale fine-tuning baseline models (up to around 3 points), and even surpasses performance of billion-scale few-shot prompted large language models (such as GPT-4). Our code is available at https://github.com/BaixuanLi/SEAVER.

## introduction semantic textual similarity (sts) has been a cornerstone task in natural language processing fields for years @xcite @xcite @xcite , which aims to measure the semantic similarity between two sentences. with the emergence of pre-trained language models @xcite @xcite , the sts task seems to have been almost solved. however, sts is an inherently ambiguous task @xcite , for the varying aspects that can influence sentence similarity, unconditionally measuring this similarity is irrational and unexplainable. to solve the ambiguity of sts task itself, @xcite ## related work excessive features extracted by language models. there is substantial evidence indicating that throughout the pre-training, language models learn not only contextualized text representations, but also a grasp of grammar @xcite , syntax @xcite , even commonsense @xcite and world knowledge @xcite . however, the semantic information mentioned above is general-purpose and unconditional. thus, for c-sts, which emphasize the conditional constraints on sentences and focus on more finegrained aspects, the excessive information can, in turn, distract the language model's attention. ## method this section starts with self-reweighting, which directly extracts condition-sentence cross-attention submatrices to reweight the outputs (section 3.1), then we use the reweighted outputs to enhance the original outputs in a specific proportion (section 3.2), namely self-augmentation. ## experiments dataset. in this study, we employ c-sts-2023 dataset collected by @xcite for training and testing, which consists of quadruples, formatted as (sentence1, sentence2, condition, label). in which label represents the level of similarity between sentence1 and sentence2 under condition, converted into a likert scale @xcite with values ranging from 1 to 5, which is common with semantic textual similarity tasks @xcite . experimental setup. we conduct a comparative analysis between various baselines and our proposed seaver, which can be categorized into: ## conclusion in this work, we argue that the reason for the subpar performance of language models in c-sts is attributed to the dilution effect: the excessive general-purpose but condition-irrelevant features distract language models' attention from the specific, condition-relevant features that occupy a relatively small proportion in the sentence. however, mitigating this distraction through mere finetuning is challenging. to address this, we propose seaver, which reallocates the model's attention weights based on specific conditions using its internal information. on the c-sts-2023 test set, our method outperforms all types of baseline models.