Varying Sentence Representations via Condition-Specified Routers (2024)

Abstract: Semantic similarity between two sentences is inherently subjective and can vary significantly based on the specific aspects emphasized. Consequently, traditional sentence encoders must be capable of generating conditioned sentence representations that account for diverse conditions or aspects. In this paper, we propose a novel yet efficient framework based on transformer-style language models that facilitates advanced conditioned sentence representation while maintaining model parameters and computational efficiency. Empirical evaluations on the Conditional Semantic Textual Similarity and Knowledge Graph Completion tasks demonstrate the superiority of our proposed framework.

## introduction sentence textual similarity (sts) @xcite , which predicts the semantic similarity between two sentences, has long been a prevalent and crucial task for evaluating the quality of sentence representations and the semantic understanding capabilities of natural language processing (nlp) models. however, traditional sts evaluation methods focus on the overall meaning of entire sentences, thereby neglecting the fine-grained semantics related to different aspects. to address this limitation, @xcite introduced conditional sts (c-sts), which measures the similarity between two sentences within the context of given condition sentences. for instance, the two sentences "a hotel room decorated in silver and white has a large mirror over the headboard of the bed." and "a small bedroom suite in a hotel setting with a bed, small table, and two chairs." are considered similar given the condition "the name of the place" since both sentences describe rooms located in a hotel. conversely, these sentences are considered dissimilar given the condition "the number of chairs" as the first sentence does not mention any chairs, whereas the second sentence explicitly mentions two chairs. solving the c-sts task necessitates that the sentence embedding model be capable of generating distinct representations for the same sentence under different conditions. previous studies @xcite have introduced three architectures designed to incorporate the influence of conditions into sentence embeddings: cross-encoder, bi-encoder, and tri-encoder, as depicted in fig. 1 . as mentioned in @xcite , the cross-encoder architecture concatenates each sentence pair (additional with each condition in c-sts) to calculate similarity scores between every two sentences, leading to computational inefficiency. similarly, the bi-encoder in the c-sts task concatenates each sentence with each condition, taking the limitation that the representation must be generated for every unique combination of sentences and conditions @xcite . in contrast, the tri-encoder employs a more computationally efficient method: it precomputes embeddings of sentences and conditions independently and then uses a separate composition function to merge the semantics of the sentence and the condition. however, @xcite utilize the hadamard product or concatenation operations on pre-computed sentence and condition embeddings to derive conditioned sentence representations for similarity calculation, observing a significant performance gap compared to the bi-encoder architecture. yoo et al. ( 2024 ) enhanced the tri-encoder architecture by integrating hypernetworks @xcite to project the original sentence embeddings into a specific condition subspace, thereby reducing the performance disparity with the bi-encoder architecture. nevertheless, the use of hypernetworks introduces an external parameter set three times larger than the backbone model (simcse @xcite ), which is necessary to achieve their improved performance. more related work can be found in appendix d. in this paper, we propose a conditioned sentence representation (csr) method based on the tri-encoder architecture, with the goal of enhancing its performance without introducing external parameters while maintaining computational efficiency. unlike tri-encoder models in @xcite , which focus on merging the semantics of the sentence and condition, our approach posits that the condition semantics ought to play the role of influencing which tokens in the sentence should contribute to the final conditionspecific sentence embedding. drawing inspiration from the router and heavy-light attention mechanism in @xcite , we design a conditionrelevant router that calculates scores for each token in the sentence, selecting the most relevant tokens that reflect the semantics emphasized by a given condition. then, the relevant tokens additionally perform "heavy attention", while the remaining tokens perform "light attention" by multiplying the normalized router scores. this approach obtains different score distributions for a sentence based on different conditions, thereby generating varied conditioned sentence representations. we evaluate our method on the c-sts task and knowledge graph completion (kgc) task, demon-strating significant improvement over previous tri-encoder frameworks @xcite while maintaining memory and computational efficiency. ## framework traditionally, the input for sentence embedding models (considering transformer-style language models in this paper) consists solely of the text itself (e.g., sentences, paragraphs) and is expected to output a vectorized representation of the input sentence. conditioned sentence representation, however, dynamically modifies the sentence embedding based on the influence of another text serving as a condition, thereby reflecting the conditioned sentence information. motivation given a sentence set @xmath0, obtaining every possible conditioned sentence representation requires |s| × |c| forward encoding processes in the bi-encoder architecture. in contrast, the tri-encoder architecture requires only |s| + |c| forward encoding processes, with an additional |s| × |c| lightweight composition computation (e.g., hadamard product). while the tri-encoder is more computationally efficient, it suffers a performance drop due to the lack of direct interaction between sentences and conditions within the encoders as discussed in section 1. our motivation is to improve the trade-off between performance and computational efficiency, effectively integrating the advantages of both architectures to enhance the tri-encoder's performance while maintaining efficient computation. method in this study, we employ a transformerstyle encoder f as the backbone of our csr framework, depicted in figure 1 . we provide a detailed description of the process to derive the conditioned representation for a sentence s. our csr initially obtains the embedding of a given condition c using an encoder f . unlike prior approaches, we do not utilize the final [cls] embedding as the representation of the condition. instead, we extract the query vector q @xmath1. the sentence s is input into the same encoder f . however, starting from the t-th layer of the encoder, a router operation is introduced. for instance, considering the t-th layer, the query vector q c of condition c queries each token of sentence s to obtain the router score for each token: from the t-th layer to the final t -th layer of encoder f , each layer's hidden state is affected by the condition-specified router. finally, following the approach of @xcite , we use an average pooling layer to obtain the conditioned sentence representation r s c from the hidden states output by the last layer of the encoder. to ensure computational efficiency, we only incorporate routers into the final few layers, i.e., @xmath2. empirical evidence indicates that integrating a router exclusively in the @xmath3. refer to section 3.3 for more details. we emphasize the advantages of our framework: £ our framework does not introduce any additional parameters. all vectors q c and k s utilized in the router mechanism are derived directly from the original backbone encoder f . £ the router score calculation is lightweight. and we directly enhance the representation of those selected condition-relevant tokens (with higher router scores) by multiplying the normalized router scores, promising simple, efficient, yet effective framework. training given a training sample (s 1 , s 2 , c, y * ), where y * represents the ground truth similarity score between sentence s 1 and sentence s 2 under the condition c, we adopt the mean squared error (mse) objective @xcite 1 ). we also demonstrate the effectiveness of our router mechanism in bi-encoder. all results are averaged on three runs. we do not consider cross-encoder due to its computational inefficiency and subpar performance (43.2% in @xcite and 43.8% in anonymous (2024)) compared to bi-encoder. ## conclusion this paper introduces a novel conditioned sentence representation method with a router mechanism to achieve substantial performance gains of tri-encoder architecture in c-sts task and kgc task without introducing any additional parameters. furthermore, it maintains computational efficiency, highlighting simplicity, efficiency, and effectiveness. yizhong wang, swaroop mishra, pegah alipoormolabashi, yeganeh kordi, amirreza mirzaei, anjana arunkumar, arjun ashok, arut selvan dhanasekaran, atharva naik, david stap, et al. 2022b. super-naturalinstructions: generalization via declarative instructions on 1600+ nlp tasks. in 2022 conference on empirical methods in natural language processing, emnlp 2022. zhou wang and alan c bovik. 2009. mean squared error: love it or leave it? a new look at signal fidelity measures. ieee signal processing magazine, 26(1):98-117. young hyun yoo, jii cha, changhyeon kim, and taeuk kim. 2024. hyper-cl: conditioning sentence representations with hypernetworks. preprint, arxiv:2403.09490.