Hyper- CL : Conditioning Sentence Representations with Hypernetworks (2024)

Abstract: While the introduction of contrastive learning frameworks in sentence representation learning has significantly contributed to advancements in the field, it still remains unclear whether state-of-the-art sentence embeddings can capture the fine-grained semantics of sentences, particularly when conditioned on specific perspectives.In this paper, we introduce Hyper-CL, an efficient methodology that integrates hypernetworks with contrastive learning to compute conditioned sentence representations.In our proposed approach, the hypernetwork is responsible for transforming pre-computed condition embeddings into corresponding projection layers. This enables the same sentence embeddings to be projected differently according to various conditions.Evaluation on two representative conditioning benchmarks, namely conditional semantic text similarity and knowledge graph completion, demonstrates that Hyper-CL is effective in flexibly conditioning sentence representations, showcasing its computational efficiency at the same time.We also provide a comprehensive analysis of the inner workings of our approach, leading to a better interpretation of its mechanisms.

## introduction building upon the established correlation between language model performance and computational capacity @xcite , there has emerged an undeniable trend towards the adoption of everlarger language models across a diverse range of nlp applications. this trend is also evident in the computation of sentence or text representations. despite the ongoing popularity of compact encoders such as bert @xcite and roberta @xcite , there is a growing inclination to leverage the capabilities of recent, † equal contribution. * corresponding author. ## background and related work in this paper, conditioning refers to the presence of two or more signals, each represented as a natural language expression c. these signals impact the interpretation of a sentence s, highlighting a specific aspect of the sentence @xcite . here, we describe two representative tasks that involve conditioning, along with an introduction to the concept of hypernetworks. conditional semantic textual similarity (c-sts) @xcite is a task composed of four elements in one quadruplet: two sentences s 1 and s 2 , a condition c to consider when calculating the similarity between the two sentences, and a similarity score y. unlike the original semantic textual similarity (sts) dataset @xcite , c-sts computes similarity scores for the same sentence pair s 1 and s 2 under two distinct conditions c high and c low . the similarity scores of the conditioned sentences are expected to be high for c high and low for c low . therefore, a model for this task is required to compute distinct representations for the same sentence under two different viewpoints. to this end, a few basic architectures illustrated in figure 2 have been proposed by @xcite . our objective is to present a revision to the previous methods, pursuing the balance between performance and computational efficiency. knowledge graph completion (kgc) is the task focused on automatically inferring missing relationships or entities in a knowledge graph. the knowledge graph is represented as a set of triplets (h, r, t), consisting of a head entity h, a relation r, and a tail entity t. link prediction, a subtask in kgc, foot_0 aims to uncover unestablished yet plausible and novel relationships between entities @xcite . when given a head entity and a relation, the task of identifying the most suitable tail entity is known as head entity prediction. conversely, the task of determining the appropriate head entity when a tail entity and relation are provided is termed tail entity prediction. while two types of methodologies are generally available for kgc-embedding-based methods and text-based methods-our primary focus is on text-based methods that rely on the processing of textual information by language models. we further categorize them into three types: cross-encoder, encoder-decoder, and bi-encoder architectures. approaches such as kg-bert @xcite and mtl-kgc @xcite , which concatenate all triple elements (i.e., [h; r; t]), are classified as cross-encoder. methods like star @xcite and simkgc @xcite , which separately embed [h; r] and t in tail prediction tasks, are classified as bi-encoder. lastly, genkgc @xcite and kg-s2s @xcite , which directly generate tail entity text based on the remaining [h; r], are classified as encoder-decoder. on the other hand, hypernetworks refer to a type of neural network that generates the weights or parameters for another neural network, known as the primary network @xcite @xcite . in essence, a hypernetwork enables the dynamic construction of the primary network, allowing its function to adapt flexibly based on the input or condition. for instance, @xcite demonstrate that, even with a compact primary network, hypernetworks can effectively learn and apply diverse functions for various inputs, provided the hypernetwork itself is sufficiently large. in our settings, we also endeavor to harness the advantages of hypernetworks, ensuring that conditioned sentence representations are dynamically computed and adapted in response to changing conditions. ## experiments we apply hyper-cl to various embedding models (i.e., encoders) and fine-tune them on the target task, denoting the resulting models with the subscript hyper-cl. if the rank of the hypernetwork (n k ) is different from n h , we denote this value as hyperk-cl. we set k as 64 and 85, for the base and large models respectively. a detailed explanation for the selection of k can be found in the appendix b. we show the effectiveness of hyper-cl by evaluating it on two downstream tasks. ## conclusion we propose hyper-cl, a method that combines hypernetworks with contrastive learning to generate conditioned sentence representations. in two representative tasks requiring conditioning on specific perspectives, our approach successfully narrows the performance gap with the bi-encoder architecture while maintaining the time efficiency characteristic of the tri-encoder approach. we further validate the inner workings of hyper-cl by presenting intuitive analyses, such as visualizations of the embeddings projected by hyper-cl. in future work, we plan to explore a broader range of applications for hyper-cl and to investigate its refinement.