Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operations (2023)

Abstract: Traditional sentence embedding models encode sentences into vector representations to capture useful properties such as the semantic similarity between sentences. However, in addition to similarity, sentence semantics can also be interpreted via compositional operations such as sentence fusion or difference. It is unclear whether the compositional semantics of sentences can be directly reflected as compositional operations in the embedding space. To more effectively bridge the continuous embedding and discrete text spaces, we explore the plausibility of incorporating various compositional properties into the sentence embedding space that allows us to interpret embedding transformations as compositional sentence operations. We propose InterSent, an end-to-end framework for learning interpretable sentence embeddings that supports compositional sentence operations in the embedding space. Our method optimizes operator networks and a bottleneck encoder-decoder model to produce meaningful and interpretable sentence embeddings. Experimental results demonstrate that our method significantly improves the interpretability of sentence embeddings on four textual generation tasks over existing approaches while maintaining strong performance on traditional semantic similarity tasks.

## introduction learning universal sentence embeddings is crucial to a wide range of nlp problems, as they can provide an out-of-the-box solution for various important tasks, such as semantic retrieval @xcite , clustering @xcite , and question answering @xcite . recently, contrastive learning has been shown to be an effective training paradigm for learning sentence embeddings @xcite @xcite . these methods optimize the sentence representation space such that the distance between embeddings reflects the semantic similarity of sentences. while the similarity structure of sentence embedding models is an important aspect of the intersentence relationship, contrastive learning methods do not provide a direct way of interpreting the information encoded in the sentence embedding. despite the existence of probes for individual linguistic properties @xcite , it is still unclear whether the embedding fully captures the semantics of the original sentence necessary for reconstruction. moreover, sentence semantics not only can be interpreted by their similarity but also via sentence operations such as fusion, difference and compression. while these operations of sentence semantics have been previously studied individually as sequence-to-sequence generation tasks @xcite @xcite , it remains an open research question whether these operations can be directly captured as operations in the sentence embedding space. we argue that the ability to interpret and manipulate the encoded semantics is an important aspect of interpretable sentence embeddings, which can bridge the continuous embedding space and the discrete text space. particularly, this ability benefits concrete tasks such as multi-hop search and reasoning @xcite , instruction following @xcite , compositional generation @xcite , and summarization @xcite . in this work, we propose intersent, an endto-end framework for learning interpretable and effective sentence embeddings that supports compositional sentence operations. our method combines both generative and contrastive objectives to learn a well-structured embedding space that satisfies useful properties for both utility and interpretability. specifically, together with an encoder-decoder model, we train several small operator networks on output sentence sentence encoder sentence decoder sentence a sentence b target sentence target sentence & operation operator sentence embedding space text space text space output embedding target embedding embedding a embedding b contrastive loss generative loss figure 1: overview of our proposed framework intersent for learning interpretable sentence embeddings. given a pair of input sentences (or a single input sentence for compression and reconstruction), intersent encodes the input sentence(s) into sentence embedding (s), which are fed into an operator network to compute the output embedding for decoding. during training, intersent ensures alignment between the output and target in both the embedding space and text space via contrastive and generative objectives respectively. ## method fig. 1 illustrates the overall architecture of intersent. our method, intersent, optimizes both a contrastive objective (in the continuous space) and a generative objective (in the discrete text space) jointly during training. in this section, we define the notion of interpretability for a sen-tence embedding space and provide a detailed description of our proposed framework intersent. ## analysis to provide a better understanding of intersent, we investigate how intersent handles longer text, and present an ablation study on the effect of individual loss functions, and choice of pretrained language models for encoders and decoders. then, we analyze the operator functions learned by intersent through a detailed case study. ## conclusion in this work, we present intersent, an endto-end framework for learning interpretable sentence embeddings that support sentence operations, including fusion, difference, compression, and reconstruction. intersent combines both contrastive and generative objectives to optimize operator networks together with a bottleneck encoder-decoder model. experimental results show that intersent significantly improves the interpretability of sentence embeddings on four textual generation tasks. moreover, we demonstrate that our interpretable sentence embedding space preserves the ability to capture semantic similarity, and even improves performance on retrieval tasks.