Sub-Sentence Encoder: Contrastive Learning of Propositional Semantic Representations (2024)

Abstract: We introduce sub-sentence encoder, a contrastively-learned contextual embedding model for fine-grained semantic representation of text. In contrast to the standard practice with sentence embeddings, where the meaning of an entire sequence of text is encoded into a fixed-length vector, the sub-sentence encoder learns to produce distinct contextual embeddings corresponding to different atomic propositions, i.e. atomic units of meaning expressed within a text sequence. The sub-sentence embeddings are contrastively learned to recognize (inferred) semantic equivalence between propositions across different text sequences. Our experiments show the effectiveness of sub-sentence encoders in applications, such as retrieving supporting facts for fine-grained text attribution or recognizing the conditional semantic similarity between texts. In practice, we demonstrate that sub-sentence encoders keep the same level of inference cost and space complexity compared to sentence encoders.

## introduction sentence embeddings are a class of techniques that represent text semantics as dense vector embedding(s), which are widely used in zero-shot or transfer learning settings on information retrieval and text classification tasks @xcite @xcite @xcite . sentence embeddings are typically learned to recognize the semantic relation between two text inputs. the common practice is to encode each text input as one fixed-length vector, where the semantic relation with the other input is modeled by a similarity function @xcite . the gothic novel dracula is written by bram stoker. ## sub-sentence encoder we introduce sub-sentence encoders. contrary to sentence encoders, sub-sentence encoders are de-signed to produce contextual embeddings for each atomic proposition in a sentence. ## conclusion we introduce sub-sentence encoders, a contrastive learning framework for learning contextual embeddings for semantic units on the sub-sentence level. beyond the use cases covered in the paper, the sub-sentence encoder architecture could potentially serve as the backbone for any cross-document information linking tasks in context, and the learning objectives could potentially apply to a broader range of tasks with various granularity of information, e.g., linking sentences or spans within different documents @xcite . we hope that the findings in this paper will facilitate further exploration along these directions.