Matching Varying-Length Texts via Topic-Informed and Decoupled Sentence Embeddings (2024)

Abstract: Measuring semantic similarity between texts is a crucial task in natural language processing. While existing semantic text matching focuses on pairs of similar-length sequences, matching texts with non-comparable lengths has broader applications in specific domains, such as comparing professional document summaries and content. Current approaches struggle with text pairs of non-comparable lengths due to truncation issues. To address this, we split texts into natural sentences and decouple sentence representations using supervised contrastive learning (SCL). Meanwhile, we adopt the embedded topic model (ETM) for specific domain data. Our experiments demonstrate the effectiveness of our model, based on decoupled and topic-informed sentence embeddings, in matching texts of significantly different lengths across three well-studied datasets.

## introduction text matching is an important research area in natural language processing (nlp) applications such as information retrieval, natural language inference, and question answering. however, many text-matching approaches assume that the texts being compared have similar lengths @xcite @xcite , and most pre-trained models, such as bert @xcite , focus on learning short sequences and are inadequate to represent complex domain-specific documents. large language models (llm) can directly process longer texts but require increasingly extensive training resources @xcite , which has evident limitations in some practical application scenarios. a natural way for humans to deal with long texts is to break them down into smaller text segments before processing them @xcite . inspired by this, for a varying-length text-matching short short text text long long text text 1 1 scl optimization scl optimization embedded topic model embedded topic model k x 300 bs x k x bs x 300 topic-informed topic-informed text embedding text embedding bs x 1068 bs x 1068 bs x 768 classifier classifier (avg.) (avg.) (avg.) (avg.) h h 2 2 sentence sentence embedding space embedding space match match mismatch mismatch sentence embedding vector sentence embedding vector doc-topic doc-topic distribution distribution topic topic embeddings embeddings task, we can split the two texts into natural sentences and put the sentences of the two texts into different sets. the motivation of our approach is: 1) if these two texts do not match, then their respective sets of sentences can be considered to belong to different classes. therefore, the sentence points belonging to the same class should be close in their representation space, while the points from different classes should be further apart. 2) if two texts are semantically similar and consistent, the topics of the two texts should match. accordingly, we present an effective method for varying-length text-matching with topic-informed and decoupled sentence embeddings (tde), as shown in figure 1 and described in section 2. first, we segment the text into sentences and utilize a state-of-the-art pre-trained model's transformer encoder to convert them into embedding points by extracting the last hidden state. we then employ scl to optimize class-wise relations in the embedding space of sentence representations. meanwhile, we discover the topic embeddings of the corpus by etm @xcite and calculate the text embeddings with topic information. finally, we concatenate the scloptimized and the topic-informed embeddings to predict scores for varying-length text matching. two related works, the hidden topic comparison method (ht) @xcite and the unsupervised concept generation network (cgnet) @xcite , share similarities with our approach. ht extracts hidden topics and compares documents with varying lengths, while cgnet relies on local phrase features and corpus-level concept features for matching. in comparison, our method is simpler, more effective, and explainable. moreover, our model can leverage the continuous evolution of llms to enhance performance. ## conclusion we present a novel and effective method for matching texts of varying lengths. we split long texts into sentences and encode them with the advanced transformer. leveraging the in-class and manyto-many optimization characteristics of supervised contrastive learning and the domain-specific ability of embedded topic modeling, our model demonstrates superiority against the baseline models on three real-world datasets in both effectiveness and explainability of varying-length text matching.