Learning Semantic Textual Similarity via Topic-informed Discrete Latent Variables (2022)

Abstract: Recently, discrete latent variable models have received a surge of interest in both Natural Language Processing (NLP) and Computer Vision (CV), attributed to their comparable performance to the continuous counterparts in representation learning, while being more interpretable in their predictions. In this paper, we develop a topic-informed discrete latent variable model for semantic textual similarity, which learns a shared latent space for sentence-pair representation via vector quantization. Compared with previous models limited to local semantic contexts, our model can explore richer semantic information via topic modeling. We further boost the performance of semantic similarity by injecting the quantized representation into a transformer-based language model with a well-designed semantic-driven attention mechanism. We demonstrate, through extensive experiments across various English language datasets, that our model is able to surpass several strong neural baselines in semantic textual similarity tasks.

## introduction semantic textual similarity (sts), which concerns the problem of measuring and scoring the relationships or relevance of pairs of text on real-valued scales, is a fundamental task in nlp. it has further driven many other important nlp tasks such as machine translation @xcite ), text summarization @xcite , question answering @xcite , and etc. recently, deep neural language models have achieved state-of-the-art performance for sts. the success of these models is attributed to their adoption of self-supervised learning on text representations, which overcomes the absence of large labeled sts data in many domains. the text representations learned by these models, usually consist of continuous latent variables, and can be fed pairwise * corresponding author into some functions (e.g. a multi-layer perceptron) to compute their semantic similarity. compared to the continuous ones, discrete latent variables have drawn much less attention in language modeling, despite that natural languages are discrete in nature, and there is growing evidence, from several nlp tasks @xcite , that they are equally good, if not more suitable, as the continuous counterparts. as a widelyused training technique for learning discrete latent variables, vector-quantized variational autoencoder (vq-vae) @xcite computes the values for the variables through the nearest neighbor lookup across the quantized vectors from the shared latent embedding space. despite its success in speech recognition and computer vision, vq-vae has yet to be investigated in its use and viability in general nlp tasks. in this paper, we intend to explore the use of vq-vae and its generated discrete latent variables for semantic textual similarity. two issues need to be resolved in order for the vq-vae to work effectively on the sts task. first, the codebook embeddings need to be carefully initialized to prevent the so-called codebook collapse, in which only a few of the embeddings are selected and learned by the model, causing a reduction in the representational capacity of the codebook. past research has indicated that sufficiently informed and sophisticated manipulation of the embedding initialization can mitigate this problem. a second issue is that most sts scenarios are designed to teach a language model to measure the textual similarity from a local perspective (e.g. over contexts within each sentence). however, global semantics underlying a broader context has already been shown to benefit a number of nlp tasks, including language generation @xcite , textual similarity calculation @xcite , keyphrase generation @xcite , etc. for sts, it can provide perspectives on the global correlations and dependencies among sentences, which can be leveraged to better distinguish the semantic nuances of sentence-level contexts. in this paper, we propose to leverage topic modeling to tackle both issues. the topic information can provide informative guidance on the initialization and the learning of vq-vae's codebook. furthermore, since the discovered topics can capture global semantics (e.g., semantics as distributions over the vocabulary shared by the whole corpus), they can be used to calibrate the bias of local semantics on the measure of textual similarity. our proposed topic-enhanced vq-vae language model features two major components to enrich the codebook representational capacity with rich contextual information. one is the topic-aware sentence encoder, which computes the sentence embedding using its (local) topic distribution and the corresponding global topic-word distributions. the other is a topic-guided vq-vae codebook, where we align its latent codes/embeddings with the topic embeddings jointly learned by an ntm throughout the training phase. we further incorporate the quantized sentence representations learned by our topic-enhanced vq-vae into transformer-based language models to guide the learning of contextual embeddings. built on top of the multi-head attention, a semantics-driven attention mechanism is proposed to compute the attention scores based on the quantized sentence representations. the proposed attention mechanism is a flexible plugand-play module that requires a small number of extra parameters and a minimum change to the existing multi-head attention implementation. our contributions are summarised as: ## modelling framework given a pair of sentence @xmath0, x j } as the input, where each sentence is represented as the bag-of-words (bow) vectors and contextualized embedding, our modelling framework, as shown in figure 1 , predicts the semantic similarity y between x i and x j . ## conclusion in this paper, we developed a topic-enhanced vq-vae model to effectively train discrete latent variables by informing the vector quantization with semantics (i.e., latent topics) learned from a broader context by a neural topic model. then we further designed a semantics-driven multi-head attention mechanism for enriching the contextual embeddings learned by transformer with topical information, which is calculated based on the quantized