Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models (2021)

Abstract: Pre-trained language models have achieved huge success on a wide range of NLP tasks. However, contextual representations from pre-trained models contain entangled semantic and syntactic information, and therefore cannot be directly used to derive useful semantic sentence embeddings for some tasks. Paraphrase pairs offer an effective way of learning the distinction between semantics and syntax, as they naturally share semantics and often vary in syntax. In this work, we present ParaBART, a semantic sentence embedding model that learns to disentangle semantics and syntax in sentence embeddings obtained by pre-trained language models. ParaBART is trained to perform syntax-guided paraphrasing, based on a source sentence that shares semantics with the target paraphrase, and a parse tree that specifies the target syntax. In this way, ParaBART learns disentangled semantic and syntactic representations from their respective inputs with separate encoders. Experiments in English show that ParaBART outperforms state-of-the-art sentence embedding models on unsupervised semantic similarity tasks. Additionally, we show that our approach can effectively remove syntactic information from semantic sentence embeddings, leading to better robustness against syntactic variation on downstream semantic tasks.

## introduction semantic sentence embedding models encode sentences into fixed-length vectors based on their semantic relatedness with each other. if two sentences are more semantically related, their corresponding sentence embeddings are closer. as sentence embeddings can be used to measures semantic relatedness without requiring supervised data, they have been used in many applications, such as semantic textual similarity @xcite , question answering @xcite , and natural language inference @xcite . recent years have seen huge success of pretrained language models across a wide range of nlp tasks @xcite . however, several studies @xcite have found that sentence embeddings from pre-trained language models perform poorly on semantic similarity tasks when the models are not fine-tuned on task-specific data. meanwhile, @xcite shows that bert without fine-tuning performs surprisingly well on syntactic tasks. hence, we posit that these contextual representations from pre-trained language models without fine-tuning capture entangled semantic and syntactic information, and therefore are not suitable for sentence-level semantic tasks. ideally, the semantic embedding of a sentence should not encode its syntax, and two semantically similar sentences should have close semantic embeddings regardless of their syntactic differences. while various models @xcite @xcite have been proposed to improve the performance of sentence embeddings on downstream semantic tasks, most of these approaches do not attempt to separate syntactic information from sentence embeddings. to this end, we propose parabart, a semantic sentence embedding model that learns to disentangle semantics and syntax in sentence embeddings. our model is built upon bart @xcite , a sequence-to-sequence transformer @xcite model pre-trained with selfdenoising objectives. parallel paraphrase data is a good source of learning the distinction between semantics and syntax, as paraphrase pairs naturally share the same meaning but often differ in syntax. taking advantage of this fact, parabart is trained to perform syntax-guided paraphrasing, where a source sentence containing the desired semantics and a parse tree specifying the desired syntax are given as inputs. in order to generate a paraphrase that follows the given syntax, parabart uses separate encoders to learn disentangled semantic and syntactic representations from their respective inputs. in this way, the disentangled representations capture sufficient semantic and syntactic information needed for paraphrase generation. the semantic encoder is also encouraged to ignore the syntax of the source sentence, as the desired syntax is already provided by the syntax input. parabart achieves strong performance across unsupervised semantic textual similarity tasks. furthermore, semantic embeddings learned by parabart contain significantly less syntactic information as suggested by probing results, and yield robust performance on datasets with syntactic variation. our source code is available at https:// github.com/uclanlp/parabart . ## related work various sentence embedding models have been proposed in recent years. most of these models utilize supervision from parallel data @xcite @xcite , natural language inference data @xcite @xcite , or a combination of both @xcite . many efforts towards controlled text generation have been focused on learning disentangled sentence representations @xcite @xcite . in the context of disentangling semantics and syntax, @xcite and @xcite utilize variational autoencoders to learn two latent variables for semantics and syntax. in contrast, we use the outputs of a constituency parser to learn purely syntactic representations, and facilitate the usage of powerful pre-trained language models as semantic encoders. our approach is also related to prior work on syntax-controlled paraphrase generation @xcite @xcite . while these approaches focus on generating high-quality paraphrases that conform to the desired syntax, we are interested in how semantic and syntactic information can be disentangled and how to obtain good semantic sentence embeddings. ## proposed model -parabart our goal is to build a semantic sentence embedding model that learns to separate syntax from semantic embeddings. parabart is trained to generate syntax-guided paraphrases, where the model attempts to only extract the semantic part from the input sentence, and combine it with a different syntax specified by the additional syntax input in the form of a constituency parse tree. figure 1 outlines the proposed model, which consists of a semantic encoder that learns the semantics of a source sentence, a syntactic encoder that encodes the desired syntax of a paraphrase, and a decoder that generates a corresponding paraphrase. additionally, we add a syntax discriminator to adversarially remove syntactic information from the semantic embeddings. given a source sentence s 1 and a target constituency parse tree p 2 , parabart is trained to generate a paraphrase s 2 that shares the semantics of s 1 and conforms to the syntax specified by p 2 . semantics and syntax are two key aspects that determine how a sentence is generated. our model learns purely syntactic representations from the output trees generated by a constituency parser, and extracts the semantic embedding directly from the source sentence. the syntax discriminator and the syntactic encoder are designed to remove source syntax and provide target syntax, thus encouraging the semantic encoder to only capture source semantics. ## experiments in this section, we demonstrate that parabart is capable of learning semantic sentence embeddings that capture semantic similarity, contain less syntactic information, and yield robust performance against syntactic variation on semantic tasks. ## conclusion in this paper, we present parabart, a semantic sentence embedding model that learns to disentangle semantics and syntax in sentence embeddings from pre-trained language models. experiments show that our semantic sentence embeddings yield strong performance on unsupervised semantic similarity tasks. further investigation demonstrates the effectiveness of disentanglement, and robustness of our semantic sentence embeddings against syntactic variation on downstream semantic tasks.