  Pre-trained language models have achieved huge success on a wide range of NLP
tasks. However, contextual representations from pre-trained models contain
entangled semantic and syntactic information, and therefore cannot be directly
used to derive useful semantic sentence embeddings for some tasks. Paraphrase
pairs offer an effective way of learning the distinction between semantics and
syntax, as they naturally share semantics and often vary in syntax. In this
work, we present ParaBART, a semantic sentence embedding model that learns to
disentangle semantics and syntax in sentence embeddings obtained by pre-trained
language models. ParaBART is trained to perform syntax-guided paraphrasing,
based on a source sentence that shares semantics with the target paraphrase,
and a parse tree that specifies the target syntax. In this way, ParaBART learns
disentangled semantic and syntactic representations from their respective
inputs with separate encoders. Experiments in English show that ParaBART
outperforms state-of-the-art sentence embedding models on unsupervised semantic
similarity tasks. Additionally, we show that our approach can effectively
remove syntactic information from semantic sentence embeddings, leading to
better robustness against syntactic variation on downstream semantic tasks.
