Learning Probabilistic Sentence Representations from Paraphrases (2020)

Abstract: Probabilistic word embeddings have shown effectiveness in capturing notions of generality and entailment, but there is very little work on doing the analogous type of investigation for sentences. In this paper we define probabilistic models that produce distributions for sentences. Our best-performing model treats each word as a linear transformation operator applied to a multivariate Gaussian distribution. We train our models on paraphrases and demonstrate that they naturally capture sentence specificity. While our proposed model achieves the best performance overall, we also show that specificity is represented by simpler architectures via the norm of the sentence vectors. Qualitative analysis shows that our probabilistic model captures sentential entailment and provides ways to analyze the specificity and preciseness of individual words.

## introduction probabilistic word embeddings have been shown to be useful for capturing notions of generality and entailment @xcite @xcite . in particular, researchers have found that the entropy of a word roughly encodes its generality, even though there is no training signal explicitly targeting this effect. for example, hypernyms tend to have larger variance than their corresponding hyponyms @xcite . however, there is very little work on doing the analogous type of investigation for sentences. in this paper, we define probabilistic models that produce distributions for sentences. in particular, we choose a simple and interpretable probabilistic model that treats each word as an operator that translates and scales a gaussian random variable representing the sentence. our models are able to capture sentence specificity as measured by the annotated datasets of @xcite and @xcite by training solely on noisy paraphrase pairs. while our "wordoperator" model yields the strongest performance, we also show that specificity is represented by simpler architectures via the norm of the sentence vectors. qualitative analysis shows that our models represent sentences in ways that correspond to the entailment relationship and that individual word parameters can be analyzed to find words with varied and precise meanings. ## proposed methods we propose a model that uses ideas from flowbased variational autoencoders (vaes) @xcite by treating each word as an "operator". intuitively, we assume there is a random variable z associated with each sentence @xmath0, w 2 , • • • , w n }. the random variable initially follows a standard multivariate gaussian distribution. then, each word in the sentence transforms the random variable sequentially, leading to a random variable that encodes its semantic information. our word linear operator model (wlo) has two types of parameters for each word w i : a scaling factor a @xmath1. the word operators produce a sequence of random variables z 0 , z 1 , • • • , z n with z 0 ∼ n (0, i k ), where i k is a k × k identity matrix, and the operations are defined as the means and variances for each random variable are computed as follows: for computational efficiency, we only consider diagonal covariance matrices, so the equations above can be further simplified. ## learning following @xcite , all of our models are trained with a margin-based loss on paraphrase pairs (s 1 , s 2 ): ## related work our models are related to work in learning probabilistic word embeddings @xcite @xcite ) and text-based vaes @xcite @xcite @xcite . the wlo is also related to flow-based vaes @xcite , where hidden layers are viewed as operators over the density function of latent variables. previous work on sentence specificity relies on hand-crafted features or direct training on annotated data @xcite . recently, @xcite used domain adaptation for this problem when only the source domain has annotations. our work also relates to learning sentence embeddings from paraphrase pairs @xcite . ## conclusion we trained sentence models on paraphrase pairs and showed that they naturally capture specificity and entailment. our proposed wlo model, which treats each word as a linear transformation operator, achieves the best performance and lends itself to analysis.