Sentence Analogies: Linguistic Regularities in Sentence Embeddings (2020)

Abstract: While important properties of word vector representations have been studied extensively, far less is known about the properties of sentence vector representations. Word vectors are often evaluated by assessing to what degree they exhibit regularities with regard to relationships of the sort considered in word analogies. In this paper, we investigate to what extent commonly used sentence vector representation spaces as well reflect certain kinds of regularities. We propose a number of schemes to induce evaluation data, based on lexical analogy data as well as semantic relationships between sentences. Our experiments consider a wide range of sentence embedding methods, including ones based on BERT-style contextual embeddings. We find that different models differ substantially in their ability to reflect such regularities.

## introduction sentence embeddings are dense vectors that reflect salient semantic properties of a sentence. similar to how commonly used word embedding methods such as word2vec @xcite capture semantic relationships between words, sentence embeddings are expected to encode semantic relationships between sentences. a number of different sentence embedding methods have been proposed (cf. section 2.1 for an overview). in recent years, pretrained language models such as elmo @xcite , bert @xcite , xlnet @xcite , and roberta @xcite have become the method of choice when encoding text. thus, such models are also often invoked to represent sentences by means of individual embeddings. 1 while important properties of word vector representations have been studied extensively, far less is known about the properties of sentence vector representations. a particularly prominent aspect of word vector representations induced by methods such as word2vec is that the vector space exhibits certain kinds of regularities. many of these are of the sort considered in word analogies. proportional analogies take the form a is to b as c is to d, e.g., paris is to france as berlin is to germany. @xcite first proposed identifying such analogies using vector representations in a euclidean space. given vector representations of concepts, derived from human similarity judgments using a multi-dimensional scaling algorithm, they proposed representing analogical relationships in terms of the difference vectors. @xcite investigated identifying such analogies using bag-of-words vector space models. @xcite showed that word2vec's word vector representations reflect certain kinds of word analogies surprisingly well. the widely used word analogy task that they proposed takes the following form. given embeddings v a , v b , v c , v d for words a, b, c, d for an analogy of the above form, the task consists in identifying the correct word d given a, b, and c. most commonly, this is achieved by optimizing ## methodology our goal is to explore to what extent different sentence embedding spaces reflect analogical regularities of the form a is to b as c is to d. in the remainder of this paper, we shall invoke the notation a : b :: c : d to refer to this sort of relationship. we will assess such relationships using the same methods as considered for word vectors. a typical choice is the method given by eq. 1 (see section 4.1 for further discussion). to be able to perform our analysis, we induce two kinds of data. in section 3.1, we create sentence analogies based on lexical analogies. in section 3.2, we induce sentence analogies based on predefined relationships between sentences. ## experiments in a sentence analogy task, we are given two pairs of sentences sharing a relation. for example, "he is very enamored with culture in egypt" : "he is very enamored with egyptian culture", and "he is very enamored with culture in bulgaria" : "he is very enamored with bulgarian culture". the goal is to identify the fourth sentence given the first three sentences. the kind of analogical relationship sought for the sentence pairs is not explicitly provided. the number of sentence pairs and analogies in each category of our analogy dataset is given in ## conclusion this paper presents several new datasets to test to what extent existing sentence embedding models exhibit regularities with regard to sentence analogies. most of the sentence embedding models we tested succeeded in recognizing syntactic analogies based on lexical ones, but had a harder time capturing semantic regularities by means of an analogy task. moreover, the remarkable success of bert-style contextual embeddings does not always translate into better regularities in the vector space of fixed-length sentence embeddings. more training data and model parameters as well do not necessarily yield better results. in many cases, word vector averages or a discrete cosine transform of word embeddings outperform more complex sentence embedding models. resources related to this study are available online at http://sentence.embeddings.org .