Contrastive Loss is All You Need to Recover Analogies as Parallel Lines (2023)

Abstract: While static word embedding models are known to represent linguistic analogies as parallel lines in high-dimensional space, the underlying mechanism as to why they result in such geometric structures remains obscure. We find that an elementary contrastive-style method employed over distributional information performs competitively with popular word embedding models on analogy recovery tasks, while achieving dramatic speedups in training time. Further, we demonstrate that a contrastive loss is sufficient to create these parallel structures in word embeddings, and establish a precise relationship between the co-occurrence statistics and the geometric structure of the resulting word embeddings.

## introduction static word embeddings take inspiration from the distributional hypothesis @xcite and assign vector representations to words based on cooccurrence statistics. such embeddings are known to implicitly encode syntactic and semantic analogies as parallelogram-type structures @xcite . this discovery inspired a series of theoretical investigations @xcite @xcite . recent studies reconsider whether analogies are indeed represented as parallelograms in the embedding space @xcite @xcite , and propose a weaker notion of viewing analogies as parallel lines @xcite as a more appropriate model (cf. figure 1 ). while this claim is shown to hold empirically for popular word embeddings @xcite , few analyze the theoretical underpinnings of this phenomenon. in this paper, we present a remarkable observation that a simple contrastive-style optimization @xcite performs just as well as highly-optimized versions of popular word embeddings while achieving 50× speedup in training time. our work theoretically analyzes the precise conditions under which this optimization procedure can recover analogies as parallel lines. we further investigate the extent to which real-world data satisfies these conditions, and the contrastive loss recovers such parallel structures. in section 2, we review recent literature on the theory of word embeddings. sections 3 and 4 present our contrastive learning objective and its analysis. section 5 showcases the performance of our approach on analogy-based benchmarks. analogies as parallel lines. to the best of our knowledge, the only theoretical work that explores analogies more generally as parallel lines is by @xcite , who propose that analogies are encoded as such when the inner products between embeddings weakly recover the pmi of word co-occurrence statistics. we take an alternate approach and show that a contrastive-style optimization suffices to encode analogies as parallel lines. ## the contrastive word model (cwm) contrastive learning methods are based on an intuitive yet powerful idea that pulling similar items closer together while pushing dissimilar items away significantly improves model performance. we can employ the same push-pull dynamics in word embeddings by placing the vector representations of words that co-occur closer together than those of words that do not. we call this the contrastive word model (cwm), detailed below. ## analysis in this section, we provide a theoretical justification for the emergence of analogies as parallel lines when we optimize for the cwm objective. consider the expression for word vectors v @xmath0: theorem 1 for any quadruple of words a, b, c, @xmath1, if there exists a constant ζ ∈ r where the co-occurrence statistics satisfy the condition: note that theorem 1 establishes a direct relationship between word co-occurrence statistics-which are solely derived from the training corpus-and the geometric structure of the word embedding. for a given quadruple a, b, c, @xmath2, the existence of ζ induces parallel structures between va , vb , vc , vd . if such a ζ exists and is equal to 1, then @xmath3. when ζ ̸ = 1, then the difference vectors vb -v a and vdvc are mainly parallel, inducing a trapezoidal structure among va , vb , vc , vd (cf. figure 1 ). one would expect that the co-occurrence statistics of real data conform with the existence of such a ζ value for analogy quadruples, whereas ζ does analogies training model pcs msm time (hrs) speedup cwm 0.677 0.469 0.59 49× sgns 0.675 0.433 29.27 1× glove 0.667 0.423 30.71 0.91× ## experiments we first compare the performance of cwm to that of other popular word embedding methods on analogy recovery (section 5.2). we then empirically verify the degree to which our assumptions regarding co-occurrences hold on real data (section 5.3) as well as the relation between ζ and the parallelogram structure (section 5.4). ## conclusion and discussion we demonstrate that optimizing a contrastive-style objective over word co-occurrences is indeed sufficient to encode analogies as parallel lines. our analysis (theorem 1) sheds light on the inner workings of word embeddings: parallel geometry is induced largely from word co-occurrence statistics for any push-pull model. our work builds upon and generalizes previous literature that illuminates the underlying mechanisms governing the geometry of word embeddings. note that while our results demonstrate the sufficiency of the push-pull mechanism for recovering analogies as parallel lines, it remains unclear whether push-pull is a necessary condition for this phenomenon. investigating alternative mechanisms and their ability to achieve similar results would provide further insight into the relationship between word co-occurrence statistics and the recovery of analogies. louis fournier, emmanuel dupoux, and ewan dunbar. 2020. analogies minus analogy test: measuring regularities in word embeddings. in proceedings of the 24th conference on computational natural language learning, pages 365-375, online. association for computational linguistics. alex gittens, dimitris achlioptas, and michael w. mahoney. 2017. skip-gram -zipf + @xmath4. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 69-76, vancouver, canada. association for computational linguistics. anna gladkova, aleksandr drozd, and satoshi matsuoka. 2016. analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn't. in proceedings of the naacl student research workshop, pages 8-15, san diego, california. association for computational linguistics.