Comparing word2vec and G lo V e for Automatic Measurement of MWE Compositionality (2020)

Abstract: This paper explores the use of word2vec and GloVe embeddings for unsupervised measurement of the semantic compositionality of MWE candidates. Through comparison with several human-annotated reference sets, we find word2vec to be substantively superior to GloVe for this task. We also find Simple English Wikipedia to be a poor-quality resource for compositionality assessment, but demonstrate that a sample of 10% of sentences in the English Wikipedia can provide a conveniently tractable corpus with only moderate reduction in the quality of outputs.

## introduction multiword expressions (mwes) are word combinations exhibiting one or more idiosyncrasies-lexical, syntactic, semantic, pragmatic or statistical @xcite . this paper is concerned specifically with semantic compositionality: the extent to which the meaning of an mwe can be understood from those of its component words. the semantics of compositional expressions such as picnic basket are clear to anyone familiar with the constituents picnic and basket, but a non-compositional phrase like iron curtain is opaque without further context. word embedding models such as glove @xcite , word2vec @xcite and doc2vec @xcite are widely used in the natural language processing (nlp) sphere, and are capable of capturing syntactic and semantic relationships between words through their representations in multi-dimensional vector space @xcite . these models therefore offer an opportunity to automatically evaluate the compositionality of an mwe candidate by comparing the embedded representation of the complete expression with those of its component words; we may expect that the vectors of more decomposable phrases will be more similar to those of their constituents. embedding models themselves also benefit from mwe discovery; by treating multi-word expressions as single units, one may obtain higher-quality representations of simplex words @xcite . our main aim in this paper is to evaluate the performance of glove models for this purpose, in comparison with word2vec. given that many state-of-the-art nlp applications have adopted bert embeddings @xcite , these were also considered. however, bert's embeddings differ according to the sentence in which a given word appears. since our methodology requires comparison between the vector representation of mwe candidates and their constituent words, the use of contextdependent embeddings seems inappropriate. section 2 outlines relevant past work in this area, in particular that of @xcite , whose methodology we adapt and whose results provide us with a valuable point of comparison. our method and resources are described in section 3, including the human-annotated reference sets used to evaluate our scores. finally, we discuss our findings in section 4. ## past research lin (1999) employs a substitution-based method to detect non-compositionality. however, while noncompositional phrases also exhibit institutionalisation (resistance to substitution of synonyms), the re-this work is licensed under a creative commons attribution 4.0 international licence. licence details: http://creativecommons.org/licenses/by/4.0/ . more recently, @xcite generated a large list (over 900k entries) of multi-word phrases, which they extracted from english wikipedia and automatically scored for compositionality using an approach inspired by @xcite . our methodology (described in section 3.3) is based on theirs, with alterations to the source corpora and reference sets as well as to the embedding models used. ## results the correlation (spearman ρ and pearson's r) between our mean cosine distance measure and human annotations is reported for n-grams appearing on both our list and the reference sets, together with the size of this overlap, in in order to explore the impact of restricting the vocabulary used for training the glove models, a further experiment was carried out on the simp20 corpus, using an unrestricted vocabulary of 1,014,614 simplex words, together with the mwe candidates assigned to each batch. we find substantially lower correlation with the glove-derived compositionality scores than those obtained using word2vec, across both corpora. the glove model with unrestricted vocabulary appears comparable to word2vec, but required greater computational resources to train. both practical and performance factors lead us to prefer word2vec for future work in this area. this aligns with the findings of @xcite if we regard glove as an evolution of the 'count-based' vector paradigm, despite its reported success elsewhere @xcite . the simple english wikipedia corpus produces fewer matches with the reference lists of mwes as well as weaker correlation with human compositionality judgements; the smaller size of this corpus and the nature of its content make it a poor hunting ground for multi-word expressions. however, our 10% sample of english wikipedia yielded reasonable results while remaining tractable foot_2 . our output lists and code resources are available at https://github.com/oddtwang/mwes . future work includes exploration of context-dependent embeddings such as doc2vec @xcite and bert @xcite for compositionality assessment, particularly for ngrams which may not always form mwes. application of the technique to other corpora and languages with suitable mwe resources, e.g. arabic @xcite would also be valuable.