Understanding the Source of Semantic Regularities in Word Embeddings (2020)

Abstract: Semantic relations are core to how humans understand and express concepts in the real world using language. Recently, there has been a thread of research aimed at modeling these relations by learning vector representations from text corpora. Most of these approaches focus strictly on leveraging the co-occurrences of relationship word pairs within sentences. In this paper, we investigate the hypothesis that examples of a lexical relation in a corpus are fundamental to a neural word embedding’s ability to complete analogies involving the relation. Our experiments, in which we remove all known examples of a relation from training corpora, show only marginal degradation in analogy completion performance involving the removed relation. This finding enhances our understanding of neural word embeddings, showing that co-occurrence information of a particular semantic relation is not the main source of their structural regularity.

## introduction the representation of words has been a longstanding task in natural language processing (nlp). the main underlying principle is known for decades, as explained by @xcite . this principle was based on the idea that the meaning of a word can be understood by its surrounding company (i.e., the words in its context). most modern representation learning theory in nlp is based on this assumption, with vector representation being the most successful area to date @xcite . more recently, low-dimensional word representations learned from text corpora using neural networks (i.e., word embeddings) have emerged @xcite @xcite stemming from cognitive frameworks based on distributed representation @xcite . neural word embeddings have been proven to contain useful information about concepts and entities, and provide a generalization boost to many nlp applications @xcite . surprisingly, these representations have also been shown to exhibit linear relationships between words in the vector space, demonstrated by analogy. for example, @xcite showed that a simple operation such as king-man+woman will result in a point near queen in the vector space foot_0 . these word analogies have been extensively investigated in the literature, aiming to shed light on this surprising property. however, while there has been a body of research seeking to understand how these analogies work @xcite @xcite , and noting issues about their methodology @xcite @xcite , there has not been a specific analysis on the source of statistical cues that leads to their high performance on this task. concurrently, a thread of research has focused on explicitly modeling lexical relationships of word pairs in text corpora @xcite @xcite @xcite . while these methods employ different means for learning relation vectors, they share a common initial premise: only co-occurring words in the corpus are considered 2 . while this simplified assumption works well enough in practice, providing a useful signal even in downstream nlp applications, in this paper we find that valuable information is likely lost in the process. in fact, we find that a text corpus provides enough information to infer pairwise relations without training on any specific examples of a given relation. in our experiments, we focus on semantic relations in particular, which are the most suited for both word and relation embedding models. neural word embeddings are used to learn representations from text corpora, with word analogies as the evaluation mechanism to test our hypothesis. we run an extensive set of control experiments where the co-occurrence information is completely removed from the reference corpora. the results show that, with relationship instance removal, analogy performance degrades to only a limited extent, overall. 3 this finding suggests that neural embeddings do not learn lexical relation regularities from examples of the relation, but that they are still able to be inferred through the semantic featurization of individual words. ## methodology in this section we explain the experimental methodology we follow to answer our main research question. first, we briefly describe how to solve word analogies using word embeddings (section 3.1). we then explain our methodology to compile corpora to train word embeddings (section 3.2). ## evaluation in this section we provide the details of our experimental setup (section 4.1) and then, the main results of our evaluation (section 4.2). ## analysis in this section we aim to better understand the results presented in the previous section. in particular, we analyze to what extent the performance drops that exist are correlated with frequencies of words in the relationship (section 5.1). we also compare, through principal component analysis (pca) visualization, the structure of the relation with the highest point decrease, before and after co-occurrence removal (section 5.2). ## conclusion and future work in this paper we have presented a large-scale analysis on the role of co-occurring relational word pairs in completing analogies. in the analyses we have measured to what extent the loss of co-occurrence information within relation types affects analogy completion using neural word embeddings. perhaps surprisingly, this effect is quite small, to the point that word embeddings can complete analogies of a relationship in the vector space even if the co-occurrence information from the reference corpus is totally removed. in order to complement this analysis, for future work it would be interesting to analyze to what extent the conclusions of this analysis apply to purely distributional models, e.g., pmi-based, as they have shown to share similarity properties with word embeddings @xcite , to the point of skip-gram being viewed as an implicit co-occurrence matrix factorization @xcite . moreover, the analysis could be extended to other types of relations, not only semantic. further investigation could then focus on how the main sources of concepts and linguistic regularities in word embeddings are learned, and how they can be leveraged to improve unsupervised relation models, e.g., @xcite . finally, as a follow-up to recent work aiming at understanding how language models and contextualized embeddings capture relations @xcite @xcite , further research could be devoted to analyze the performance of such models with and without pairwise co-occurrence information.