n e u r A l n e t w o r K s can capture both the semantic and syntactic information of words from a large unlabeled corpus 4 and has attracted considerable attention from many researchers. In recent years, several models have been proposed, many of which have yielded state-of-the-art results in various natural language processing (NLP) tasks. Although these studies always claim that their models are better than previous ones based on various evaluation criteria, very little research compares existing wordembedding methods. This article focuses on this issue and presents a detailed analysis of several important features for training word embeddings via an experimental review, including model construction, training corpus, and parameter design. To the best of our knowledge, no such study has been performed previously (see the "Related Work in Word Embedding Comparison" sidebar). To design an effective word-embedding method, we must first clarify the model construction. Almost all methods for training word embeddings are based on the same distributional hypothesis: words that occur in similar contexts tend to have similar meanings. 5, 6 Given this hypothesis, different methods model the relationship between a target word w and its context c in a corpus in different ways, where w and c are embedded into vectors. Generally, existing methods vary in two major aspects in model construction: the relationship between the target word and its context, and the representation of the context. Table 1 shows a brief comparison of commonly used models. In terms of the relationship between the target word and its context, the first five models are identical. They use an object similar to conditional probability P(w|c), which predicts the target word w based on its context c. Ronan Collobert and Jason Weston (C&W) 2 use an object similar to joint probability, where the (w, c) pairs in the corpus are trained to obtain higher scores. In terms
Training elements such as the model, the corpus, and the parameters have been analyzed in detail in REF .
How to Generate a Good Word Embedding
IEEE Intelligent Systems
IEEE Intelligent Systems
Computer Science