Corrected CBOW Performs as well as Skip-gram (2021)

Abstract: Mikolov et al. (2013a) observed that continuous bag-of-words (CBOW) word embeddings tend to underperform Skip-gram (SG) embeddings, and this finding has been reported in subsequent works. We find that these observations are driven not by fundamental differences in their training objectives, but more likely on faulty negative sampling CBOW implementations in popular libraries such as the official implementation, word2vec.c, and Gensim. We show that after correcting a bug in the CBOW gradient update, one can learn CBOW word embeddings that are fully competitive with SG on various intrinsic and extrinsic tasks, while being many times faster to train.

## introduction pre-trained word embeddings are a standard way to boost performance on many tasks of interest such as named-entity recognition (ner), where contextual word embeddings such as bert @xcite are computationally expensive and may only yield marginal gains. word2vec @xcite ) is a popular method for learning word embeddings because of its scalability and robust performance. there are two main word2vec training objectives: (1) continuous bag-of-words (cbow) that predicts the center word by averaging context embeddings, and (2) skip-gram (sg) that predicts context words from the center word. 1 it has been frequently observed that while cbow is fast to train, it lags behind sg in performance. 2 this observation is made by the inventors of word2vec themselves @xcite , and also independently in subsequent works @xcite . this result is surprising since the cbow and sg objectives lead to very similar weight updates. this is also contrary to the enormous success of contextual word embeddings based on masked language modeling (mlm), as cbow follows a rudimentary form of mlm (i.e., predicting a masked target word from a context window without any perturbation of the masked word). in this work, we find that the performance discrepancy between cbow and sg embeddings is founded less on theoretical differences in their training objectives but more on faulty cbow implementations in standard libraries such as the official implementation word2vec.c @xcite , gensim @xcite and fasttext @xcite . specifically, we find that in these implementations, the gradient for source embeddings is incorrectly multiplied by the context window size, resulting in incorrect weight updates and inferior performance. we make the following contributions. first, we show that our correct implementation of cbow indeed yields word embeddings that are fully competitive with sg while being trained in less than a third of the time (e.g., with a single machine it takes less than 73 minutes to train cbow embeddings on the entire wikipedia corpus). we present experiments on intrinsic word similarity and analogy tasks, as well as extrinsic eval uations on the sst-2, qnli, and mnli tasks from the glue benchmark @xcite , and the conll03 english named entity recognition task @xcite . second, we make our implementation, kōan, publicly available. 3 with this implementation, it is possible to train 768-dimensional cbow embeddings on one epoch of english c4 @xcite in 1.61 days on a single 16 cpu machine. foot_3 ## cbow implementation the parameters of cbow are two sets of word embeddings: "source-side" and "target-side" vectors v w , v @xmath0. a window of text in a corpus consists of a center word w o and context words w 1 . . . w c . for instance, in the window the dog laughed, we have w @xmath1. given a window of text, the cbow loss is defined as: however, the cbow negative sampling implementations in word2vec.c foot_4 , gensim foot_5 , and fasttext foot_6 incorrectly update each context vector, v w j , by eq. ( 1 ), without normalizing by the number of context words, given in eq. ( 2 ). in fact, this error has been pointed out in several gensim issues as well as a fasttext issue. 8 why this error matters aside from being incorrect, this update matters for two reasons. first, in both gensim and word2vec.c, the width of the context window is randomly sampled from 1, . . . , c max for every target word. this means that source embeddings which were averaged over wider context windows will experience a larger update than their contribution, relative to source embeddings averaged over narrower windows. second, in the incorrect gradient, ∂l ∂v is incorrectly scaled by c, while ∂l ∂v is not; the correct stochastic gradient with respect to all embeddings actually points in a different direction than what was implemented in word2vec.c. section 5 touches on both of these issues in more detail. ## experiments we evaluated cbow and sg embeddings under gensim and our implementations (both corrected and original cbow). unless otherwise noted, we learn word embeddings on the entire wikipedia corpus ptb sentence split and tokenized with default settings, where words occurring fewer than ten times were dropped. training hyperparameters were held fixed for each set of embeddings: negative sampling rate 5, maximum context window of 5 words, number of training epochs 5, and embedding dimensionality 300 unless otherwise noted. we found that the default initial learning rate in gensim, 0.025, learned strong sg embeddings. however, we swept separately for cbow initial learning rate for gensim and our implementation. we swept over initial learning rate in {0.025, 0.05, 0.075, 0.1} selecting 0.025 to learn gensim cbow embeddings and 0.075 for corrected cbow, to maximize average performance on the development fold (random partition of 50% of examples) of intrinsic evaluation tasks described in 3.1. we selected learning rate as this was a critical hyperparameter, and we found that cbow embeddings learned with 0.075 learning rate with gensim suffered compared to learning with a low learning rate. ## results intrinsic evaluation cbow embeddings trained with our implementation achieve similar intrinsic performance to sg embeddings, surpassing sg embeddings for the rw and synt tasks ( ## discussion the gensim-trained cbow embeddings achieve better test accuracy on sst-2 despite achieving similar performance on the development set as corrected cbow embeddings. gensim cbow embeddings also outperform on qnli. however, performance on multinli is slightly different. in this case, corrected cbow embeddings are almost 2% more accurate than gensim embeddings when evaluated on the out of domain test set. for these glue tasks, it is unclear whether cbow embeddings learned with our implementation are universally more performant than those learned by gensim. we hypothesize this may be due to the susceptibility of models overfitting to the development set (especially since our classifier has 1024 dimensional hidden layers). ## conclusion before the widespread adoption of automatic differentiation libraries, it was the modeler's responsibility to derive the correct gradient for updating model weights. this word2vec.c gradient bug could have been avoided if a finite difference check was run to ensure the derivation of the stochastic gradient was correct. these checks are standard practice @xcite , and we strongly encourage other researchers to use finite difference gradient checks to verify the correctness of manually derived gradients. according to @xcite : "when the computation of the gradients is slightly incorrect, stochastic gradient descent often works slowly and erratically... it is not uncommon to discover such bugs in sgd code that has been quietly used for years." we find that cbow can learn embeddings that are as performant as sg, when trained with the correct update, allowing efficient training of strong word embeddings on web-scale datasets on an academic budget. we release our implementation, kōan, along with trained c4 cbow embeddings at https://github.com/ bloomberg/koan .