One of these words is not like the other: a reproduction of outlier identification using non-contextual word representations (2020)

Abstract: Word embeddings are an active topic in the NLP research community. State-of-the-art neural models achieve high performance on downstream tasks, albeit at the cost of computationally expensive training. Cost aware solutions require cheaper models that still achieve good performance. We present several reproduction studies of intrinsic evaluation tasks that evaluate non-contextual word representations in multiple languages. Furthermore, we present 50-8-8, a new data set for the outlier identification task, which avoids limitations of the original data set, such as ambiguous words, infrequent words, and multi-word tokens, while increasing the number of test cases. The data set is expanded to contain semantic and syntactic tests and is multilingual (English, German, and Italian). We provide an in-depth analysis of word embedding models with a range of hyper-parameters. Our analysis shows the suitability of different models and hyper-parameters for different tasks and the greater difficulty of representing German and Italian languages.

## introduction unsupervised word embeddings have largely replaced language-specific hand-designed representations of syntax and semantics @xcite @xcite . models based on deep neural networks such as the bert family @xcite @xcite construct contextualized word vector representations. showing state-of-theart results in benchmarks such as glue @xcite , they are computationally expensive for both training and inference @xcite with significant cost for the environment @xcite . in this paper, ⇤ equal contribution we turn our attention back to the non-contextual, less resource-hungry word representations of the word2vec family @xcite . we contribute reproduction studies on the quality of the non-contextual word representations using outlier identification @xcite and the classic word analogy task @xcite . replicability and reproducibility have gained increasing importance in the nlp community: focus on the publication of code and data with papers, special sections in leading journals @xcite , and dedicated shared tasks @xcite . unfortunately, there exist opposing definitions of the terms reproduction and replication (e.g., @xcite and @xcite ), while others propose a spectrum of reproducibility @xcite . while we aim to reproduce the experiments in our target papers closely, we go beyond a straight-forward reproduction and address further questions such as effect of hyper-parameters, linear contexts (cbow vs. skipgram), and non-linear dependency-based contexts @xcite . we also propose 50-8-8, an alternative to the 8-8-8 outlier identification data set @xcite that is several times larger, includes both semantic and syntactic evaluations, and addresses result variance issues that affect the original 8-8-8 data set. finally, our 50-8-8 data set is multilingual, covering english (en), german (de), and italian (it). the three languages are challenging for word representations due to their large vocabulary, heavy reliance on word compounding (de), and complex grammar and sentence structure @xcite . in our paper, we contribute: ## related work contextualized neural word embeddings @xcite show impressive performance in downstream nlp tasks, at the cost of training time; pre-training of the base version of bert took four days using 16 tpu chips @xcite . efforts to reduce the training time still require significant computing power on dedicated hardware @xcite , with high environmental cost @xcite . some reduction of memory usage @xcite or of training time and memory usage @xcite still does not eliminate the high resource consumption. as such, less computationally expensive models, such as word2vec @xcite , word2vecf @xcite , fasttext @xcite , and glove @xcite , are attractive when showing good performance on nlp tasks. computationally cheaper models, like word2vec, have some of the same evaluation drawbacks as their more complicated and expensive counterparts: there is no generally agreed upon evaluation. @xcite compare word2vec and word2vecf on attributional similarity, extended by @xcite for combinations of context representations and context types for cbow, skipgram, and glove. but, @xcite and @xcite note that attributional similarity is subjective, lacks statistical significance, and has a low correlation with extrinsic evaluation, making it inconsistent and not necessarily indicative of model properties. however, @xcite argue that different extrinsic evaluation tasks prefer different embeddings, suggesting that extrinsic tasks might not be indicators of general embedding quality either. the outlier identification task (camacho-collados and navigli, 2016) avoids subjective similarity measurements. instead, it employs relative word vector similarity to identify an outlier from a group of otherwise semantically related words. @xcite expanded the outlier identification data set algorithmically based on wikidata. however, the automatic approach has several limitations, including ambiguous, infrequent, or duplicate words in the same category, and word variants in the same category, likely due to hierarchy inconsistencies in wikidata @xcite . in this paper, we return to manually curated data sets with controlled quality and difficulty. in light of recent revelations into the instability of word2vec @xcite , we reproduce several word vector evaluations. we find that the original 8-8-8 data set used in the outlier identification evaluation leads to high results variance. we address this issue by proposing an expanded evaluation data set we call 50-8-8. both the original outlier identification (camacho-collados and navigli, 2016) and word similarity publications @xcite do not fully explore the effects of hyper-parameters and randomness. we systematically evaluate models and hyper-parameters on ten training runs and measure average performance and variance. finally, most evaluations of word2vec embeddings focus on english, with notable exceptions @xcite @xcite @xcite @xcite . however, these are translations of word similarity tasks and share the weaknesses of their english language counterparts. we reproduce the evaluation of core word analogy evaluations of @xcite and @xcite and expand them by comparing word2vec to its dependency-based counterpart, word2vecf. we use the word analogy task from @xcite to give a reference point for model performance and ease comparison with other research, even though the pitfalls from the similarity tasks also apply to this task @xcite . to supplement the evaluations on non-english languages, we manually translate our new 50-8-8 data set into german and italian and thus provide a multilingual outlier identification data set and evaluation. ## tasks in this section, we introduce the intrinsic tasks and data sets we use for evaluation. furthermore, we summarize previous data sets' limitations and introduce a new data set for the outlier identification task. ## models and corpora this section introduces the word embedding models and the training corpora we use for the evaluation. ## experiments this section presents experimental results, focusing on the reproduction, new data set, window size, different corpora, and languages. in our tables and figures, we denote the different approaches as follows: cbow, sg (skip-gram), w2vf (word2vecf), w2vf+ (word2vecf+); each followed by the size of the window used. we include a detailed description of the experimental setup in appendix c. ## conclusions we contribute several reproduction studies of the outlier identification task and the classic word analogy task, both intrinsic evaluations of noncontextual word representations. we provide an in-depth analysis of word2vec, word2vecf, and word2vecf+ on the two tasks analyzing the effects of window size, context type, and context representation on english, german, and italian. we find that the context construction strategy of word2vecf and word2vecf+ is not always effective. sometimes the two models underperform even the weakest of the word2vec models. our reproduction of outlier identification shows high variance, which we attribute to the original data set's limitations. to address these limitations, we propose 50-8-8, a new data set that is multiple times larger, manually curated, multilingual, and contains syntactic and semantic tests. besides eliminating the variance issues, 50-8-8 quantifies the drop in performance in representations of languages with more complicated grammar and mor-phology than english.