Evaluating Word Embeddings on Low-Resource Languages (2020)

Abstract: The analogy task introduced by Mikolov et al. (2013) has become the standard metric for tuning the hyperparameters of word embedding models. In this paper, however, we argue that the analogy task is unsuitable for low-resource languages for two reasons: (1) it requires that word embeddings be trained on large amounts of text, and (2) analogies may not be well-defined in some low-resource settings. We solve these problems by introducing the OddOneOut and Topk tasks, which are specifically designed for model selection in the low-resource setting. We use these metrics to successfully tune hyperparameters for a low-resource emoji embedding task and word embeddings on 16 extinct languages. The largest of these languages (Ancient Hebrew) has a 41 million token dataset, and the smallest (Old Gujarati) has only a 1813 token dataset.

## introduction imagine you're given the task of training a text classification model for middle english. this form of english was spoken in the middle ages from 1066-1500 ce. it is significantly different from modern english @xcite , and only a handful of historians speak this language today. a natural first step would be to train word embeddings. so you use the classical language toolkit (cltk) @xcite to download the largest corpus of known middle english documents (only 7 million tokens, 0.3 million unique tokens), and gensim @xcite to train the embeddings. to evaluate the embeddings, you follow the current standard practice established by @xcite of using an analogy test set. of course, you can't use @xcite 's test set-it is in modern english, and middle english is not modern english. but you also can't even use translations of their test set-many of the figure 1 : the standard analogy task @xcite fails to measure the quality of word embeddings trained on small datasets, but our novel oddoneout and topk tasks succeed in this regime. london is to england as paris is to france can be translated perfectly fine into middle english, but the concept of nations and capitals didn't exist in the middle ages, so the analogy is not semantically meaningful. to create a meaningful analogy test set, you hire a historian fluent in middle english, and with considerable effort and research she creates custom analogies that make sense in middle age england. with this analogy test set in hand, you train dozens of models with varying hyperparameters. unfortunately, all these models get 0 accuracy on your test set. you simply don't have enough data to get good results on the analogy task. as figure 1 shows, the analogy task requires a large training dataset before it begins getting non-zero results (details provided in section 3.1 below). but that does not mean that you cannot train word embeddings on middle english. in this paper, we introduce the oddoneout and topk tasks for evaluating word vectors on low-resource languages, successfully train word embeddings for 16 extinct languages (including middle english), and perform a low-resource emoji embedding task. to get a sense of scale, the original word2vec paper trained english word embeddings on a dataset with 6 billion tokens @xcite with subsequent work improving performance by training on datasets as large as 630 billion tokens @xcite . in this paper, the largest dataset we consider has 41 million tokens, and the smallest only 1813 tokens. we argue that different evaluation techniques are needed for datasets like ours that are more than 1000 times smaller. figure 1 shows that the oddoneout and topk tasks are better suited than the analogy task to measure improvement in embedding quality with datasets like these. ## evaluation methods the oddoneout and topk tasks are simple and widely applicable. both tasks require a test set consisting of a list of categories, where each category contains a list of words belonging to that category. figure 4 shows some example categories generated through a fully automated process (described in section 4.1). the topk task measures a model's ability to identify words that are related to each category, and conversely the oddoneout task measures a model's ability to identify words that are unrelated to each category, formally, assume that there are m categories, that each category has n words foot_1 , that there are v total words in the vocabulary, and that the words are embedded into r d . the method of generating the embedding (e.g. word2vec, glove, fasttext) does not matter. let c i,j be the jth word in category i, and let c @xmath0,1 , c i,2 , ..., c i,n } be the ith category. ## experiments we demonstrate the usefulness of our evaluation metrics with two experiments. first, we show that the oddoneout and topk metrics are better measures of word embedding quality than the analogy metric in the low-resource regime using simulated english data. second, we show that the oddoneout and topk metrics are useful for model selection in an emoji embedding task where the analogy task is not well defined. this experiment also demonstrates that the oddoneout and topk metrics correlate with downstream task performance. ## conclusion in this paper we introduce the first word vector evaluation methods designed specifically for the lowresource domain, oddoneout and topk, along with a method for automatically producing test sets in wikidata's 581 supported languages. we believe wikidata is an underutilized resource in the nlp evaluation community, and in particular, that its massively multilingual support can be used to better serve under resourced languages.