Aligning Word Vectors on Low-Resource Languages with W iktionary (2022)

Abstract: Aligned word embeddings have become a popular technique for low-resource natural language processing. Most existing evaluation datasets are generated automatically from machine translations systems, so they have many errors and exist only for high-resource languages. We introduce the Wiktionary bilingual lexicon collection, which provides high-quality human annotated translations for words in 298 languages to English. We use these lexicons to train and evaluate the largest published collection of aligned word embeddings on 157 different languages. All of our code and data is publicly available at https://github.com/mikeizbicki/wiktionary_bli .

## introduction a bilingual lexicon is a mapping of words from a source language into a target language. the bilingual lexicon induction (bli) problem is the task of learning such a mapping from data. most recent solutions to this problem follow a two step procedure: first, train word vectors on a large monolingual corpus for each language individually using a standard algorithm like word2vec @xcite , glove @xcite , or fasttext @xcite . then, learn a transformation that aligns these two vector spaces into a common space (e.g. @xcite @xcite @xcite @xcite . the bli problem is then solved by performing nearest neighbor queries in the common space. the focus of this work is the ground truth bilingual lexicon used to train and evaluate these models. recent previous work has relied on the muse lexicon collection @xcite . this collection provides bilingual lexicons between 45 languages and english. this lexicon is generated from a machine translation system, and so suffers from a number of problems. first, many of the mappings in the lexicon do not contain real words in either the source or target language (see figure 1 for examples from thai). second, the distribution of words is inconsistent between languages, with many languages containing only proper nouns in their training and test sets. due to these problems, @xcite suggest that future research "avoids drawing conclusions from quantitative results on this bli dataset." other datasets (described in section 2 below) have even worse limitations. this paper introduces a new bilingual lexicon collection based on wiktionary. wiktionary contains more than 7 million words in 8166 languages and has been collaboratively edited by 3.9 million users. 1 our specific contributions are: thai "word" english "translation" แคลอรี ่ calories โคมลอย lanterns univ univ bdfutbol bdfutbol efm efm พล็ อต plot getparent getparent roca roca เป๊ ะ exactly annie annie figure 1 : the last 10 data points for the thai test files in the widely used muse dataset @xcite . these translation pairs were machine generated without any human input, and this results in bad translation pairs. for example, thai words should always written in the thai script, but many words are written in latin script. words like getparent do not even correspond to words in any natural language and are an artifact of javascript code incorrectly included in the original source material. our wiktionary dataset contains only high-quality human verified translations and so does not have these problems. azerbaijani, basque, belarusian, esperanto, galician, georgian, malayalam, norwegian nynorsk, serbian, serbo-croatian, and welsh) and thus potentially suitable for downstream cross-lingual tasks. the remainder of this paper is organized as follows. section 2 discusses related work. section 3 describes the lexicon construction procedures. section 4 experimentally demonstrates that the resulting lexicons are of high quality, and trains the new models. wiktionary. wiktionary is a valuable resource and widely used by the nlp community. a google scholar search for "wiktionary" produces 21 000 results on diverse tasks such as synonym detection @xcite , idiom extraction @xcite , and word sense disambiguation @xcite . the prior works most closely related to our own are general purpose information extractors (e.g. @xcite @xcite @xcite . although these extractors can be used to extract translation information, they have not been used explicitly for the purpose of generating datasets for machine translation problems like the bli problem. alternative datasets. many datasets have been proposed for the training and evaluation of word vectors. prior to the muse lexicons @xcite , papers studying bli all used their own ad-hoc datasets. for example @xcite introduce spanish-english and czech-english lexicons; @xcite introduce an italian-english lexicon; @xcite introduce german-english and finish-english lexicons; and @xcite introduce spanish-english and chinese-english lexicons. since the introduction of muse, @xcite followed a similar procedure to create an additional 28 bilingual lexicons for high-resource non-english language pairs. using a machine translation system makes it impossible to create lexicons for low-resource languages without introducing serious mistakes as seen in figure 1 . furthermore, inter-language comparisons should not be done because the topics covered by the languages' test sets vary considerably @xcite . our wiktionary dataset fixes all of these problems. ## dataset overview in this section, we first describe the data extraction process, then we describe how we split the data into training and test sets. both steps use languageagnostic approaches. our goal is to make the data for each language as similar as possible so that cross-lingual evaluations can be made in a fair and consistent manner. ## experiments we perform three experiments. the first experiment measures the importance of the size of the bli training dataset on model performance. the second experiment compares the quality of the muse and wiktionary lexicons. the final experiment trains bli models on 112 new, previously unstudied languages. for all experiments, we align the common crawl vectors provided by @xcite to the english-language vectors trained on the common crawl provided by @xcite . we use we use this non-standard setting because our preliminary tests (not shown) found it to give significantly better results for the low-resource languages that we study in section 4.3 and equivalent results for the high-resource languages.