Pre-tokenization of Multi-word Expressions in Cross-lingual Word Embeddings (2020)

Abstract: Cross-lingual word embedding (CWE) algorithms represent words in multiple languages in a unified vector space. Multi-Word Expressions (MWE) are common in every language. When training word embeddings, each component word of an MWE gets its own separate embedding, and thus, MWEs are not translated by CWEs. We propose a simple method for word translation of MWEs to and from English in ten languages: we first compile lists of MWEs in each language and then tokenize the MWEs as single tokens before training word embeddings. CWEs are trained on a word-translation task using the dictionaries that only contain single words. In order to evaluate MWE translation, we created bilingual word lists from multilingual WordNet that include single-token words and MWEs, and most importantly, include MWEs that correspond to single words in another language. We release these dictionaries to the research community. We show that the pre-tokenization of MWEs as single tokens performs better than averaging the embeddings of the individual tokens of the MWE. We can translate MWEs at a top-10 precision of 30-60%. The tokenization of MWEs makes the occurrences of single words in a training corpus more sparse, but we show that it does not pose negative impacts on single-word translations.

## introduction cross-lingual word embeddings (cwes) are realvalued vector representations of words in multiple languages placed in a shared vector space, with the intention that words with closer meanings have closer locations in the vector space. first, monolingual word embeddings are trained based on the hypothesis of distributional semantics @xcite that context approximates meaning. they are learned from data in a way that words used in similar contexts have similar vectors. following that, the monolingual word embeddings are aligned to produce cwes. cwes are an essential building block in modern cross-lingual methods and can also be used to induce bilingual lexicons from a small seed dictionary @xcite . an important and overlooked fact is that before cwes are trained, the corpus is pre-processed by a word tokenizer. this illustrates a clear limitation of the state-of-the-art cwes: they can only align words that happen to be considered as single tokens by the word tokenizer. multi-word expressions (mwes) are combinations of orthographic words, whose meaning, form, use, or distribution is non-compositional or unpredictable in some way @xcite . they come in diverse forms such as compound nouns (dance floor), named entities (united states), phrasal verbs (give up), and connectives (as well as). word tokenizers do not recognize mwes as single units but rather as a sequence of their components, a deficiency carried into cwe construction. in this position paper, we argue that the token units of word embeddings should be discussed more carefully, and, in particular, that mwes should be recognized as single units before training and evaluating word embeddings. in cross-lingual applications, mwes are particularly important. a single token in one language is often translated into an mwe in another language. so, failure to tokenize mwes is a critical flaw of cwes in the task of word translation and presumably in other cross-lingual tasks as well. some studies @xcite have suggested representing phrase and sentence embeddings by taking the average or sum of their component word vectors. however, such a simple approach is not sufficient, as the meaning 1 ). english word embeddings trained with single-word tokenization (2) do not have united states in the vocabulary, and we represent its embedding by the average embedding. word embeddings with mwe tokenization ( ) assigns a unique embedding to united states, which is better aligned with its chinese translation 美国. note that the configuration of single-word embeddings also changes by having mwe embeddings. to illustrate the advantage of having mwes in the cwe vocabulary, we compare the alignments of english-chinese cwes with and without mwe tokens (figure 1 ). with mwe-based tokenization, the single token united states aligns with 美国 (united states; meiguo) with a high cosine similarity of 0.82. the pre-tokenization of united states into a single token solves additional problems as well. when we treat united states as two separate tokens, we distort the embeddings of united and states. on the left sides of the arrows in a lexicon-based approach to mwe detection comes with another advantage. supervised methods for mwe detection require annotated texts @xcite , which may not be available for all languages. on the other hand, the high availability of lexical resources containing mwes in many languages, such as wiktionary and wordnet, makes a lexicon-based approach for mwe detection possible in many languages. our focus in this paper is not to study the automatic extraction of mwes, but rather to establish that tokenization of mwes can contribute to improvements in cwe. since mwe lexicons exist for the languages we are interested in, we have used those for the time being. of course, using automatically discovered mwes would be an interesting direction for future research. to explore the effect of pre-tokenization of mwes, we evaluate cwes in the task of word translation between english and 10 languages, arabic), bulgarian, chinese, german, hebrew, hindi, japanese, russian, spanish, and turkish, which span a wide typological variety. we find that our simple lexicon-based tokenization can align embeddings of mwes at a precision@10 score of 30-60% without negative impacts on single word translation. furthermore, we find some single-token words are correctly translated into mwes, which are not attested in the common evaluation practice. in summary, we argue that cwe studies should consider mwes in development and evaluation. mwes are pervasive in many languages and should not be ignored when the alignment of words is discussed. we present a lexicon-based method to this end ( §3-4) and show its effectiveness in the task of word translation ( §5). we have created a new word translation dataset that contains mwes ( §3.2). the dataset is in ten language pairs and contains mwes in addition to single orthographic tokens. foot_1 2 related work ## data creation this section describes the methods we used for creating the data that we are releasing with this paper: (1) monolingual lists of mwes in eleven languages for pre-tokenizing mwes in corpora and (2) bilingual dictionaries (ten languages each paired with english) for evaluating the resulting mwe embeddings in the word translation task. the languages are arabic (ar), bulgarian (bg), chinese (zh), english (en), german (de), hebrew (he), hindi (hi), japanese (ja), russian (ru), spanish (es), and turkish (tr) ## training cwes: components this section describes our pipeline for training cwes, including the following three steps (figure 2 ): (1) identifying mwes in a corpus, (2) training monolingual word embeddings, and (3) aligning embeddings across languages. ## experiments to examine the effect of pre-tokenization of mwes, we conduct the task of word translation between each of the ten languages and english, in both directions. a word embedding in a source language is projected into the embedding space of a target language using a trained linear mapping w ( §4.3). the translation candidates of the source word are retrieved by k-nearest neighbor search in terms of csls. the performance is measured by top-k precision (precision@k). 7 our evaluation involves two tasks. in the first task, we focused on the translation of mwes using our new evaluation dictionaries that contain tokenized mwes ( §3.2). in the second task, we evaluated the translation of single words on the existing benchmark, muse @xcite to investigate the influence on single word embeddings of pre-tokenizing mwes. ## conclusion we studied the impact of pre-tokenizing mwes on cross-lingual alignments of word embeddings. we found that simple lexicon-based tokenizations can align embeddings of mwes at a high precision without breaking alignments of single-words. we believe our results will motivate researchers to pay more attention to the existence of mwes and how they are aligned across languages. ideally, the real mwes have higher scores, and non-mwe phrases have lower scores. examining this, however, is not easy. it is very expensive to manually check all the candidates in step 1. so, in our experiments, we aimed to obtain a rough estimate using the mwe lists we compiled. the phrases in our lists are true positives and should be assigned high association scores. figure 3 shows the results. the horizontal axis denotes the dice coefficients calculated in step 3, and the vertical axis shows the number of mwes in each bin of dice coefficients. the orange bars shows the number of mwes that exist in our eomw+parseme lexicon, which are true positives. this result gives us two important implications.