On the Importance of Tokenization in A rabic Embedding Models (2020)

Abstract: Arabic, like other highly inflected languages, encodes a large amount of information in its morphology and word structure. In this work, we propose two embedding strategies that modify the tokenization phase of traditional word embedding models (Word2Vec) and contextual word embedding models (BERT) to take into account Arabic’s relatively complex morphology. In Word2Vec, we segment words into subwords during training time and then compose word-level representations from the subwords during test time. We train our embeddings on Arabic Wikipedia and show that they perform better than a Word2Vec model on multiple Arabic natural language processing datasets while being approximately 60% smaller in size. Moreover, we showcase our embeddings’ ability to produce accurate representations of some out-of-vocabulary words that were not encountered before. In BERT, we modify the tokenization layer of Google’s pretrained multilingual BERT model by incorporating information on morphology. By doing so, we achieve state of the art performance on two Arabic NLP datasets without pretraining.

## introduction word embeddings are one of the main building blocks of most natural language processing tasks. many word embedding techniques have been proposed @xcite @xcite that try to capture better word representations. although most of the approaches are language agnostic, they were historically designed to be used with english. nonetheless, it was shown that these techniques work well in other languages @xcite . one feature that is unique to arabic, and other highly inflected languages, is the expressiveness of its words. the fact that arabic encodes a large amount of information in its word structure leads to potential problems in learning embeddings due to the large number of forms for each word, the more likely chances of out-of-vocabulary (oov) instances, and the increase in model size. in this work, we propose two embedding strategies for arabic that take into consideration its rich morphology by modifying the tokenization phase. the first technique concerns traditional embedding models and the second a contextual one. our experiments are done on word2vec @xcite and bert @xcite since they are the most popular traditional and contextual embedding techniques, respectively. nonetheless, the approaches we propose are embedding-agnostic and can be applied to other embedding techniques. figure 1 summarizes our two approaches and illustrates what happens in the training and inferences stages of each approach. in traditional embeddings, we analyze the effect of tokenizing words into subwords by splitting their suffixes and prefixes before training an embedding model and then combining these subwords using an algorithm we propose. we show that by doing so we get: in contextual embeddings, we investigate different tokenization schemes when using bert without needing any pretraining, in contrast to previous approaches such as @xcite . we simply modify the tokenization part of google's pretrained multilingual bert model @xcite resulting in models that: the rest of the paper is structured as follows: section 2 introduces the background and explains some of the fundamental ideas of word embeddings; section 3 defines the process of gathering and cleaning our data; section 4 highlights the embedding approaches we are proposing; section 5 details the experiments and the results; section 6 discusses related work; and section 7 concludes by summarizing our findings and pointing to potential directions for future work. ## background word embedding techniques rely on the distributional hypothesis @xcite which suggests that words that appear in similar contexts tend to have similar meanings. @xcite popularized word embeddings when they introduced word2vec and showed that it produces representations that capture not only syntax but also words' semantics. many related techniques have been produced after that @xcite @xcite . the word vectors produced by such techniques capture interesting semantics; one popular example is how the the vectors capture relationships between them. for example, if we subtract the vector for 'man' from the vector of 'king', and then add the vector of 'woman' we get very close to the vector of 'queen'. contextual word embeddings were proposed @xcite @xcite ) to tackle the issues that arise from words have multiple senses and meanings. in traditional embeddings such as word2vec, each word is encoded in a vector, which is a fixed representation. this may cause problems with homographs and words that have multiple senses depending on the context. for example the wear 'bear' means two very different things in the following sentences: "the right of the people to keep and bear arms shall not be infringed." and "a wild bear was seen in the city." yet, traditional embeddings will only capture one fixed representation. contextual word embeddings aim to solve this issue by modeling embeddings where the context of the word will affect its generated representation. verb forms go go, went, going, gone, goes ## data in this section, we describe the process of gathering and preparing the data used for training the embeddings. we do not require any training data for contextual embeddings since we do not require pretraining: we use google's multilingual bert model @xcite , which supports 104 languages (including arabic) and was trained on their respective wikipedia dumps. for traditional embeddings, we use arabic wikipedia as a corpus. we downloaded the wikipedia dump from january 2018 and then cleaned it by using wikiextractor foot_0 , which is a utility that generates plain text from an xml formatted wikipedia dump. we then use a custom set of regexes to filter out all non-arabic words, such as english words and numbers, and remove all diacritics and kashidas resulting in over 86 million tokens. ## approach one example of arabic's morphological complexity is in the number of verb forms it possesses which is much higher than in english as we can see in ## experiments and results in this section we detail our experiments and highlight the results produced by the models we proposed. ## related work many works have noted how sensitivity to arabic's morphological complexity can result in better performance in standard nlp tasks. however, we do not know of any previous work that has specifically focused on the effect of tokenization on different arabic embedding models. antoun et al. (2020) trained an arabic specific bert model. they also trained another arabic bert model in which they segment the text before training the model and showed that it usually improves performance. @xcite analyzed morphological relations in arabic word embeddings. they noted that some morphological features are captured in embeddings representations. @xcite proposed utilizing subword information in training embeddings to enrich the representations and showed that it improves the performance on word similarity tasks. @xcite investigated morphological-based embeddings and lemma-based embeddings. they utilized part-of-speech information to train their embeddings, similar to @xcite , and then build lemma-based embeddings from them by aggregating on different senses of each word first and them combining words that share the same lemma. @xcite tackled the problem of extracting roots of words and proposed an extension to fasttext @xcite that utilize morphemes. ## conclusion we show that tokenization that pays attention to arabic's morphology can create better traditional and contextual embedding models. breaking words into subwords in word2vec not only leads to an increase in performance and reduction in the vocabulary size and hence the model size, but also provides a simple way to produce good out-of-vocabulary representations. we also show the importance of tokenization in bert where we were able to achieve impressive performance without requiring any pretraining. one possible future work would be to investigate tokenization's effect in other morphologically rich languages such as hebrew and turkish and see if our results can be generalized to other highly inflected languages.