D ia L ex: A Benchmark for Evaluating Multidialectal A rabic Word Embeddings (2021)

Abstract: Word embeddings are a core component of modern natural language processing systems, making the ability to thoroughly evaluate them a vital task. We describe DiaLex, a benchmark for intrinsic evaluation of dialectal Arabic word embeddings. DiaLex covers five important Arabic dialects: Algerian, Egyptian, Lebanese, Syrian, and Tunisian. Across these dialects, DiaLex provides a testbank for six syntactic and semantic relations, namely male to female, singular to dual, singular to plural, antonym, comparative, and genitive to past tense. DiaLex thus consists of a collection of word pairs representing each of the six relations in each of the five dialects. To demonstrate the utility of DiaLex, we use it to evaluate a set of existing and new Arabic word embeddings that we developed. Beyond evaluation of word embeddings, DiaLex supports efforts to integrate dialects into the Arabic language curriculum. It can be easily translated into Modern Standard Arabic and English, which can be useful for evaluating word translation. Our benchmark, evaluation code, and new word embedding models will be publicly available.

## introduction word embeddings are the backbone of modern natural language processing (nlp) systems. they encode semantic and syntactic relations between words by representing them in a low-dimensional space. many techniques have been proposed to learn such embeddings @xcite @xcite from large text corpora. as of today, a 1 https://github.com/ubc-nlp/dialex . large number of such embeddings are available in many languages including arabic. due to their importance, it is vital to be able to evaluate word embeddings, and various methods have been proposed for evaluating them. these methods can be broadly categorized into intrinsic evaluation methods and extrinsic evaluation ones. for extrinsic evaluation, word embeddings are assessed based on performance in downstream applications. for intrinsic evaluation, they are assessed based on how well they capture syntactic and semantic relations between words. although there exists a benchmark for evaluating modern standard arabic (msa) word embeddings @xcite , no such resource that we know of exists for arabic dialects. this makes it difficult to measure progress on arabic dialect processing. in this paper, our goal is to facilitate intrinsic evaluation of dialectal arabic word embeddings. to this end, we build a new benchmark spanning five different arabic dialects, from eastern, middle, and western arab world. namely, our benchmark covers algerian (alg), egyptian (egy), lebanese (leb), syr-ian (syr), and tunisian (tun). figure 1 shows a map of the five arab countries covered by di-alex. for each one of these dialects, dialex consists of a set of word pairs that are syntactically or semantically related by one of six different relations: male to female, singular to plural, singular to dual, antonym, comparative, and genitive to past tense. overall, dialex consists of over 3, 000 word pairs in those five dialects, evenly distributed. to the best of our knowledge, dialex is the first benchmark that can be used to assess the quality of arabic word embeddings in the five dialects it covers. to be able to use dialex to evaluate arabic word embeddings, we generate a set of word analogy questions from the word pairs in dialex. a word analogy question is generated from two word pairs from a given relation. these questions have recently become the standard in intrinsic evaluation of word embeddings @xcite @xcite . to demonstrate the usefulness of dialex in evaluating arabic word embeddings, we use it to evaluate a set of existing and new arabic word embeddings. we conclude that both available and newly-developed word embedding models have moderate-to-serious coverage issues and are not sufficiently representative of the respective dialects under study. in addition to the benchmark of word pairs, our newlydeveloped dialectal arabic word embeddings will also be publicly available. 2 beyond evaluation of word embeddings, we envision dialex as a basis for creating multidialectal arabic resources that can facilitate study of the semantics and syntax of arabic dialects, for example for pedagogical applications @xcite . more broadly, we hope dialex will contribute to efforts for integrating dialects in the arabic language curriculum (al-batal, 2017). di-alex can also be used to complement a growing interest in contextual word embeddings @xcite and self-supervised language models @xcite , including in arabic @xcite @xcite . extensions of dialex can also be valuable for nlp, for example, dialex can be easily translated into msa, other arabic dialects, english, or other languages. this extension can enable evaluation of word-level translation sys-tems, including in cross-lingual settings @xcite . our resources can also be used in comparisons against contextual embeddings @xcite and embeddings acquired from language models such as bert @xcite . for example, it can be used in evaluation settings with arabic language models such as arabert @xcite , gigabert @xcite , and the recently developed arbert and mar-bert @xcite . more generally, our efforts are motivated by the fact that the study of arabic dialects and computational arabic dialect processing is a nascent area with several existing gaps to fill @xcite @xcite @xcite . the rest of the paper is organized as follows. in section 2, we describe how dialex was constructed. section 3 offers our methodical generation of a testbank for evaluating word embeddings. in section 4, we provide a case study for evaluating various word embedding models, some of which are newly developed by us. section 5 is about related work. section 6 is where we conclude and present future directions. ## benchmark construction dialex consists of a set of word pairs in five different arabic dialects and for six different semantic and syntactic relations, namely male to female, singular to plural, singular to dual, antonym, comparative, and genitive to past tense. we chose only this set of relations as they are standard in previous literature. in addition, they are comprehensive enough to reflect dialect specificity. a good word embeddings model should thus have close representation of word pairs for each of these relations in the embeddings space. for each dialect, word pairs were manually generated by at least one native speaker of the dialect. each person independently came up with the word pairs representing a given relation based on their knowledge of the dialect and while trying to include words that are typically representative and unique in that dialect. that is, to the best of our ability, the words were chosen so that they are frequently-used words in the dialect and are not the same as in msa. one challenge we faced when generating the word pairs was orthographic variation. for example, consider the antonym of the word ("behind") in the egyptian dialect. it can be written as or ("in front of") . we decided to include all variations of the same word in the benchmark. a second challenge was the need to present the relationship using more than one word. for instance, consider algerian and the relationship dual, the word ("pair") is sometimes used to describe two items of something. so, for instance, for the word ("hour"), the dual can be either ("pair of hours") or ("two hours"). again, we opted for including both variants in the benchmark. overall, for each dialect and each relation, around 100 word pairs were generated. ## testbank for evaluating word embeddings and evaluation metric given the word pair lists in dialex, we generate a testbank consisting of 260,827 tuples. each tuple consists of two word pairs (a, b) and (c, d) from the same relation and the same dialect. for each of our five dialects and for each of our six relations, we generate a tuple by combining two different word pairs from the same relation in the same dialect. once tuples have been generated, they can be used as word analogy questions to evaluate different word embeddings as defined by mikolov et al. @xcite . a word analogy question for a tuple consisting of two word pairs (a, b) and (c, d) can be formulated as follows: "a to b is like c to ?". each such question will then be answered by calculating a target vector @xmath0. we then calculate the cosine similarity between the target vector t and the vector representation of each word w in a given word embeddings v . finally, we retrieve the most similar word w to t, i.e., argmax @xmath1,b,c} w•t ||w||||t|| . if @xmath2.e., the same word) then we assume that the word embeddings v has answered the question correctly. moreover, we extend the traditional word analogy task by taking into consideration if the correct answer is among the top k, with @xmath3, 10}, closest words in the embedding space to the target vector t, which allows us to more leniently evaluate the embeddings. this is particularly important in the case of arabic since many forms of the same word exist, usually with additional prefixes or suffixes such as the equivalent of the article "the" or possessive determiners such as "her", "his", or "their". for example, consider one question which asks to is like to " ?", i.e., "man" to "woman" is like "prince" to "?", with the answer being " " or "princess". now, if we rely only on the top-1 word and it happens to be " " which means "for the princess" in english, the question would be considered to be answered wrongly. to relax this, and ensure that different forms of the same word will not result in a mismatch, we use the top-5 and top-10 words for evaluation rather than just the top-1. note that we consider a question to be answered wrongly if at least one of the words in the question are not present in the word embeddings. that is, we take into consideration the coverage of the embeddings as well @xcite . finally, we report the number of questions that were answered correctly over the total numbers of questions available. that is, assume the number of questions for a given dialect and a given relation is n, and assume that a given embeddings model m correctly answered m out of those n questions as explained above. then, the accuracy of the model dialect male-fem sing-pl sing-dual algerian ---msa ---eng. rooster -chicken sandwich -two sandwiches shoe -pair of shoes egyptian ---msa ---eng. nice (m) -nice (f) room -rooms ear -two ears lebanese ---msa ---eng. man -woman lighter -lighters spoon -two spoons syrian ---msa ---eng. mouse (m) -mouse (f) cake -cakes belt -belts tunisian ---msa ---eng. handsome -beautiful sheep (sing) -sheep (pl) curtain -two curtains dialect antonym comparative genitive-past tense algerian ---msa ---eng. strong -weak nosy -nosier promenade -promenaded egyptian ---msa ---eng. back -front bad -worse cried -crying lebanese ---msa ---eng. restful -restless handsome -more handsome entered -entering syrian ---msa ---eng. push -bump into smart -smarter inspect -inspection tunisian ---msa ---eng. unlucky -lucky poor -poorer slap -slapping m will be m n . ## evaluation of arabic word embeddings using dialex in this section, we demonstrate how dialex can be used to evaluate word embeddings across the different dialects it covers. particularly, we evaluate two large word embeddings models based on word2vec @xcite released by @xcite . one model is based on skip grams (zah sg) and the other is a continuous bag-of-words (zah cbow). both of these models have a vocabulary size of 626, 3435 words. we also create four cbow word2vec models, all of which have 300 word vector embedding dimensions, as we describe next. ## conclusion we described dialex, a benchmark for evaluating dialectal arabic word embeddings. dialex comes in five major arabic dialects, namely algerian, egyptian, lebanese, syrian, and tunisian. across these dialects, dialex offers a testbank of word pairs for six syntactic and semantic relations, namely male to female, singular to dual, singular to plural, antonym, comparative, and genitive to past tense. to demonstrate the utility of di-alex, we used it to evaluate a set of available and newly-developed arabic word embedding models. our evaluations are intended to showcase the utility of our new benchmark. dialex as well as the newly-developed word embeddings will be publicly available. dialex can be used to support integration of dialects in the arabic language curriculum, and for the study of the syntax and semantics of arabic dialects. it can also complement evaluations of contextual word embeddings. in the future, we plan to use dialex to for more extensive evaluation of all publicly-available arabic word embedding models. we will also train a dialectal arabic word embeddings model with a larger dataset and evaluate it using dialex. finally, we will translate dialex into msa and english to facilitate use of the resource for evaluating word translation.