NPV ec1: Word Embeddings for N epali - Construction and Evaluation (2021)

Abstract: Word Embedding maps words to vectors of real numbers. It is derived from a large corpus and is known to capture semantic knowledge from the corpus. Word Embedding is a critical component of many state-of-the-art Deep Learning techniques. However, generating good Word Embeddings is a special challenge for low-resource languages such as Nepali due to the unavailability of large text corpus. In this paper, we present NPVec1 which consists of 25 state-of-art Word Embeddings for Nepali that we have derived from a large corpus using Glove, Word2Vec, FastText, and BERT. We further provide intrinsic and extrinsic evaluations of these Embeddings using well established metrics and methods. These models are trained using 279 million word tokens and are the largest Embeddings ever trained for Nepali language. Furthermore, we have made these Embeddings publicly available to accelerate the development of Natural Language Processing (NLP) applications in Nepali.

## introduction recent deep learning (dl) techniques provide state-of-the-art performances in almost all natural language processing (nlp) tasks such as text classification @xcite @xcite , question answering @xcite , named entity recognition @xcite and sentiment analysis @xcite . dl techniques are attractive due to their capacity of learning complex and intricate features automatically from the raw data @xcite . this significantly reduces the required time and effort for feature engineering, a costly step in traditional feature-based approaches which further requires considerable amount of engineering and domain expertise. thus, dl techniques are very useful for low-resource languages such as nepali. many deep learning techniques require word embeddings to represent each word by a vector of real numbers. word embeddings learn a meaningful representation of words directly from a large unlabeled corpus using co-occurrence statistics @xcite . the closer the word representations to actual meanings, the better the performance. consequently, word embeddings have received special attention from the research community and are predominantly used in current nlp researches. word embeddings can generally be divided into two categories: context-independent embeddings such as glove @xcite , word2vec @xcite , and fasttext @xcite , and context-dependent embeddings such as bert (bidirectional encoder representations from transformers) @xcite and elmo (embeddings from language models) @xcite . context-dependent word embedding is generated for a word as a function of the sentence it occurs in. thus, it can learn multiple representations for polysemous words @xcite . to learn these deep contextualized representations, bert uses a transformer based architecture pretrained on masked language modelling and next sentence prediction tasks, whereas, elmo uses a bidirectional lstm architecture for combining both forward and backward language models. in this paper, we present npvec1, a suite of word embedding resources for nepali, a lowresource language, which is the official language and de-facto lingua franca of nepal. it is spoken by more than 20 million people mainly in nepal and many other places in the world including bhutan, india, and myanmar @xcite . even though word embeddings can be directly learned from raw texts in an unsupervised fashion, gathering a large amount of data for its training remains a huge challenge in itself for a low-resource lan-guage such as nepali. in addition, nepali is a morphologically rich language which has multiple agglutinative suffixes as well as affix inflections and thus proves challenges during its preprocessing i.e. tokenization, normalization and stemming. we have collected data over many years and combined it with multiple other publicly available data sets to generate a suite of word embeddings, i.e. npvec1, using glove, word2vec, fasttext and bert. it consists of 25 word embeddings corresponding to different preprocessing schemes. in addition, we perform the intrinsic and extrinsic evaluations of the generated word embeddings using well established methods and metrics. our pretrained embedding models and resources are made publicly available foot_0 for the acceleration and development of nlp research and application in nepali language. the novel contributions of this study are: the rest of this paper is organized as follows. we review related works in section 2. we describe the data collection and corpus construction in section 3. we describe our experiments to develop word embedding methods in section 4. we present model evaluations in section 5 and conclusion and future directions in section 6. ## related works word embeddings provide continuous word representations and are the building blocks of many nlp applications. they capture distributional information of words from a large corpora. this information helps the generalization of machine learning models especially when the data set is limited @xcite . word embedding tools, technologies and pre-trained models are widely available for resource rich languages such as english @xcite @xcite and chinese @xcite . due to the wide use of word embeddings, pre-trained models are increasingly available for resource poor languages such as portuguese @xcite , arabic @xcite , and bengali @xcite . most word embedding algorithms are unsupervised. which means that they can be trained for any language as long as the corpus data is available. one such effort is by @xcite who generated and made available word vectors for 157 languages, including nepali, using wikipedia and common crawl data. the pre-trained models for skip-gram and cbow are available at https: //fasttext.cc . another useful resource is http: //vectors.nlpl.eu/repository which is a community repository for word embeddings maintained by language technology group at the university of oslo @xcite . it currently hosts 209 pre-trained word embeddings for most languages but not nepali. word embeddings for nepali are derived in small scale by @xcite using fasttext and by @xcite using word2vec. both of these efforts have major limitations. first, they have limited diversity in the corpus. @xcite wikipedia and common crawl data while lamsal uses news corpus. second, their corpus is very small compared to ours (section 3). third, they do not provide any evaluation of the generated models. fourth, they have done limited or no prepossessing on the data. we show later in section 3.3 that tokenization and text normalization are critical for processing morphologically rich nepali text. in contrast, we have conducted a large scale study of word embeddings in more diverse and large data sets using glove, fasttext, and word2vec. our corpus is nearly four times bigger than the corpus used by aforementioned approaches (see section 3). we have constructed 8 inputs for each combination of binary variables: tokenization, normalization and stemming which has resulted in 24 pre-trained embeddings for glove, word2vec, and fasttext combined. additionally, we have trained bert for one of these preprocess-ing schemes and performed intrinsic and extrinsic evaluations for each of these 25 models. ## corpus preparation in this section, we present our data sources and preprocessing techniques for the corpus. to help readers understand the nepali words used in this paper, we have provided a gloss in section 8 with their transliterations and english translations. ## conclusion and future work in this paper, we trained 25 word embedding models for nepali language with multiple preprocessing schemes and made them publicly available for accelerating nlp research in low-resource language nepali foot_6 . this, to our knowledge, is the first formal and large scale study of word embeddings in nepali. we compared the performances of these models using intrinsic and extrinsic evaluation tasks. our findings clearly indicate that these word embedding models perform exceptionally well in identifying related words compared to discovering semantically similar words. we also suggest that further comparisons be made with an improved stemmer, which has fewer over-stemming error rates than what we've used, to study the effects of over-stemming in word embeddings. performance of these word embeddings in clustering of related words also suggest us that these models will obtain good results in tasks such as named entity recognition and pos tagging. this is something that we would like to explore in future. as far as our study with bert goes, we obviously recommend training the original bert architecture, rather than what we have used, with more data. for comparison, the original bert model was trained on a total of 3.3 billion words whereas we've trained our model in just 360 million words. unfortunately, for a resource poor language like nepali, this is not a trivial task. similarly, it would be most interesting to see performances of other context-dependent embedding models such as elmo, gpt2 @xcite , xlnet @xcite and roberta @xcite in case of nepali language. original transliteration meaning नाला naala rivulets झरना jharana waterfall गोर टो goreto trail बाटो baato road घ ghumti bend चौतारा chautara rest area या ा yatra travel हउ hiu snow ह रयाल hariyali greenery द उराल deurali hilltop ताल taal lake उकाल ukali uphill रा ो ramro good स ो sasto inexpensive जा ग रलो jagarilo energetic ठ लो thulo big अ ो aglo tall सफा safaa clean हल को haluko lightweight कोमल komal soft उ ालो ujyalo bright ब मान buddhiman wise सा चो sacho truth लाभ laabh gain नश nisulka free छटो cheeto fast सफल safal successful अथ aartha meaning ाय nyaya justice स म sakchyam capable धन dhani rich नरा ो naramro bad महगो mahango expensive पातलो patalo skinny सानो saano small होचो hocho short फोहोर fohor waste भार bhaari heavy कठोर kathor hard अँ ारो adhyaro dark अ छ alche lazy म ख murkha fool झ ठो jhutho lies हा न haani damage सस sasulka not-free ढलो dhilo late असफल asafal failure अनथ anartha meaningless अ ाय anyaya injustice अस म asakchyam incompetent ग रब gareeb poor