Towards B engali Word Embedding: Corpus Creation, Intrinsic and Extrinsic Evaluations (2020)

Abstract: Distributional word vector representation or word embedding has become an essential ingredient in many natural language processing (NLP) tasks such as machine translation, document classification, information retrieval and question answering. Investigation of embedding model helps to reduce the feature space and improves textual semantic as well as syntactic relations. This paper presents three embedding techniques (such as Word2Vec, GloVe, and FastText) with different hyperparameters implemented on a Bengali corpus consists of 180 million words. The performance of the embedding techniques is evaluated with extrinsic and intrinsic ways. Extrinsic performance evaluated by text classification, which achieved a maximum of 96.48% accuracy. Intrinsic performance evaluated by word similarity (e.g., semantic, syntactic and relatedness) and analogy tasks. The maximum Pearson (rˆ) correlation accuracy of 60.66% (Ssrˆ) achieved for semantic similarities and 71.64% (Syrˆ) for syntactic similarities whereas the relatedness obtained 79.80% (Rsrˆ). The semantic word analogy tasks achieved 44.00% of accuracy while syntactic word analogy tasks obtained 36.00%.

## introduction word embedding is a distributional vector representation of words in which syntactic and semantic interpretations are derived from the enormous amount of unlabeled texts. recently, the word embedding is considered as a powerful tool due to its many applications in nlp, thus, gained much attention by nlp experts. this is a growing up research issue for well-resourced language like english, where an embedding algorithm generates a model @xcite . however, it is a very complicated task to adopt an embedding algorithm (of any language) directly for the resourceconstrained languages such as bengali due to the scarcity of resources. as a result, the low-resource language is trail end in nlp tools development. bengali is the most popularly communicated language of bangladesh while second most communicated of the 22 official languages of india which makes bengali is the 7 th most spoken language in the world @xcite . however, due to the shortage of resources, the development of nlp tools are striving. bengali speaking people are suffering to access modern nlp tools that might be affected their sustainable use of language technologies. therefore, bengali word embedding is an essential prerequisite to developing any bengali language based nlp tools. there are two well-known evaluation methods used extensively to evaluate embedding techniques such as extrinsic and intrinsic @xcite . extrinsic evaluation refers to downstream tasks like as machine translation (mt) @xcite , and part of speech (pos) tagging @xcite . intrinsic evaluation goals to evaluate the quality of language processing tasks, such as semantic and syntactic word similarity @xcite , word relatedness @xcite , and word analogy @xcite . unavailability of standard bengali embedding corpus and inadequacy of resources are antecedents that make such a model generation and evaluation very challenging. moreover, there is no generalized embedding model available to date for bengali downstream tasks. thus, the proposed work introduces a bengali embedding model generation and evaluation techniques with different hyperparameters settings. specifically, the key contributions of this work are: as far as we are aware, the proposed work is the first attempt to generate large-scale embedding models evaluates with intrinsic and extrinsic evaluators. ## methodology the principal aim of our research is to investigate the affect of intrinsic and extrinsic evaluations on bengali word embedding models. thus, the proposed scheme comprises of three main parts: corpus creation, word embedding model development, and evaluation. figure 1 illustrates the abstract view of our work. ## results and discussion intrinsic evaluations are performed for a total of ninenty (e.g., @xmath0, @xmath1 @xmath2. among these, the results of best four embedding models presented for extrinsic and intrinsic evaluators. ## conclusion and future work in this work, we have been generated about ninety embedding models for the bengali language. these models have developed using the combinations of three embedding techniques (such as glove, word2vec, and fasttext) and various hyperparameters. all models have evaluated by extrinsic and intrinsic evaluators on our developed corpus. the performance of an embedding model significantly depends on the hyperparameters, corpus and nature of the model. although glove model performed better than word2vec and fasttext, there is no generalized embedding model for intrinsic and extrinsic nlp tasks. the embedding models are highly corpus oriented, and hyperparameters also vary from one task to another. in the future, the existing bengali corpus can be extended for embedding model generation to alleviate the outof-vocabularies problems. the context-dependent feature represents technique (such as bert, elmo and xlnet) will be investigated to find suitable embedding technique for bengali. in addition to that, more analogy tasks can be considered to assess the performance of different embedding models with various intrinsic and extrinsic evaluators.