On Evaluation of B angla Word Analogies (2023)

Abstract: This paper presents a benchmark dataset of Bangla word analogies for evaluating the quality of existing Bangla word embeddings. Despite being the 7th largest spoken language in the world, Bangla is still a low-resource language and popular NLP models often struggle to perform well on Bangla data sets. Therefore, developing a robust evaluation set is crucial for benchmarking and guiding future research on improving Bangla word embeddings, which is currently missing. To address this issue, we introduce a new evaluation set of 16,678 unique word analogies in Bangla as well as a translated and curated version of the original Mikolov dataset (10,594 samples) in Bangla. Our experiments with different state-of-the-art embedding models reveal that current Bangla word embeddings struggle to achieve high accuracy on both data sets, demonstrating a significant gap in multilingual NLP research.

## introduction the bangla language, having over 300 million native speakers and ranking as the seventh most spoken language in the world, is still regarded as a language with limited resources @xcite . despite the breakthrough in the field of natural language processing (nlp) recently, it has been reported that popular nlp models often fail to perform well for low-resource languages like bangla while showing human-like performance in highresource languages like english. as such, there is a growing need for high-quality bangla word embeddings, and several efforts have been made in this direction by training embeddings on large bangla corpora @xcite . for evaluation, there have been a few efforts to create bangla benchmark data sets for downstream nlp tasks like sentiment analysis, machine translation, and summarization @xcite . on the contrary, there is a clear absence of word analogy data sets for evaluating the syntactic and semantic properties of bangla word embeddings, although such word analogy evaluation data sets are available for other languages like turkish, german, spanish, arabic, etc. @xcite . as word embeddings have direct impacts on the performance of downstream nlp tasks, creating a high-quality evaluation test set for bangla word embeddings will enable researchers to benchmark the performance of existing bangla embedding models and guide further research. with this motivation, we present a mikolov-style @xcite high-quality word-analogy evaluation set exclusively for bangla, with a sample size of 16,678 foot_0 . to the best of our knowledge, we are the first ones to do so. in addition, we translated and curated mikolov's original dataset for bangla, resulting in 10,594 more analogies. for both these data sets, we also provide an analysis of the performance of several stateof-the-art bangla/multilingual embedding models: word2vec @xcite , glove @xcite , fasttext @xcite , laser @xcite , labse @xcite , bnbert (bhattacharjee et al., 2022a), bnbart @xcite . our experiments with different state-of-the-art embedding models reveal that current bangla word embeddings struggle to achieve high accuracy on both data sets, signifying that bangla has its own unique characteristics, and further research is warranted for low-resource languages like bangla. therefore, we suggest that future research on bangla word embeddings should report accuracy using this new benchmark data set in order to track consistent research progress in this direction. ## dataset to evaluate the quality of word vectors, the authors (who are also native bengali speakers) independently first proposed different types of relationships for the bangla language. subsequently, they created a list of related word pairs for each relationship and then formed analogies by grouping two pairs. the pairs were then independently reviewed by other annotators and removed if there was any disapproval from the annotator. thus, a comprehensive test set was developed that contains three types of semantic analogies and nine types of syntactic analogies, as shown in additionally, we translated mikolov's original dataset foot_1 and manually removed english words that do not have bangla translations. we also removed word pairs that translate to duplicated terms in bangla, such as present participles and plural verb forms. this cleaning step resulted in a dataset of 10,594 samples from the original 19,544 samples. in summary, while our new dataset focused on the linguistic specifics of the bangla language, the translated mikolov's dataset is more focused on common words in both bangla and english. the translation of mikolov's dataset provides a useful resource for cross-lingual research and analysis. ## experimental setup we evaluated the quality of pretrained word vectors trained using both traditional and transformerbased models on both data sets. for classical models, we used word2vec, glove, and fasttext, while for transformer-based models, we employed laser, labse, bnbert, and bnbart, to evaluate the quality of word embeddings. for glove, we used the bengali glove model that was trained on bangla wikipedia 3 and bangla news articles. specifically, we utilized the model trained with 39m tokens. for word2vec, we used the bengali word2vec model, also trained with bengali wikipedia data @xcite . we also used fasttext, which provides pretrained bangla word embeddings @xcite . for transformer-based models, we used laser @xcite , labse @xcite , bnbert (bhattacharjee et al., 2022a), bnbart @xcite , which provide sentence embeddings. to obtain the word embedding for a particular word, we passed the word to the model and collected the contextualized token embeddings for all the tokens of that word, which were then averaged to obtain the word embedding. this way, we created an exhaustive embedding dictionary for 178,152 bengali words and used them directly to perform the word analogy task. for experiments, we used nvidia quadro rtx 5000 gpus. ## results our first set of results is focused on our handcrafted bangla analogy set (presented in next, we compare our handcrafted dataset against the translated mikolov dataset in figure 2 and 3. again, labse and bnbart were the overall winners, and both methods performed better on the handcrafted dataset than the translated one. ## conclusion in this paper, we have introduced a high-quality dataset for evaluating bangla word embeddings through word analogy tasks, similar to mikolov's word analogy data set for english. additionally, we translated mikolov's original dataset into bangla, contributing a second data set. our exper-iments with different word embeddings reveal that current word embeddings for bangla still struggle to achieve high accuracy on both data sets. to improve performance, future research should focus on training embeddings with larger data sets while taking into account the unique morphological characteristics of bangla.