Evaluating Word Embeddings in Extremely Under-Resourced Languages: A Case Study in B ribri (2022)

Abstract: Word embeddings are critical for numerous NLP tasks but their evaluation in actual under-resourced settings needs further examination. This paper presents a case study in Bribri, a Chibchan language from Costa Rica. Four experiments were adapted from English: Word similarities, WordSim353 correlations, odd-one-out tasks and analogies. Here we discuss their adaptation to an under-resourced Indigenous language and we use them to measure semantic and morphological learning. We trained 96 word2vec models with different hyperparameter combinations. The best models for this under-resourced scenario were Skip-grams with an intermediate size (100 dimensions) and large window sizes (10). These had an average correlation of r=0.28 with WordSim353, a 76% accuracy in semantic odd-one-out and 70% accuracy in structural/morphological odd-one-out. The performance was lower for the analogies: The best models could find the appropriate semantic target amongst the first 25 results approximately 60% of the times, but could only find the morphological/structural target 11% of the times. Future research needs to further explore the patterns of morphological/structural learning, to examine the behavior of deep learning embeddings, and to establish a human baseline. This project seeks to improve Bribri NLP and ultimately help in its maintenance and revitalization.

## introduction word embeddings are dense vectors that describe the semantics of words @xcite . they are calculated by collecting the words around a specific word and using them to create a numerical vector that can determine semantic similarity. for example, the words spinach and kale share neighboring words like garlic and cooked, which would make them similar to each other and different from words with other neighbors, such as hammer. embeddings can vary in size, or dimensional-ity: word embeddings from systems like word2vec @xcite and glove @xcite can range from size 10 to 300, while the embeddings from gpt-3 @xcite can have up to 12288 dimensions. embeddings are an integral part of a number of nlp tasks. because these vectors can be used to calculate cosine distance, one could calculate the distance between words like king and queen, and examine if this is similar to the distance between man and woman. learning these patterns entails that the system is learning the gender distinctions in the words and that it is generalizing patterns across words. this would make an embedding a type of language model which could be enlisted to solve tasks like finding the "odd" word in a triad of words. for example, it could determine that tiger is the odd word in the set {man, woman, tiger} given its greater cosine distance from the other two. this is known as the odd-one-out task. embeddings can also solve analogies by using vector algebra. the embedding vectors can be used for the operation king+woman-man, and the word closest to this result should be queen. these two tasks, odd-one-out and analogies, are some of the main tasks used to evaluate the training of embeddings @xcite . the evaluation of embeddings is relevant not just for these specific semantic tasks, but because contemporary deep learning systems like transformers @xcite , which use embeddings for their learning. embeddings are used as semantically rich input in the training of sequenceto-sequence tasks such as question answering and machine translation. because embeddings are so widely used it is necessary to understand their behavior in low-resource environments. this would allow researchers to better describe the lexical structures of indigenous and other minority languages, and it would also help optimize the training of deep learning algorithms for these languages. making nlp for indigenous languages is a complex task because of the chronic lack of data to train models, so determining the optimal configurations to train embeddings for them can help move these efforts forward. there is relatively little research in this regard @xcite , so this paper seeks to add a case study for how to perform these evaluations in an indigenous language of the americas. ## methodology this paper will examine the evaluation of word embeddings in extremely under-resourced languages by using a realistic example from an indigenous language in central america. the bribri language (glottolog brib1243) is spoken by approximately 7000 people in southern costa rica @xcite . it is a chibchan language and it is vulnerable @xcite , which means that some children no longer speak the language. there are efforts to generate nlp tools for bribri to aid with language maintenance and revitalization efforts, including the construction of digital dictionaries and corpora @xcite @xcite , tools for phonetic analysis @xcite , machine translation @xcite , speech recognition @xcite , natural language understanding @xcite , parsing @xcite and morphological analysis @xcite , as well as tools for the input of the language into digital media @xcite . the motivation for this paper is to contribute information to analyze the training process of deep learning tools for the language. ## discussion the results indicate that there are certain combinations of hyperparameters that could provide better performance for extremely under-resourced languages. embeddings with larger windows and low to mid-sized dimensionality (@xmath0. word2vec also appears to need every word it can get, and its best performance comes when all words in the corpus are included in the training. skip-grams might be a better alternative for this task because they avoid overfitting for very frequent words, thereby absorbing more of the information from the relatively sparse dataset at hand @xcite . cbow has been reported to be better at learning morphological relations, but this was not the case in the bribri dataset, where the cbow showed very low rates of structural learning. indeed, an open question in this paper is the extent to which the model is learning bribri grammar. the results from the basic similarities and the odd-one-out experiments seem to indicate that the model learned grammar at roughly the same rate as it did lexical relationships. however, this is contradicted by the analogies experiment, where there was almost no evidence of structural knowledge in the language model. it might be the case that large windows are interfering with local structural learning (e.g. learning which counter words go with which nouns) in favor of semantic knowledge @xcite , and that the good performance in the odd-one-out structural tests might actually have to do with the semantics of the chosen pairs. more research and a larger test set is necessary to fully understand this effect. importantly, the corpus provided here appears to be enough for the system to learn general semantic patterns. future experiments need to expand on this by providing more pairs, including pairs with more culturally specific words. another future experiment will be to use the data to train other embeddings such as glove, fasttext @xcite , bert-type dynamic embeddings and multilingual embeddings. in the case of bert, we need to study the effect of their high dimensionality on low-resource semantic learning. the preliminary hypothesis would be that this increase in dimensionality would have a negative impact in semantic learning. this needs to be verified because those larger embeddings are necessary for numerous deep learning techniques. one important addition to this experiment would be a human baseline. one of the obstacles in working with indigenous languages is that there are few people who read and write these languages. moreover, there are few speakers of bribri that are familiar with tasks like analogies, which are an unusual type of exercise mostly reserved for academic contexts. therefore, carrying out such an experiment is relatively complex. this is the next step in this project, a necessary one so that the skip-gram's performance can be placed in context. the main ethical concern in the project has to do with data sovereignty. the results will be used to train deep learning systems in collaboration with bribri partners, but there is currently no overarching community organization which controls the access to bribri data. this paper has restricted itself to data that is publicly available or licensed through creative commons, so no private or new data was included. however, an effort needs to be made so that the results of this project benefit the bribri partners in particular and the bribri community in general, by using them to produce useful nlp tools. ## conclusions this paper presents an evaluation of a word embedding in a truly under-resourced environment. it presents a methodology for embedding evaluation that could be adapted to other indigenous languages. it provides evidence that some embedding configurations have better performance when dealing with under-resourced scenarios (i.e. skip-gram trained embeddings, with around 100-dimensions, where the skip-gram attempts to predict words in a window of size 10 and use every word in the corpus to calculate their predictions). the results also confirm the intuition that semantics might be easier to learn than morphology, particularly in morphologically complex languages with little (and sparse) data. finally, the results here will be used to continue nlp work in the bribri language with the objective of training deep learning models and understanding their performance and errors, with the ultimate goal of using these to contribute to efforts of language revitalization.