Fine-tuning Sentence- R o BERT a to Construct Word Embeddings for Low-resource Languages from Bilingual Dictionaries (2023)

Abstract: Conventional approaches to learning word embeddings (Mikolov et al., 2013; Pennington et al., 2014) are limited to relatively few languages with sufficiently large training corpora. To address this limitation, we propose an alternative approach to deriving word embeddings for Wolastoqey and Mi’kmaq that leverages definitions from a bilingual dictionary. More specifically, following Bear and Cook (2022), we experiment with encoding English definitions of Wolastoqey and Mi’kmaq words into vector representations using English sequence representation models. For this, we consider using and finetuning sentence-RoBERTa models (Reimers and Gurevych, 2019). We evaluate our word embeddings using a similar methodology to that of Bear and Cook using evaluations based on word classification, clustering and reverse dictionary search. We additionally construct word embeddings for higher-resource languages English, German and Spanishusing our methods and evaluate our embeddings on existing word-similarity datasets. Our findings indicate that our word embedding methods can be used to produce meaningful vector representations for low-resource languages such as Wolastoqey and Mi’kmaq and for higher-resource languages.

## introduction word embeddings @xcite are real-numbered vector representations of the meanings of words and are a fundamental component of many natural language processing (nlp) systems. although word embeddings can often be learnt while training nlp systems end-to-end, pretrained word embeddings have been shown to bolster the performance of nlp systems in tasks such as machine translation @xcite and information retrieval @xcite . despite their utility, quality word embeddings can be difficult to obtain as they generally require large corpora of running text to train. this represents a significant limitation of conventional word embedding methods as, due to these data requirements, quality word embeddings can only be learnt for relatively few languages. today, a majority of languages spoken around the world are low-resource @xcite , and thus lack the text resources required to train high quality word embeddings. as this is the case, an alternative embedding approach is desirable to make better use of what data exists for low-resource languages. in the case of wolastoqey (also referred to as passamaquoddy-maliseet) and mi'kmaq, there simply isn't enough data available in these languages to train quality word embeddings using conventional methods. wolastoqey and mi'kmaq are both low-resource eastern algonquin languages. there are currently approximately 300 remaining first language speakers of wolastoqey and 7k speakers of mi'kmaq @xcite in canada. due to the low-resource state of these languages, developing language technologies for wolastoqey and mi'kmaq is challenging because there are no large corpora or annotated datasets available in these languages to train nlp systems. despite not having large corpora or datasets available, both a bilingual wolastoqey-english dictionary, known as the passamaquoddy-maliseet dictionary @xcite , and a bilingual mi'kmaq-english dictionary, known as the mi'gmaq/mi'kmaq online dictionary @xcite , are available. these dictionaries contain english definitions for wolastoqey and mi'kmaq headwords and consist of a total of 18.6k and 6.5k entries, respectively. in our work, we experiment with using these dictionaries to construct word embeddings for wolastoqey and mi'kmaq. previous work has demonstrated that bilingual lexicons and monolingual corpora can be leveraged to train cross-lingual word embeddings for low-resource languages. for example, @xcite showed that, by combining a large english corpus with a small yongning na corpus, and by replacing words with their translations using a small bilingual lexicon, a pseudo-bilingual corpus can be created which can be used to train crosslingual word embeddings. we do not consider this approach in our work because wolastoqey and mi'kmaq are polysynthetic languages, and as such, many tokens that occur in a corpus would not be expected to be found as dictionary headwords, which limits the applicability of this approach. instead, we look towards approaches based on sequence representation. prior work has demonstrated that, by leveraging bilingual dictionaries, useful vector representations can be constructed for nêhiyawêwin (plains cree) words. by averaging word embeddings corresponding to words that appear in english definitions of nêhiyawêwin words, embeddings can be obtained which can be used to effectively cluster nêhiyawêwin words @xcite and map them to preconstructed ontologies @xcite . @xcite extended the methodology of @xcite and @xcite to construct word embeddings for wolastoqey. they used the average of word2vec embeddings to represent words from their dictionary definitions, as well as roberta, and sentence-roberta models to encode definitions into vector representations. these embeddings were then evaluated based on word classification tasks focused on predicting part-of speech, animacy, and transitivity; semantic clustering; and reverse dictionary search. in each evaluation, it was found that approaches using these embeddings outperformed task-specific baselines, indicating that sentencetransformer models can outperform approaches based on word embeddings for this purpose. as this approach has been shown to perform relatively well, in this paper, we build upon the work of @xcite by fine-tuning sequence representation models for this task. more specifically, we propose fine-tuning sentence-roberta models on monolingual dictionary definitions to determine if doing so could improve the overall quality of the representations. using these finetuned models, we construct word embeddings for wolastoqey and mi'kmaq from english definitions in the passamaquoddy-maliseet dictionary and mi'gmaq/mi'kmaq online dictionary. following @xcite , we evaluate our wolastoqey and mi'kmaq word embeddings on word classification tasks focused on predicting part-of speech, animacy, and transitivity as well as semantic clustering and reverse dictionary search. we compare our word embeddings against taskspecific baselines and embeddings produced using the techniques of bear and cook. to assess if this technique is viable for other higher-resource languages, we also construct word embeddings for english, spanish and german, and evaluate the performance of our models on word similiarity datasets, comparing against previously reported results. ## methodology to obtain embeddings for wolastoqey and mi'kmaq words, we experiment with encoding english definitions of wolastoqey words in the passamaquoddy-maliseet dictionary, and mi'kmaq words in the mi'gmaq/mi'kmaq online dictionary, into vector representations. to construct vector representations from english definitions, we consider fine-tuning and using sentence-transformer models, masked language models specifically trained for sequence representation. more specifically, we consider fine-tuning sentence-roberta models using three training regimens from @xcite . we compare our embeddings constructed with this approach to those created using the methodology of @xcite . in our work, we fine-tune our sentence-roberta models on the dataset of @xcite . this dataset consists of word-definition pairs collected from several english dictionaries and wordnet @xcite . in our experiments, we use the training and development splits from @xcite . this gives us a training set consisting of a total of 667.5k word-definition pairs corresponding to 45k unique types and a development set consisting of 75.8k definitions corresponding to 5k unique types. however, due to the data requirements of our fine-tuning regimens, we filter out any definitions corresponding to words with only one unique definition and filter out duplicate definitions from both our training and development sets. this reduces our effective training corpus size to 664.7k definitions corresponding to 42.5k unique types, and our development set to 75.6k definitions corresponding to 4.7k unique types. we use our development set to ensure overfitting does not occur and to monitor training performance. to fine-tune our sentence-roberta models, we continue training from the nli-roberta-base-v2 model available in the sentence transformer 2.1.0 library. 1 this model represents a checkpoint that has been pretrained on a large natural language inference dataset, constructed by combining the stanford nli corpus @xcite and the multi-genre nli corpus @xcite . we consider fine-tuning three models using the softmax, cosine, and triplet loss training objectives outlined in @xcite . each of these training objectives requires the model to be trained in a siamese configuration in which two or more examples are passed through the network independently before being compared to compute training loss at a given time-step. the softmax training objective is based on classification. in our work, the classification task we fine-tune our model on is determining if two definitions correspond to the same word. to construct training pairs for this fine-tuning regimen, we pair each definition in our training set with another definition to form either a positive or negative training example. we assign half of our definitions another definition corresponding to the same word, forming a positive pair, and we assign the other half definitions that do not correspond to the same word, forming negative pairs. this gives us 664.7k training examples, equal to the number of definitions in our training corpus. the cosine training objective is based on regression. more specifically, in this fine-tuning regimen, we attempt to match the cosine similarity between two output vectors to some ground truth label. to obtain examples, we form training pairs similarly to how we did for the softmax fine-tuning regimen. however, instead of assigning a binary label to pairs, we assign a ground truth cosine similarity. for positive pairs, this is simply equal to 1.0. however, for negative pairs, to obtain ground-truth cosine similarities, we use the cosine similarities computed from vectors in a word2vec model. for this purpose, we use a word embedding model that has been trained on a google news corpus consisting of roughly 100 billion words. 2 we obtain these embeddings using gensim 3.8.3 @xcite . for each negative sample, the groundtruth cosine similarity used for training is set to the cosine similarity calculated using the embeddings for the words each definition corresponds to. in this training configuration, loss is calculated as the mean squared error between the cosine similarity of the two input vectors and the ground truth reference. finally, triplet loss considers three inputs, in our case definitions, at a given timestep. more specifically, this training scheme requires an anchor, as well as two additional inputs that act as positive and negative instances. when fine-tuning with this training objective, we attempt to learn weights such that the representations produced for each anchor are closer to their corresponding positive than negative instance. as this is the case, to form training examples, we treat each definition in our training set as an anchor and assign each an accompanying positive instance -a definition corresponding to the same word -and a negative instance -a definition corresponding to a different word. like before, this gives us a total of 664.7k training examples to fine-tune our model with. for each training technique considered, we finetune our models using the default training parameters of the sentence-transformers library. we finetune our models for a single epoch, as, training for three epochs appeared to degrade performance on our word classification tasks in early testing. after fine-tuning, we are left with three models, each fine-tuned using a different training regimen. to construct word embeddings using these models, we first preprocess our input definitions using the same preprocessing steps as @xcite . namely, we consider removing bracketed content from our input definitions, as, in the dictionaries we use in our work, this typically consists of topical information that does not contribute to the core meaning of definitions. we then pass our preprocessed input definitions to our sentence-roberta models to obtain a vector representation based on the mean output vectors of our sentence-roberta models. ## word classification following @xcite , we evaluate our word embeddings on word classification tasks to determine if they are capable of capturing information about the syntactic properties of words. we consider three word classification tasks focused on predicting, 1.) part-of-speech, 2.) noun animacy and 3.) verb type. for each task, we train logistic regression classifiers to predict the syntactic labels of words from their embeddings. to construct datasets for these evaluations, we use gold-standard labels from the passamaquoddy-maliseet dictionary and mi'gmaq/mi'kmaq online dictionary. for our part-of-speech classification tasks, we consider a total of 18k entries from the passamaquoddy-maliseet dictionary, consisting of 53 pronouns, 231 preverbs, 570 particles, 13.7k verbs and 3.3k nouns, for wolastoqey and 6.4k entries from the mi'gmaq/mi'kmaq online dictionary, consisting of 16 pronouns, 119 particles, 4.6k verbs and 1.6k nouns, for mi'kmaq. for our noun animacy classification tasks, we remove any entries corresponding to words that can occur as both animate and inanimate. in total, we use 1.7k animate, and 1.3k inanimate nouns for wolastoqey and 756 animate, and 806 inanimate, nouns for mi'kmaq. in both wolastoqey and mi'kmaq, verbs are categorized into four distinct groups based on their combination of animacy and transitivity. more specifically, wolastoqey and mi'kmaq verbs can be, animate intransitive, inanimate intransitive, transitive animate, or transitive inanimate. we remove any entries that correspond to more than one of these categories. this gives a total of 5.3k animate intransitive, 2.1k inanimate intransitive, 3k transitive animate, and 2.7k transitive inanimate wolastoqey verbs, and 2k animate intransitive, 753 inanimate intransitive, 1k transitive animate and 847 transitive inanimate mi'kmaq verbs, for our verb type classification tasks. to conduct this evaluation, we first construct embeddings for each wolastoqey and mi'kmaq word using our proposed methodology. we then train logistic regression classifiers for each task and method. for this evaluation, we implement our logistic regression classifiers using scikit-learn 0.24.2. we use the default training parameters of this library, except max-iterations, which we set to 6000, so that all models finish converging. we train and evaluate in a 10-fold cross validation setup. we use macro-averaged accuracy, precision, recall, and f1-score as our evaluation metrics and compare our models to a most-frequent class baseline as well as the pretrained sentence-roberta based approach proposed by @xcite as it has been shown to achieve strong performance in this task. ## clustering here we explore using our embedding models to semantically cluster wolastoqey and mi'kmaq words. for this experiment, we largely follow the evaluation procedures of @xcite . for wolastoqey, we reproduce the experiments of bear and cook for the purpose of comparison. ## reverse dictionary here we use our wolastoqey and mi'kmaq word representations to create reverse dictionary search systems. such systems could potentially help wolastoqey and mi'kmaq learners to more-easily access language resources. ## word similarity although our primary interest is methods for learning wolastoqey and mi'kmaq word representations, here we consider whether the proposed approach to encoding dictionary definitions can also be applied to represent words in higher-resource languages. word similarity datasets are available for many languages and are commonly used to evaluate how well word embedding models are able to capture the similarity or relatedness between words. here we consider constructing word embeddings for english, german and spanish using our proposed methodologies and evaluating on word similarity datasets. as these datasets are frequently used in other works, where available, we compare against previously reported results for word2vec baselines. ## conclusions in this paper, we considered approaches to forming word embeddings for wolastoqey and mi'kmaq based on their english definitions in bilingual dictionaries. specifically we considered approaches to fine-tuning sentence-roberta for this. our findings indicate that our proposed approaches can be used to construct embeddings for wolastoqey and mi'kmaq words that capture syntactic and semantic information, and that fine-tuning often gives improvements over pre-trained sentence-roberta, although this improvement is not consistent across languages, tasks, and approaches to fine-tuning. our results from reverse dictionary evaluations indicate that these embeddings cannot yet be used to build a practical reverse dictionary search system. we further showed that these approaches can be applied to form embeddings for higher-resource languages. although here these embeddings achieved significant correlations on word similarity and relatedness evaluations, they did not improve over conventional word2vec embeddings. in future work, we intend to explore ways to improve the embeddings. although we observed that fine-tuning sentence-roberta did not give consistent improvements across tasks, we hypothesize that an alternative approach could give improvements. definitions for verbs in the passamaquoddy-maliseet dictionary in particular tend to be longer, while definitions for nouns are typically quite short and often composed of only a single word. this disparity in definition complexity could hinder the effectiveness of our proposed word embedding techniques. we therefore intend to explore the use of ulr-bert @xcite , which is capable of representing words, phrases and sentences proficiently, for forming improved embeddings for wolastoqey and mi'kmaq words from their english definitions in bilingual dictionaries. in addition to using ulr-bert, we also intend to fine-tune sentence-transformer models that make use of different network architectures and pretraining regimens. in our work, we use a single roberta checkpoint, pretrained on natural language inference, as a uniform starting point for finetuning. however, since the release of the original work on sentence transformers, other models have been made available through the sentence-bert library, for example models based on mpnet @xcite , which have been shown to outperform sentence-roberta on sentence embedding benchmarks. as this is the case, the use of these models in-place of sentence-roberta could potentially improve the quality of word embeddings produced using our methodology. in our work, we demonstrated that we can construct meaningful word embeddings for wolastoqey and mi'kmaq dictionary headwords. in future work we will consider evaluating the impact of these embeddings on down-stream applications.