TEMA : Token Embeddings Mapping for Enriching Low-Resource Language Models (2024)

Abstract: The objective of the research we present is to remedy the problem of the low quality of language models for low-resource languages. We introduce an algorithm, the Token Embedding Mapping Algorithm (TEMA), that maps the token embeddings of a richly pre-trained model L1 to a poorly trained model L2, thus creating a richer L2’ model. Our experiments show that the L2’ model reduces perplexity with respect to the original monolingual model L2, and that for downstream tasks, including SuperGLUE, the results are state-of-the-art or better for the most semantic tasks. The models obtained with TEMA are also competitive or better than multilingual or extended models proposed as solutions for mitigating the low-resource language problems.

## introduction at the forefront of artificial intelligence, large language models have achieved extraordinary levels of performance in most natural language processing tasks. these advancements, driven by deep learning techniques, highlight the ability of these models to process complex linguistic contexts. however, their effectiveness seems to be intrinsically linked to the enormous amount of data required for pre-training, thus raising concern about how low-resource languages could benefit from them. low-resource languages, that is, languages that do not have a massive amount of text, risk being almost excluded from the possibility of having good nlp applications. for coping with the problems of low-resource languages in large language models, there have been different proposals, mainly: building multilingual models and applying transfer learning techniques and the so-called data augmentation methods. augmentation methods for language modelling are those that automatize the production of new texts by different means: template-based scripts @xcite , machine translation @xcite , and delexicalized seq2seq models @xcite , for instance. however, it is costly to guarantee the semantic correctness of the produced texts. multilingual models are models trained with texts in different languages @xcite @xcite . in addition to accepting different languages, they can then be fine-tuned on a downstream task using labelled data in only one language and it is expected to generalize in order to handle samples of the other languages it has been pre-trained with. however, low-resource languages are still under-represented in the vocabulary impairing proper modelling. in this context, we propose to remedy the low quality of language models due to poor pre-training data by translating (i.e. mapping) token embeddings of a richly trained model l1, to the position of the translation equivalent token embedding in a poorly pre-trained model l2, thus creating a richer l2' model. our token embedding mapping algorithm (tema) works as if the learned lexical parameters of l1 were transferred to l2, so that even tokens of l2 for which only one example is available get a rich representation. we assessed the efficacy and the impact of tema with different experiments with quechua, which is a low-resource language and, in addition, it is a polysynthetic language with more than 100 inflectional suffixes. languages with inflectional or synthetic morphology have a larger number of different tokens than other languages such as chinese or english and are harder to model @xcite . in addition, and in order to be able to evaluate the impact of using tema with standard benchmarks, we provide the results of working with english, german and french for which evaluation datasets are available and for which we have reproduced poor pre-training conditions. finally, we compare the results of our approach with other cross-lingual transfer methods and with standard multilingual models, which are standard proposed solutions for having models for low-resource languages. ## related work multilingual models and data augmentation methods have been proposed as solutions for transformer-based language models to satisfactorily process low-resource languages. for instance, the multilingual mbert @xcite was trained with data belonging to 104 different languages and no special supervision like parallel corpora or bilingual dictionaries. the resulting multilingual representations were said to be shared tokens, mostly subwords, for many different languages. these results encouraged the research on transfer learning techniques @xcite under the assumption that pre-training a very large model with many languages is to get representations with general-purpose "knowledge" that would improve the performance on downstream tasks for many different languages. however, @xcite already show that performance degraded across all languages and that low-resource languages are under-represented in the vocabulary, which mostly contains the tokens of the largest languages. the common tokens that multilingual models use as crosslingual token embeddings to transfer information from one language to another are at the end subwords made of small groups of characters for which the model cannot learn good representations. @xcite demonstrated that crosslingual learning transfer to get a shared vocabulary could be done without joint training. their method transferred a monolingual model l1 to a new language l2 by only substituting token embeddings of l1 with the new token embeddings learnt by further training the monolingual model with data of the new language l2, while freezing the transformer. to evaluate the impact of transferring the model from one language to another, among others, these authors used the wic dataset (word in context, (pilehvar and camacho-collados, 2019)) delivering competitive results. dobler and de melo (2023) and @xcite propose to use token embeddings from a model trained with abundant resources as initialization embeddings for a low-resource model. our method keeps the idea of working only at the lexical level, but instead of fur-ther training with l2 data, transferring the lexical knowledge, i.e. projecting the token embeddings, from a richly trained l1 to a poorly pre-trained monolingual model of l2. in the method we are presenting, the transfer is done by a mapping function inspired by other previously proposed methods. @xcite and @xcite worked under the hypothesis that words from different languages get similar static embedding representations. therefore, to align the monolingual spaces should be possible with minimal supervision, for instance, common tokens, to create bilingual dictionaries automatically with a mapping function @xcite @xcite . the idea of automatically producing texts, or data augmentation, was first used to improve system robustness. @xcite or @xcite proposed to evaluate the understanding capacities of systems by creating synthetic data, following the idea from computational vision, where, in order to test a learner's oversensitivity, a bit of noise is added to the test sample to evaluate whether the system changes the decision taken for the original test input. on the same basis, there were different proposals to automatically generate semantically equivalent sentences using, for instance, paraphrase generation techniques. @xcite method was based on neural machine translation back-translation methods. once paraphrases were found, a number of heuristic pattern-matching rules were induced, and later filtered to produce a set of rules to automatically produce new sentences from any given reference dataset. other methods for creating new sentences are based on: word-level modifications of original sentences based on synonym replacement @xcite , using lstm language models @xcite , multilingual language modelling @xcite and auto-regressive pre-trained language models @xcite . however, these methods suffer from generating semantically unrealistic sentences that have to be manually revised in most cases. proposals of methods to preserve semantics are @xcite al. (2018), zhou et al. (2019), or zevallos et al. (2022a) that used a delexicalized seq2seq model for generating some similar sentences each with different words of the same semantic field as found in resources like wordnet @xcite . eventually, this method allows for increasing the vocabulary of the model, a side effect that has proved to be beneficial in all the scenarios. enhancing vocabulary is one of the benefits highlighted in eb-bert @xcite . eb-bert consists of a bilingual model, b-bert, which has an extended vocabulary and is fine-tuned with data from the target low-resource language. the vocabulary extension involves adding the 10,000 most frequent tokens from the low-resource language to the original b-bert vocabulary. the fine-tuning process uses only the masked language modeling objective with the l2 data. this method was proposed to avoid training a multilingual model from scratch with a new language, addressing the issue that in a multilingual model, there is often limited space in the vocabulary for tokens from languages with smaller training datasets. @xcite proposed to fine-tune a cross-lingual model with mined translation word pairs extracted from two monolingual language models, and confirmed that extending the vocabulary was one of the keys for improving the coverage of low-resource languages. also close to our approach is the line of research inspired by the word replacement strategies by @xcite , which proposed a virtual data augmentation method for fine-tuning pre-trained monolingual language models. in embedding augmentation, the original token embeddings are replaced by ones which are derived by a probabilistic mixture of the embeddings of the vocabulary terms that are predicted by the masked language model conditioned on the input. our approach also substitutes specific tokens created in a monolingual language model but with the embeddings created in another monolingual model that however was trained with a larger amount of data. ## token embedding mapping we propose to remedy the low quality of a language model poorly pre-trained l2 by mapping its token embeddings to the position of its translation equivalents in a richly pre-trained model l1. the token embedding mapping algorithm (tema) we propose works as if the learned lexical parameters of l1 were transferred to l2, so that even tokens of l2 for which only one example is available get a rich representation. figure 1 illustrates how mapping token embeddings is done by translating token embeddings of a poorly trained model l2 (u n ) to the position of the translation equivalent token embedding in a richly pre-trained model l1 (w m ), thus creating a richer n , that is the projection of u n equivalent token in l1, w m , taking as references the token embeddings v x and v y , that correspond to the token '1' in each models l2' model. to do this, we employed the affine transformation method @xcite using as references the tokens of the number '1' (v x and v y ) which exists in the two models, a method proposed in research on inducing bilingual word embeddings @xcite . in equation 1, u ′ n represents the enriched version of u n , and proj sp (v xw m ) is the projection of the vector v xw m in the l2 vector space s p .the projection proj sp (v xw m ) onto s p using v y as a reference is obtained with equation 2: in this section, we describe tema (token embedding mapping algorithm). the algorithm takes the reference token embeddings v x and v y as input from l1 and l2, respectively, as well as two lists of tuples d x and d y . these tuples contain the index of two translation-equivalent tokens (i m and j n , respectively), where each index is associated with a word in the corresponding language, along with their respective embeddings (w m and u n ). the algorithm utilizes the function t that identifies token pairs between l1 and l2 translation equivalents as found in a bilingual dictionary. tema begins by using t , and in case it returns u @xmath0, we add the target token (tgt_token) using the index j n from d y to the list v , which contains all tokens missing in the model l2. we repeat this process until we traverse all of d x . if v ̸ = ∅ then for each token in v , we add a sentence (which can be an example phrase from the dictionary or other sources) to the file named tgt_sent. after adding all the phrases, we fine-tune the model l2 with the tgt_sent file, resulting in the model l2'. finally, we iterate again through each i m and w m in d x . in each iteration, we use t , and if t returns u n ̸ = ∅ , we calculate the projection using equations 1 and 2. once we obtain u ′ n , we update d y with it. when the iteration finishes, we update the model l2' (or l2 if @xmath1. ## experimental setup our main target was the use of tema for modelling quechua, which is a low-resource language. additionally, as already mentioned, in order to accurately evaluate its impact with standard benchmarks and compare with other methods, we designed different experiments involving another three l2-languages, for which we reproduced lowresource conditions. for the methods that involve two languages, we paired them as follows: l1-spanish and l2-english (es-en), l1-english and l2-german (en-de), l1-english and l2-french (enfr), and l1-spanish for the l2-quechua (es-qu). in the subsections 4.1 and 4.2, we describe the resources used: bilingual dictionaries and training corpora, in subsection 4.4 we describe the different language models that have been trained for the experiments and in subsection 4.3 the used tokenizers: bpe and deepspine. as for the evaluation, we provide results of intrinsic evaluation in terms of pseudo-perplexity @xcite , and of different downstream tasks as extrinsic evaluation. ## discussion results in all the experiments show that the affine transformations on the vector space of the model are not causing trouble. on the contrary, the experiments we have carried out validate that tema indeed mitigates the issues due to the low quality of the representations of low-resource languages. first, low-resource languages have problems finding enough training data in terms of the total amount of tokens, that is, the vocabulary, but also in the number of samples of each token. token frequency has proved to be critical in transformerbased language modelling. for instance, @xcite demonstrated that when a token appears fewer than 15 times in the training data, a bert model will disregard it, while a token that appears 100 times or more will be predicted more accurately. @xcite found evidence too that bert models cannot correctly predict tokens that occur less than 10 times in the corpus. therefore, for low-resource languages, many of the tokens occurring in the corpus do not achieve the frequency required for the model to represent them correctly. in the results shown in perplexity reduction is notable for all languages. note that, as expected, a tokenization that delivers subwords that are more similar to word stems improves tema results for all the languages. these modelling improvements can be further appreciated with the fill-mask tests for the es-qu roberta + tema model compared to the quechua roberta 10m model. second, the semantic quality of the token representations is improved using tema. figure 2 shows a sample of the vector space of quechua models with tokens related to different semantic groups: locations, animals, food and agriculturerelated words. the improvement after tema is applied is that the distance between related words is reduced, showing a better-structured space that should better support semantic tasks. indeed, the results of the more semantic super-glue downstream tasks validated this hypothesis. results in the improvement in the semantic quality of the embeddings is also evident from the examples from the fill-mask test shown in ## conclusions we have introduced tema: token embedding mapping algorithm, a method for mapping the token embeddings of a richly pre-trained language model to a poorly pre-trained model. in this paper, we have provided experimental evidence that the language models resulting from using tema for four different languages showed reduced perplexity and very competitive performance in xtreme and superglue tasks compared to other systems based on transfer techniques and lexicon expansion methods. another contribution of our experiments is the confirmation that the amount of lexical items of a language in the vocabulary is directly affecting the results. however, our experiments demonstrate that tema crucially transfers adequately the lexical parameters learned with larger data, getting better results by overcoming the problem of the amount of data available for each token in a small corpus, or after just enlarging the lexicon. therefore, tema contributes to the research on nlp methods for low-resource languages by proposing a method that only requires a bilingual dictionary and a rather small monolingual corpus of 10m tokens to deliver state-of-the art results or even better for basic nlp tasks such as pos tagging and ner. resources and code are available at https: //github.com/iulaterm-trl-upf/tema