Few-shot S panish- A ymara Machine Translation Using E nglish- A ymara Lexicon (2023)

Abstract: This paper presents the experiments to train a Spanish-Aymara machine translation model for the AmericasNLP 2023 Machine Translation shared task. We included the English-Aymara GlobalVoices corpus and an English-Aymara lexicon to train the model and limit our training resources to train the model in a \textit{few-shot} manner.

## introduction aymara is a language spoken in bolivia, peru and chile. it is one of the larger languages in the americas, and has more than 2 million speakers foot_0 , yet it has received worryingly little attention from nlp researchers. the development of language technologies encourage potential work in the documentation, promotion, preservation and revitalization of the languages @xcite . recent initiatives to promote research on languages of the americas brings nlp researchers closer to the americas languages communities and activists @xcite @xcite @xcite . particularly, machine translation is a useful tool that encourages more research in the languages as it bridges the communication gaps in nlp researchers' understanding of the models' capabilities and limitations. the americasnlp 2021 workshop hosted the open machine translation (omt) shared task focusing on indigenous and endangered americas languages @xcite . the organizers provided a seed collection of publicly available corpora and highlighted the various nuances and variability of the translations due to the geographical and linguistic diversity between the language varieties. the spanish data for development and test sets created in the americasnlp 2021 shared task are translated into the aymara la paz jilata variant, which is the same variant used in the global voices corpus @xcite . while aymara is mutually intelligible across different dialects, they might differ in specific terminologies and minor grammatical preferences. this paper presents our submission to the amer-icasnlp 2023 machine translation shared task @xcite . we submitted our system that focuses only on translating from spanish into aymara. we fine-tuned a multilingual t5 model @xcite by adding an aymara-english lexicon foot_1 to the existing spanish-aymara and english-aymara global voices corpus and the spanish-aymara shared task training data @xcite . other than presenting the results of our amer-icasnlp shared task submission, parts of this paper will also serve as a demonstration of how the model was modified from typical model training using huggingface suite of libraries @xcite @xcite , this is especially useful for low-resource sequence-to-sequence tasks. ## pre-trained tokenizer and new languages while the current state of vogue in using massively multilingual pre-trained models on low-resource languages allows researchers to extend the models' sub-word tokenizers, the models implicitly re-use the tokens from how it was previously pre-trained and simply ignore the new tokens by labelling them as @xcite . in cases where the character set of the low-resource languages' orthography matches the languages that the models were pre-trained on, it is possible that the models repurpose the subwords to learn new parameter behaviors given sufficient computes and hyperparameter tuning experiments. to preserve the learned model parameters, a researcher using the multilingual model can extend its tokenizer's sub-word vocabulary by relearning the sub-word tokenizer from scratch, then apply it to dataset with the new language and finally extending the new sub-words to the pre-trained vocabulary. to assign new parameters in the model for these new sub-words tokens, the embedding layer of the model needs to be extended. the code snippet above demonstrates the function to extend the new language's vocabulary to existing pre-trained mt5 model. the following snippet below presents the differences of the input token indices depending on how the tokenizer was extended for a new language. from transformers import autotokenizer @xmath0.from_pretrained('google/mt5-base') @xmath1.from_pretrained('alvations/mt5-aym-lex') @xmath2.decode([s]) for s in tokenized_old_ids] @xmath3.decode([s]) for s in tokenized_new_ids] ## experimental setup all models fine-tuned in this paper uses the mt5 architecture using a100 gpus with 40gb ram. we use the all default hyperparameters of the hug-gingface's seq2seqtrainingarguments except: we fine-tuned a zero-shot lexicon-enriched system mt5 model with aymara-english lexicon, the spanish-aymara and english-aymara global voices corpus and spanish-aymara xnli training data split for the training data. and we use the spanish-aymara xnli development data split provided by the shared task organizers to select the best performing model. our official submission to the shared task is selected from the best-performing system that scored the lowest perplexity loss and highest bleu score. other than the best performing zeroshot lexicon-enriched system (mt5-lex), we experimented and a baseline model that only finetuned spanish-aymara global voice and xnli dataset (mt5-base) and a second baseline that adds on the english-aymara global voices data to spanish-aymara global voices and xnli dataset (mt5-zero). ## results our official submission to the shared-task scored a measly 0.12 bleu @xcite and 9.22 chrf score @xcite on the americasnlp 2023 shared task test set. the best performing team in the shared task achieved 4.45 bleu and 36.24 chrf. the target aymara text from the test set was not released publicly, hence we present the results of our model variants on the development set. ## conclusion in this paper we present our participation in the americasnlp 2023 spanish-aymara machine translation shared task. we experimented with adding an english-aymara lexicon and training we share the follow resources created in our participation for future researchers to improve english/spanish-aymara translations.