Experiments in Mamba Sequence Modeling and NLLB -200 Fine-Tuning for Low Resource Multilingual Machine Translation (2024)

Abstract: This paper presents DC_DMV’s submission to the AmericasNLP 2024 Shared Task 1: Machine Translation Systems for Indigenous Languages. Our submission consists of two multilingual approaches to building machine translation systems from Spanish to eleven Indigenous languages: fine-tuning the 600M distilled variant of NLLB-200, and an experiment in training from scratch a neural network using the Mamba State Space Modeling architecture. We achieve the best results on the test set for a total of 4 of the language pairs between two checkpoints by fine-tuning NLLB-200, and outperform the baseline score on the test set for 2 languages.

## introduction the 2024 americasnlp shared task on machine translation (mt) for indigenous languages consists of developing an mt system (or systems) for the purpose of translating spanish to 11 indigenous languages of the americas: aymara (aym), bribri (bzd), asháninka (cni), chatino (ctp), guaraní (gn), wixarika (hch), nahuatl (nah), hñähñu/otomí (oto), quechua (quy), shipibo-konibo (shp), and rarámuri (tar). we take two approaches in parallel, namely finetuning @xcite and training a mamba architecture-based neural network @xcite ) from scratch. 1 2 data ## results we present our results in tables 5 and 6 , alongside results for the two baseline systems. the reported scores are calculated using the chrf++ metric @xcite , as stipulated by the shared task. our nllb+ft(v2) model beats both baseline systems on the development set for aymara and quechua, and both baseline systems on the test set for quechua and rarámuri. additionally, several of our models beat at least one baseline system on the development set for bribri, nahuatl, quechua, and shipibo-konibo. of all submissions this year, our nllb+ft(v2) model achieves the best result for aymara, shipibo-konibo, and rarámuri, and our nllb+ft(v4) model achieves the best result for bribri, as evaluated on the test set. our nllb+ft(v2) and nllb+ft(v4) models achieve average chrf++ scores across all languages of 22.17 and 23.32 respectively, with nllb+ft(v4) representing the second best overall submission. interestingly, while our models did not achieve the best result on the test set for asháninka, hñähñu, and quechua as measured by the official metric, at least one of our nllb+ft models outperformed the best submission in bleu score @xcite . we report these scores in our mamba model shows poor performance at the stage in training at time of submission. however, we believe much of this to be due to undertraining given that our model is trained from scratch. with this in mind, we believe continued training may lead to success of our mamba model, and plan to continue experiments with this architecture. ## conclusion in this paper, we presented our submission to the americasnlp 2024 shared task on machine translation systems for indigenous languages. our submissions included six versions of a fine-tuned 600m parameter distilled variant of nllb-200, and one mamba-based model trained from scratch. we trained all of our models on multilingual data to translate from spanish to 11 target indigenous languages. we achieve the best chrf++ scores on 4 languages with our fine-tuned nllb-200 models, improving upon the baseline systems for two languages and setting a new highest score for rarámuri. additionally, we find our mamba-based