CUFE at NADI 2024 shared task: Fine-Tuning Llama-3 To Translate From A rabic Dialects To M odern S tandard A rabic (2024)

Abstract: LLMs such as GPT-4 and LLaMA excel in multiple natural language processing tasks, however, LLMs face challenges in delivering satisfactory performance on low-resource languages due to limited availability of training data. In this paper, LLaMA-3 with 8 Billion parameters is finetuned to translate among Egyptian, Emirati, Jordanian, Palestinian Arabic dialects, and Modern Standard Arabic (MSA). In the NADI 2024 Task on DA-MSA Machine Translation, the proposed method achieved a BLEU score of 21.44 when it was fine-tuned on thedevelopment dataset of the NADI 2024 Task on DA-MSA and a BLEU score of 16.09 when trained when it was fine-tuned using the OSACT dataset.

## introduction the transformer architecture has revolutionized natural language processing, leading to the development of large self-supervised neural networks with tens or hundreds of billions of parameters trained on trillions of tokens. recent models based on the transformer architecture include gpt series @xcite , llama @xcite , and others. as of late 2024, the release of llama-3 sparked the growth of a thriving opensource community, resulting in the development of near-state-of-the-art models like mistral @xcite and @xcite . these llms can adapt to new tasks and instructions with minimal finetuning. to save on finetuning costs, the nlp community uses parameter-efficient fine-tuning (peft) techniques instead of full-parameter fine-tuning. these peft techniques are shown to significantly reduce a model's computational footprint @xcite . low-rank adaptation (lora) @xcite , an innovation among these peft techniques, effectively balances performance and efficiency through low-rank matrix approximations during the finetuning phase. other peft techniques, include llm model weights compression through quantization techniques, cutting down storage and memory usage during both training and inference. by employing techniques such as normalfloat quantization @xcite , these methods aim to preserve model performance while considerably reducing llms' resource requirements, opening avenues for resource-efficient fine-tuning through their integration with peft methods like lora. large language models (llms) can be finetuned and used for translation of low-resource languages @xcite . the adaptm-llm system outperformed baselines from the loresmt2021 shared task @xcite in translating low-resource languages such as irish. the improvements were 14% greater in the english to irish direction (5.2 bleu points) compared to the irish to english direction (an increase of 117%, or 40.5 bleu points). in this paper, a llama-3 model with an 8-billions parameters model was finetuned to develop a sentence-level machine translation (mt) of four arabic dialects into msa. the rest of the paper is organized as follows. the related work is summarized in section 2. the dataset used for training and validation was detailed in section 3. in section 4, the system is presented. section 5 summarizes the study's key findings. ## data the development dataset provided by osact 2024 shared task @xcite one final comment about the model, when the model is fine-tuned using the osact dataset, the osact dataset doesn't have any "emirati", "jordonain" or "palestinian" sentences, so, during the generation of the translated sentences, the "emirati" sentence were considered as "gulf" dialect, and the "jordanian" and "palestinian" sentences were considered as "levantine" dialect. ## conclusions and future work in this paper, peft methods like quantization and lora were used to finetune a llama-3 model with 8 billion parameters, this model was adapted to translated between 4 different arabic dialects and modern standard arabic. the proposed method is efficient and it achieved a bleu score of 21.44 on the nadi 2024 task on da-msa machine translation when finetuned using the development dataset of the nadi shared task, and a bleu score of 16.09 when finetuned on the osact dataset. in the future, three interesting topics can be further explored: (1) we can further explore the effect of changing the model parameters on the results (i.e. learning rate, batch size, lora rank, etc.), (2) we can try different llms other than llama-3, and (3) we can assess the proposed method on other arabic dialects.