A fri I nstruct: Instruction Tuning of A frican Languages for Diverse Tasks (2024)

Abstract: Large language models (LLMs) for African languages perform worse compared to their performance in high-resource languages. To address this issue, we introduce AfriInstruct, which specializes in instruction-tuning of multiple African languages covering various tasks. We trained the LLaMa-2-7B using continual pretraining and instruction fine-tuning, which demonstrates superior performance across multiple tasks. Our mixed task evaluation shows that our model outperforms GPT-3.5-Turbo and other baseline models of similar size. Our contributions fill a critical gap of LLM performance between high-resource and African languages.

## introduction the development of large language models (llms) has revolutionized the field of natural language processing (nlp), enabling significant advancements in tasks such as machine translation @xcite @xcite , sentiment analysis @xcite @xcite , and question answering @xcite @xcite . however, the vast majority of these breakthroughs have been concentrated on high-resource languages (hrls), particularly english, due to the abundance of training data and resources available @xcite @xcite . in contrast, low-resource languages (lrls), such as many african languages, have been largely left behind by the latest developments in nlp, despite their importance to millions of speakers worldwide @xcite @xcite @xcite . gpt-3.5-turbo ## experimental settings abalation study to determine the importance of hyperparameters, we evaluated the effectiveness of continual pretraining and lora @xcite fine-tuning with different ranks using llama2-7b foot_2 . the continued pretraining on llama-2-7b using the wura dataset was done using the run_llmmt.py script provided in the alma codebase @xcite . we used eight nvidia-a100 40gb gpus, and we ensured that we trained 1b tokens by training 8000 steps, with a perdevice batch size of 2 and 16 gradient accumulation steps. since the sequence length is 512, we get 8 * 8000 * 2 * 16 * 512 ≈ 1b tokens. the lora instruct-tuning process was limited to 500 steps, which is significantly less than 1 epoch, but sufficient to observe the convergence of model training. we used the unsloth repository to fine-tune the model during this experiment. comparative study in comparison with other base models, we conducted one epoch fine-tuning with lora using axolotl. we used 2 x a10 24 gpus to fine-tune the continual pre-trained model based on llama2-7b. for important parameters, we employed lora rank: 32, lora alpha: 16, and lora dropout: 0.05, and learning rate: 0.00002. note that based on the result of experiment one, we assessed the model on three evaluation tasks: translation, topic classification, and question answering ( ## result and analysis our ablation study demonstrates the effectiveness of lora rank and continual pretraining ( in the comparative verification by rank, no significant improvement in accuracy was observed from rank 32 to 256, indicating that knowledge transfer does not vary significantly within this rank range. however, an improvement of about 5 points was seen at rank 512 compared to rank 32, suggesting that higher-rank lora training can be expected to facilitate certain levels of knowledge injection. due to the dataset and computer resources available, training at rank 512 for one epoch was not feasible, so we conducted lora fine-tune with one epoch of training at rank 32. additionally, we conduct this comparative evaluation on llama3-8b ( ## conclusion this paper showcases our advancement for african languages through the development of afri-instruct via afriinstruct-model-7b and afriinstruct-data. the afriinstruct-data dataset supports instruction tuning of diverse tasks such as machine translation, topic classification, and more. afriinstruct-model-7b, enhanced by continual pretraining with the wura dataset and fine-tuning with the lora technique, excels particularly in question-answering, outperforming prominent models like llama2-7b, llama3-8b, and gpt-3.5-turbo. this implies the effectiveness of targeted instruction tuning datasets for pretraining and fine-tuning of african languages, addressing the critical need for comprehensive datasets and models for low-resource languages.