Walia- LLM : Enhancing A mharic- LL a MA by Integrating Task-Specific and Generative Datasets (2024)

Abstract: Large language models (LLMs) have received a lot of attention in natural language processing (NLP) research because of their exceptional performance in understanding and generating human languages. However, low-resource languages are left behind due to the unavailability of resources. In this work, we focus on enhancing the LLaMA-2-Amharic model by integrating task-specific and generative datasets to improve language model performance for Amharic. We compile an Amharic instruction fine-tuning dataset and fine-tuned LLaMA-2-Amharic model. The fine-tuned model shows promising results in different NLP tasks. We also explore the effectiveness of translated instruction datasets compared to the dataset we created. Our dataset creation pipeline, along with instruction datasets, trained models, and evaluation outputs, is made publicly available to encourage research in language-specific models.

## introduction large language models (llms) such as gpt series @xcite , llama-2 @xcite , phi2 @xcite , mistral @xcite , @xcite , palm @xcite , gemini @xcite , bloom @xcite , have exhibited exceptional performance in understanding and generating human language, showcasing a range of capabilities from basic linguistic comprehension to complex text generation. llama-2 @xcite , a family of pre-trained and fine-tuned large language models (llms), demonstrated impressive performance across multiple tasks, particularly in dialoguebased interactions. regardless of these achieve1 for data generation pipeline, see https: //github.com/ethionlp/afri-sft-data . for models and datasets, refer to https://huggingface. co/ethionlp . ments, llama-2 pre-training supports a small number of languages, which does not include lowresource languages like amharic. this makes adapting llms to low-resource languages that are not included a significant challenge. adopting these llms to local languages requires the preparation of a quality instruction dataset. amharic is one of the semitic languages under the afroasiatic language family spoken in ethiopia with more than 57m speakers. there are numerous task-specific datasets for amharic @xcite compared to other ethiopian languages. this paper focuses on enhancing the llama-2-amharic @xcite ) model with quality datasets that are created by converting existing datasets in english into instruction-based amharic datasets. furthermore, we create new instruction datasets following the approach by @xcite . llama-2-amharic model by @xcite was created by pre-training llama-2 7b model using open-source amharic and translated corpus. after performing vocabulary expansion and pre-training, @xcite fine-tuned the created model by translating english instruction datasets into amharic using commercial translation tools. in our research, we aim to improve the performance of the amharic llama model by integrating task-specific and generative datasets, as shown in ## related work the introduction of open-source llms like llama-2 @xcite enabled the creation of several language models that focus on specific applications. this application gives more capabilities for these llms by teaching them to use tools @xcite , write code (roziere et al., 2023), understand videos @xcite , or work for different languages @xcite . to achieve remarkable understanding and generation abilities, llms require large training data and huge compute resources @xcite . the work by @xcite explores how llms' generation, natural language understanding, and problem-solving abilities relate to the data they are trained on and its composition. this work suggests that the amount of composition data is more important for these abilities to show in a lowresource scenario. using self-instructed fine-tuning, the work by @xcite ; @xcite ; @xcite showed a new approach to align the generation outputs of the generative models through the application of nlp tasks. these tasks are structured around natural language instruction templates, providing a novel means to guide the model's generation process toward better adherence to task-specific requirements. llama-adapter @xcite ) also shows that it is possible to reduce the fine-tuning time of llama-7b by introducing lightweight adapters on top of the model. acquiring and preparing a dataset for instruction fine-tuning presents a significant challenge due to the extensive labor and resources required. there are several ways of acquiring instruction data, including manual dataset creation, using generative models @xcite , or using machine translation instruction data for training llms for specific languages @xcite . fine-tuning llms such as llama-2 for specific tasks is an area of exploration as well. advanced language model-based translator (alma) @xcite outperformed state-of-the-art (sota) no language left behind (nllb) @xcite model mt task. they worked on fine-tuning monolingual data and subsequent finetuning with parallel data. apart from llama-2, @xcite worked on mistral 7b fine-tuning for medical domain machine translation, where they showed improvement in spanish to english translation from baseline performance. after the llama-2 was released, researchers successfully adapted the model for other languages. the work by @xcite involved creating a unique tokenizer for chinese, extending the pretraining phase, and then fine-tuning the model. this work incorporates secondary pre-training using chinese data and fine-tunes the model with chinese instruction datasets. the result shows a significant enhancement of the model's ability to comprehend and execute instructions. ## experiments we followed chinese llama @xcite experiments to perform supervised fine-tuning (sft) on our dataset using different types of the dataset we created. figure 2 shows the full training pipeline that summarizes the overall experiment steps we followed. we used codes available on the chinese-llama-alpaca foot_8 repository. we used 4 a100 gpus with the default parameters in the repository. all training is done for three epochs. all models and evaluation codes will be available in our repository. for the mt task, we also worked on m2m100 @xcite and nllb @xcite models. during the evaluation of the models, we used gpt-4-0613 for gpt-4. for our llama-based models, we used fixed generation parameters across the models. we also evaluated various models that purportedly support this language but excluded them due to their inability to perform the required tasks. this is further discussed in appendix a. our main experiment includes: ## results below, we discuss the performance of each model we tested by task type and evaluation strategy. ## conclusion and future works in this work, we created amharic instruction finetuning dataset, evaluated the performance of existing and our fine-tuned models in the new dataset, and explored the effect of carefully curated datasets on the models' performance. we observed a possibility of reusing task-specific datasets to improve the generation and task performance of the existing llama-2-amharic model. this shows the advantage of task dependant datasets in overall tasks. our data generation pipeline that generates instruction datasets from task datasets can be used for the generation of similar datasets for other languages given template instructions. we are working on this kind of dataset for all languages included in masakaner @xcite , masakhanews @xcite , afrisenti @xcite and more to improve multilingual llama models. we plan to opensource the instruction datasets with the generation code. moving forward, we aim to improve both the quality and volume of the data utilized. taskspecific dataset creations are meant to complement, not replace, language-specific instruction dataset creations, and we plan to work on creating quality instruction datasets in addition to using existing task datasets. we also plan to explore the relevance of using llms for evaluation in low-resource languages like amharic and incorporate llms to evaluate our llms.