R o QL lama: A Lightweight R omanian Adapted Language Model (2024)

Abstract: The remarkable achievements obtained by open-source large language models (LLMs) in recent years have predominantly been concentrated on tasks involving the English language. In this paper, we aim to advance the performance of Llama2 models on Romanian tasks. We tackle the problem of reduced computing resources by using QLoRA for training. We release RoQLlama-7b, a quantized LLM, which shows equal or improved results compared to its full-sized counterpart when tested on seven Romanian downstream tasks in the zero-shot setup. Also, it consistently achieves higher average scores across all few-shot prompts. Additionally, we introduce a novel Romanian dataset, namely RoMedQA, which contains single-choice medical questions in Romanian.

## introduction transformer models @xcite represent the state-of-the-art solution adopted by natural language processing (nlp) tasks @xcite . due to current breakthroughs in computational capabilities, models were scaled in terms of parameters, acquiring new remarkable abilities in terms of natural language understanding @xcite . as a result, a series of proprietary and open large language models (llms) were created. one major downside of llms is the enormous amount of computational resources and training data they require. democratizing llms constitutes a vital research direction, increasing the possibility of breakthroughs. in this sense, we release a new lightweight romanian language-adapted llm with 7 billion parameters and quantized to 4 bits by employing the state-of-the-art quantized lora (qlora) training technique @xcite . we evaluate our model on several romanian datasets, covering seven tasks and comparing it to its original counterpart. our results showed that roqllama-7b outperformed the other llama models on four out of the seven tasks investigated using zero-shot prompting. furthermore, due to quantization, the model has a significantly smaller memory footprint, up to three times less than the base model. to summarize, the contributions of this work are: ## romedqa more publicly available datasets are needed for the romanian nlp tasks. to contribute to this area, we introduce romedqa, a dataset that amounts to 4,127 single-choice questions regarding the medical field in the romanian language. the dataset consists of advanced biology questions used in entrance examinations in medical schools in romania. each question has five possible answer choices, numbered from 1 to 5, with only one correct answer. see appendix b for a more detailed dataset description. ## evaluation we evaluate the roqllama-7b model and the original llama2 on seven romanian nlp tasks as follows: we show the original prompts used to evaluate the language models on each task, as well as their translation in english in appendix d. ## results the results using zero-shot prompting are depicted in we also evaluate the performance of the newly introduced roqllama-7b model on few-shot prompting (i.e., one-shot, three-shot, and five-shot). this allows us to analyze how the model improves when additional examples for each task are given. the results of this analysis are depicted in figure 1 , which outlines the variations in average scores when an increasing number of examples are presented to each model in the prompt. we can observe that roqllama-7b consistently outperforms both llama2-7b and llama2-7b-chat on all the few shots prompting tasks evaluated in this work. furthermore, because the language models can hallucinate and produce inadequate output for each classification task, we also evaluate the not followed instruction (nfi) score, which measures the percentage of samples on which the model fails to adhere to the given instructions. the following subsections present results on zero-shot prompting for each evaluation task. ## conclusions in this paper, we advance state-of-the-art nlp techniques for romanian and address the scarcity of romanian datasets. we introduce a lightweight llm for romanian and a new medical dataset of single-choice exam questions. roqllama-7b, a quantized version of llama2-7b, achieves higher average scores across romanian tasks while using three times less memory. romedqa is the first romanian dataset of medical questions and answers based on entrance exams for medical schools in romanian. it is valuable for training and testing llms in medical knowledge and language comprehension. future work includes enhancing the dataset with contextual information, adapting it for smaller models, and integrating the results into the romanian liro benchmark @xcite .