Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of V ietnamese Large Language Models (2024)

Abstract: Recent advancements in large language models (LLMs) have underscored their importance in the evolution of artificial intelligence. However, despite extensive pretraining on multilingual datasets, available open-sourced LLMs exhibit limited effectiveness in processing Vietnamese. The challenge is exacerbated by the absence of systematic benchmark datasets and metrics tailored for Vietnamese LLM evaluation. To mitigate these issues, we have finetuned LLMs specifically for Vietnamese and developed a comprehensive evaluation framework encompassing 10 tasks and 31 metrics. We observe that finetuning can help LLMs transfer knowledge across languages, serving as an efficient way to bolster their capabilities in non-English languages. Moreover, our analysis indicates that larger models can introduce more biases and uncalibrated outputs and the key factor influencing LLM performance is the quality of the training or finetuning datasets. These insights underscore the significance of meticulous finetuning with high-quality datasets in enhancing LLM performance.

## introduction large language models (llms) such as @xcite , bloom @xcite , llama-2 (touvron et al, 2023), mistral @xcite , @xcite , gemma @xcite have made significant contributions to the field of natural language processing (nlp). despite their advancements, a gap remains in their specialization for many languages, including vietnamese. this paper addresses the development and evaluation of vietnamese-centric llms. vietnam, with a population surpassing 100 million, ranks as the 16th most populous country globally. current models exhibit limitations in effectively handling vietnamese nlp tasks, especially in accurate comprehension and response @xcite . consequently, there is an increasing demand for a robust, dedicated vietnamese llm. several factors constrain the practical application of llms. concerns regarding the precision, inherent biases, potential toxicity, and fairness of their outputs are notable obstacles @xcite @xcite . moreover, there is a lack of research evaluating llms in the vietnamese context. to facilitate the effective use of state-of-the-art llms for vietnamese speakers, thorough evaluations are essential prior to their widespread use. such evaluations not only ensure the reliability of these llms but also highlight areas where these llms could be better. this leads to developing targeted reinforcement learning strategies to rectify these issues in the next phase. in response to the aforementioned challenges, we aim to develop open-source vietnamese llms. initiating an llm from scratch is impractical due to the scarcity of extensive training datasets and limited computational resources. however, the advent of qlora @xcite , incorporating quantization techniques @xcite and @xcite , provides an efficient approach for fine-tuning llms, particularly in resource-constrained environments. we employ fine-tuning on the llama-2, mixtral 8×7b, gemma, and conduct a comprehensive evaluation of vietnamese llms across various scenarios and settings. throughout the thorough evaluation process, we observe the following: (i) larger language models exhibit unseen capabilities compared to smaller counterparts; (ii) larger language models tend to manifest more biases, produce uncalibrated results, and are more susceptible to the influence of input prompts; (iii) the quality of training or fine-tuning datasets is the key for unlocking llm performance. our key contributions include: ## results and discussion we present the overall capacities of evaluated llms in figure 1 , separating commercial and open-sourced models across six aspects, including general performance, robustness under weaker prompts, performance with chain-of-thought (cot), ability to deal with unfair input (fairness) and toxicity, bias in generated outputs. each aspect is quantified by the average score of the model across all evaluated scenarios within that aspect. for each scenario, we present the standard deviation for each metric by using bootstrapping @xcite , wherein the process involves (i) drawing random samples with replacement from the original dataset, (ii) computing the metric for each sampled subset and (iii) iteratively repeating steps (i) and (ii) for a total of 1000 iterations to ascertain the standard deviation across these repetitions. overall, gpt-4 demonstrates the highest performance across all tasks. however, the gpt family exhibits more biases than the others. our finetuned models outperform their base model, llama-2. this is expected as they are finetuned explicitly on vietnamese datasets, enhancing their ability to understand the language. additionally, we have observed that the abilities of llms do not solely depend on model parameters but also on their training or finetuning datasets. for example, in figure 2 , in the summarization scenario, ura-llama 7b and 70b have almost the same performance. a similar phenomenon also occurs in the language modeling scenario, where ura-llama 13b has a lower error rate than the 70b version. larger models do not always guarantee better performance and might per-form worse than smaller ones if not trained on these specific data types. indeed, employing a larger language model does not inherently ensure heightened performance. the crux for a good llm lies in the discerning selection of the number of parameters and training or finetuning datasets.