V i GPTQA - State-of-the-Art LLM s for V ietnamese Question Answering: System Overview, Core Models Training, and Evaluations (2023)

Abstract: Large language models (LLMs) and their applications in low-resource languages (such as in Vietnamese) are limited due to lack of training data and benchmarking datasets. This paper introduces a practical real-world implementation of a question answering system for Vietnamese, called ViGPTQA, leveraging the power of LLM. Since there is no effective LLM in Vietnamese to date, we also propose, evaluate, and open-source an instruction-tuned LLM for Vietnamese, named ViGPT. ViGPT demonstrates exceptional performances, especially on real-world scenarios. We curate a new set of benchmark datasets that encompass both AI and human-generated data, providing a comprehensive evaluation framework for Vietnamese LLMs. By achieving state-of-the-art results and approaching other multilingual LLMs, our instruction-tuned LLM underscores the need for dedicated Vietnamese-specific LLMs. Our open-source model supports customized and privacy-fulfilled Vietnamese language processing systems.

## introduction large language models (llms), especially instruction-following models have achieved remarkable success in a wide range of natural language processing (nlp) tasks, demonstrating their ability to understand and generate human-like text. these models, including proprietary models such as chatgpt, bingai, and bard, and open-source models such as llama @xcite , alpaca @xcite , and vicuna @xcite , have been trained on vast amounts of text data, enabling them to learn intricate language patterns and capture semantic nuances. while llms have shown impressive performance on various languages, there has been a noticeable gap in efforts dedicated to developing llms for vietnamese, a low-resource language. vietnamese possesses its own linguistic characteristics and contextual nuances, making it imperative to explore and optimize language models tailored to this unique language. additionally, initial efforts for evaluating performances of multilingual llms @xcite have been carried out only for dominant languages, such as english and chinese. as a result, it is important for thorough evaluation of vietnamese llms. comprehensive benchmarking will offer insights into the capabilities and potential limitations of llms when used with vietnamese, enabling researchers and developers to fine-tune and optimize these models for optimal performance. llms have significantly empowered various applications across multiple domains @xcite . for example, they can be used to create question answering systems that provide more accurate and informative responses than traditional systems. figure 1 illustrates our real-world question answering system called vigptqa, in which llms are combined with an embedding module to query and extract input from users for factual and referenced responses. a crucial point to highlight is that these applications require a highly proficient llm to be feasible. in this work, we propose, implement, evaluate, and open-source a vietnamese monolingual instruction-tuned llm, named vigpt. by finetuning a pre-trained language model with specific instructions, vigpt aims to enhance its perfor-mance and adaptability to the vietnamese language. in addition to the general-purpose vigpt model, we also introduce a law domain-specific variant, named vigpt-law, to power our vigptqa system. vigpt-law is specifically trained on a legal text corpus, allowing it to generate more accurate and informative responses to law-related queries. moreover, we curate a comprehensive set of benchmark datasets specifically designed for evaluating vietnamese llms. these datasets include both ai-generated and human-generated data, covering a wide range of emergent capabilities evaluations and task-specific challenges. this diverse range of benchmark datasets offer a standardized framework for assessing and comparing the performances of vietnamese llms. our main contributions are listed as follows: source code, benchmark datasets, and model weights are made publicly available at https: //github.com/dopikai-labs/vigpt for further advancement of customized and privacy-fulfilled systems for vietnamese language processing. ## related work the development of large language models (llms) has gained significant attention in the natural language processing (nlp) community, leading to a plethora of research efforts on various aspects of llms for different languages @xcite @xcite @xcite . instruction-following language models have emerged as a promising direction to enable llms to generate targeted and controlled outputs based on user instructions. recent studies have explored various methods for finetuning llms with instruction data, enhancing their performance on specific tasks and domains @xcite @xcite @xcite . prior research on vietnamese language processing has been carried out to pre-train vietnamese monolingual language models @xcite , with downstream application to tasks such as question answering @xcite , named entity recognition @xcite , and text summarization @xcite , exploring challenges specific to vietnamese language. existing models show promise in traditional nlp tasks but lack dedicated efforts for vietnamese-specific llms and real-world applications. to the best of our knowledge, as of the time of writing this work, this is one of the first studies to introduce a billionparameter vietnamese instruction-tuned llm with thoroughly benchmarked results, emphasizing realworld applicability. ## methodology in this section, we will outline our fine-tuning approach and data curation process for training the generic vigpt model and adapting it to the specific domain of vietnamese laws, referred to as vigpt-law. our primary objective is to expand the boundaries of llm for vietnamese, thereby empowering our vigptqa system. ## experiments in this section, we conduct experiments on traditional benchmarking tasks for vietnamese language models to demonstrate the effectiveness of our proposed large language model, vigpt. we first evaluate vigpt and its variants on a newly curated vietnamese law question answering dataset to assess the ability of the llms in powering vigptqa system. then, we thoroughly benchmark various characteristics of vigpt, including truthfulness and reasoning capability, and compare it against multilingual llms. the results show the strong capability of our model and highlight areas for future improvement. ## conclusion this work focuses on advancing large language models for the vietnamese language and developing real-world applications, namely vigptqa. we introduce vigpt, an instruction-following llm for vietnamese. we propose novel datasets for finetuning language models (instruction data), adaptation to a specific domain (vilaws dataset), and benchmarking datasets for vietnamese llms. initial evaluations showcase the usefulness of vigpt and variants on downstream tasks, and its emergent capability, compared to other multilingual llms. we provide public access to our datasets, model codes, and weights, fostering collaboration and enabling reproducibility. our research contributes to the development of llms for vietnamese, paving the way for specialized and efficient llms.