Cendol: Open Instruction-tuned Generative Large Language Models for I ndonesian Languages (2024)

Abstract: Large language models (LLMs) show remarkable human-like capability in various domains and languages. To bridge this quality gap, we introduce Cendol, a collection of Indonesian LLMs encompassing both decoder-only and encoder-decoder architectures across a range of model sizes. We highlight Cendol’s effectiveness across a diverse array of tasks, attaining ~20% improvement, and demonstrate its capability to generalize to unseen tasks and indigenous languages of Indonesia. Furthermore, Cendol models showcase improved human favorability despite their limitations in capturing indigenous knowledge and cultural values in Indonesia. In addition, we discuss the shortcomings of parameter-efficient tunings, such as LoRA, for language adaptation. Alternatively, we propose the usage of vocabulary adaptation to enhance efficiency. Lastly, we evaluate the safety of Cendol and showcase that safety in pre-training in one language such as English is transferable to low-resource languages, such as Indonesian, even without RLHF and safety fine-tuning.

## introduction indonesia is the fourth most populous country in the world, with around 280 million people spread across more than 17,000 islands within a humongous area of ∼2 million square kilometers. with such a large archipelago surrounding the country, digital services become immensely crucial, making indonesia the fourth largest internet user in the world, with ∼220 million users. despite the huge demand, the technology supporting indonesian digital businesses still lags compared to other much smaller countries. one aspect that it still left behind is the access to state-of-the-art large language model (llm) technology, such as chat-gpt (openai, 2023a) and gpt4 @xcite . although these llms support indonesian and its local languages, these llms often have much weaker language representation for such low-resource and underrepresented languages @xcite . the weak language representation in existing llms hurts their ability to generate responses in indonesian and other underrepresented languages. this also leads to inefficiency during inference due to the vocabulary mismatch, hence texts in these languages are tokenized into much longer tokens @xcite . additionally, these llms are more prone to safety issues, e.g., giving unsafe responses @xcite , hallucinations @xcite , and jailbreaking @xcite . to overcome the challenge of weak language representation in indonesian languages, we introduce cendol 2 , a series of large-scale instruction-tuned llms specifically tailored for handling indonesian indigenous languages. cendol covers both decoder-only and encoder-decoder llms that spread across various scales from 300m up to 13b parameters. various strategies are incorporated to enable instruction tuning across various scales. we assess the effectiveness of cendol on a comprehensive evaluation suite, covering various general nlp tasks (e.g., sentiment analysis, topic modeling, machine translation, summarization, etc.), local knowledge, and cultural values evaluations. our work highlights the following contributions: ## cendol collection in total, we create 53.5m prompts, covering a wide range of prompt types including nlp task-based prompts (41m), indonesian general knowledge prompts (6.2m), local language generative prompts (6.3m), and human-aligned prompts (8.2k). figure 2 shows the detailed coverage of cendol collection across different sources. ## cendol recipe in this section, we describe the configurations for preparing our cendol models and report the computational resources used in our experiments. ## evaluation suite we evaluate cendol on various aspects of language proficiency: nlu and nlg ( §5.1); generalization capability on unseen tasks and languages ( §5.1); as well as local knowledge and cultural commonsense ability ( §5.2). in addition, our evaluation includes the first indonesian safety evaluation for llms (appendix b). our evaluation suite consists of 10 local languages spoken in indonesia and spreads across 23 evaluation datasets. ## conclusion we introduce cendol, a collection of indonesian llms covering both decoder-only and encoderdecoder architecture over various model sizes, and cendol collection, a large-scale instruction-tuning dataset for indonesian and its local languages. we highlight the effectiveness of cendol on a wide range of tasks, achieving ∼20% improvement for both nlu and nlg tasks. cendol also generalizes to the local languages in indonesia. furthermore, we demonstrate our effort for human alignment through a supervised fine-tuning approach, which yields a significant improvement in terms of human favorability. we also discuss two limitations of cendol: 1) human-preferred response needs to be further enhanced with a better human alignment approach, and 2) despite their amazing performance on nlp tasks, cendol models still fall behind on capturing local knowledge and cultural values in indonesia. moreover, we analyze the generalization of cendol to unseen tasks and languages, the ineffectiveness and inefficiency of lora for language adaptation, vocabulary adaptation as an efficient tuning alternative, and the safety of cendol models.