H yper T 5: Towards Compute-Efficient K orean Language Modeling (2023)

Abstract: Pretraining and fine-tuning language models have become the standard practice in industrial natural language processing (NLP), but developing and deploying general-purpose language models without the abundant computation or data resources is a real-world issue faced by smaller organizations or communities whose main focus is languages with less accessible resources (e.g., non-English). This paper explores the sequence-to-sequence (seq2seq) language model architecture as a more practical and compute-efficient alternative to the decoder-oriented approach (e.g., GPT-3), accompanied by novel findings in compute-optimality analyses. We successfully trained billion-scale Korean-language seq2seq language models that strongly outperform other competitive models in Korean benchmarks. Moreover, we demonstrate that such language models can be more efficiently utilized by employing a heavy pre-finetuning strategy, by showcasing a case study on dialog-task adaptation. Our case study shows that adopting language models with more readily available domain-specific unlabeled data greatly improves fine-tuning data efficiency in low-resource settings.

## introduction pretraining large-scale transformer-based language models and finetuning them for specific tasks have become the cornerstone of modern nlp pipelines. among various transformer-based language model architectures proposed in the field, generative decoder-based architectures, such as the gpt family @xcite , have gained more traction from their impressive ability to scale well into large language models (llms) @xcite and follow highlevel natural language instructions with few or even in the absence of demonstrations @xcite . ## related work plms and efficiency as we gain more understanding of core plm architectures @xcite @xcite and their scaling laws @xcite , research efforts for improving their training efficiencies have diverged in various directions, including improving the scaling curve @xcite , maximizing optimality @xcite , and efficient fine-tuning @xcite . our work offers a comprehensive overview and case study of optimizing the language modeling efficiency for a less resourceful language. non-english language models previously, significant works have explored pretraining language models on multiple languages to support lowresource languages and maximize cross-lingual knowledge transfer without explicit supervision @xcite @xcite . recently, lan-guage models that target specific non-english languages started to become more common @xcite @xcite , especially low-resource or non-latin languages that share fewer commonalities with english. there have been several korean textto-text transformers proposed in the past, such as kobart * and ke-t5 @xcite , but our work, among other korean seq2seqs, is the first to systematically achieve powerful billion-scale seq2seq models and conduct extensive analyses in terms of efficiency and performance. dialog-oriented language models adapting language models for the purpose of building dialog agents has been a long-standing goal in the language modeling community @xcite @xcite . however, from the industrial application perspective, building dialog response generators is not the only task that can benefit from the advances in language modeling. in a more recent line of work, several approaches have been proposed to prepare language models for various dialog-related tasks @xcite @xcite . in parallel to dialog adaptation, multi-task fine-tuning is another line of work that covers dialog-related tasks, as a subset of dialog-related tasks is included in the multi-task set, and expanding the task set to cover dialog-related tasks is trivial @xcite . ## hypert5 this section describes the details of the pretraining corpus, pretraining strategy, and evaluation methods. ## conclusion in this paper, we introduced hypert5 and di-aloght5 as state-of-the-art on korean language modeling. we also demonstrated the feasibility of performing resource-aware strategization for language models. through the compute-optimality analyses, we found that the seq2seq architecture may be more cost-efficient than decoders below a certain compute-budget threshold. for future work, we wish to generalize the domain adaptation approach and study the efficacy of multi-task learning @xcite from the industrial perspective. furthermore, we look forward to conducting comprehensive investigations into cross-architectural optimality. and the corresponding news topic labels. the dataset has 45.6k training data samples, 91k validation data samples, and 91k test data samples.