What Changes Can Large-scale Language Models Bring? Intensive Study on H yper CLOVA : Billions-scale K orean Generative Pretrained Transformers (2021)

Abstract: GPT-3 shows remarkable in-context learning ability of large-scale language models (LMs) trained on hundreds of billion scale data. Here we address some remaining issues less reported by the GPT-3 paper, such as a non-English LM, the performances of different sized models, and the effect of recently introduced prompt optimization on in-context learning. To achieve this, we introduce HyperCLOVA, a Korean variant of 82B GPT-3 trained on a Korean-centric corpus of 560B tokens. Enhanced by our Korean-specific tokenization, HyperCLOVA with our training configuration shows state-of-the-art in-context zero-shot and few-shot learning performances on various downstream tasks in Korean. Also, we show the performance benefits of prompt-based learning and demonstrate how it can be integrated into the prompt engineering pipeline. Then we discuss the possibility of materializing the No Code AI paradigm by providing AI prototyping capabilities to non-experts of ML by introducing HyperCLOVA studio, an interactive prompt engineering interface. Lastly, we demonstrate the potential of our methods with three successful in-house applications.

## introduction due to its remarkable zero-shot and few-shot performances, gpt-3's in-context learning has gained significant attention in the ai community @xcite . in the in-context learning approach, discrete prompts that consist of a natural language task description and few-shot examples control * equal contribution. large-scale language models (lms) to infer predictions for the target task. using openai's gpt-3, studies have proposed methods that can further boost in-context learning performance of gpt-3 @xcite . recently, prompt-based learning methods have been reported to improve the performances of bert, gpt-3, and t5 without any parameter updates of the main model @xcite @xcite . we consider the three following practical issues of using gpt-3. first, the language composition of the training corpus is heavily skewed towards english with 92.7%. this makes it difficult to apply it to tasks in other languages. we also know little about how to train similar models in another language with different linguistic properties, and where the originally proposed methods will be naturally applied and where they might fail. second, while it is pragmatic and useful to know the capabilities of various sized models considering the operation costs of using large-scale lms, we only have access to a thorough analysis of models of @xcite but none in between. lastly, advanced prompt-based learning methods that require backward gradients of inputs, including continuous prompt-based tuning, have not yet been experimented for an in-context largescale lm learner. here we address these issues by introducing a non-english gpt-3 with various parameter sizes and intensively investigating their capabilities on diverse real-world classification and generation tasks under in-context few-shot learning and promptbased optimization. we introduce a korean in-context large-scale lm with 82b parameters, i.e., hyperclova. this is the first discovery on near 100b-scale non-english lm. we present the corpus composition of korean datasets used for hy-perclova, and describe how we crawl and refine such data to collect 561b tokens of korean corpus ( §3.1). we also design a new korean tokenization method based on the agglutinative property for hyperclova. we use byte-level bpe @xcite with a morpheme analyzer ( §3.3). our results show that such tokenization strategy is important for the performance of downstream tasks in large-scale in-context learning ( §4.4). we report the state-of-the-art in-context learning performance of our model on korean datasets in zero and few-shot settings ( §4.2). in addition, we are the first to discovery the applicability of the continuous prompt-based optimization techniques, such as p-tuning @xcite , to largescale lms. hyperclova leveraged by p-tuning achieves outstanding results for both classification and generation tasks. also, we investigate the effects of p-tuning on two mid-size hyperclova ( §4.3). subsequently, we illustrate the versatility of operating a single large-scale lm in the ai industry. developing an ai product involves heavy collaboration among non-technical experts. this incurs substantial communication overhead because the level of technical abstraction varies across job functions. we introduce hyperclova studio, an interactive prompt engineering interface which provides gui and api interfaces like the openai playground 1 . the interactive interface helps nonexperts of ml to easily use hyperclova for prototyping ai products. we also share three in-house application scenarios using hyperclova studio as novel task environments. with minimal efforts of a domain expert in these scenarios, hyperclova presents performances qualitatively comparable to human experts, despite their difficulty in designing their objective function and training data ( §5.2). we then discuss how the functionality of hy-perclova studio can be extended. for example, hyperclova studio can provide input gradient functionality, to fine-tune small prompt encoder with few number of instances, thus enabling any user to achieve state-of-the-art performance using 1 https://beta.openai.com/ hyperclova ( §5.3). finally, we discuss the possibility of no/low code ai paradigm using hyper-clova studio, in which one large lm empowers people to create ai systems with no need for training individual deep learning models or collecting and labeling suitable datasets ( §5.4). our contributions are summarized as: ## discussion on industrial impacts what change can large-scale lms bring? we claim "accelerating the life-cycle of nlp ml operation" as one of the possible answers. unlike the protocol of most deep learning research where a model is trained with a well-collected dataset by ml experts and its corresponding well-defined objective function, there are several additional steps to make an ai product in a production-level pipeline, which yield tremendous communication overhead and costs. a platform with large-scale lms may make huge progress by allowing only one non-developer, such as a service designer, to build the prototype system. section 5.1 introduces hyperclova studio as our distribution method of hyperclova. section 5.2 introduces our three in-house usages of hy-perclova studio. section 5.3 discusses possible extensions of hyperclova studio, prompt-based optimization, input module, and output module. using the evidence above, section 5.4 discusses no/low code ai paradigm. ## conclusion we present hyperclova, various billions-scale korean-centric lms. in particular, hyperclova with 82b parameters shows state-of-the-art incontext zero-shot and few-shot performance and can further be boosted by prompt-based learning method. we will share our model by hyperclova studio where non-developers can easily build their own ai-backed products. we argue that a framework like hyperclova studio can potentially achieve no code ai paradigm and hope that cases of such paradigm become popular, although opportunities and challenges coexist. our goal is to create an ecosystem using hy-perclova studio in korea and help people not familiar with machine learning make their own ai models.