Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data (2023)

Abstract: Chat models, such as ChatGPT, have shown impressive capabilities and have been rapidly adopted across numerous domains. However, these models are only accessible through a restricted API, creating barriers for new research and progress in the field. We propose a pipeline that can automatically generate a high-quality multi-turn chat corpus by leveraging ChatGPT to engage in a conversation with itself. Subsequently, we employ parameter-efficient tuning to enhance LLaMA, an open-source large language model. The resulting model, named Baize, demonstrates good performance in multi-turn dialogues with guardrails that minimize potential risks. Additionally, we propose a new technique called Self-Distill with Feedback, to further improve the performance of the Baize models with feedback from ChatGPT.

## introduction the rapid advancement of natural language processing (nlp) techniques in recent years has led to the emergence of highly capable chat models, such as lamda @xcite , chatgpt (ope-nai, 2023a) and @xcite . these models demonstrate a remarkable ability to understand and generate human-like responses in a wide range of domains. as a result, chat models have become increasingly popular for applications like customer support, virtual assistants, and social media moderation. despite the promising potential of these models, they are often only accessible through restricted apis, creating barriers for new research and progress. furthermore, the limited availability of chat models poses obstacles for researchers and practitioners, hindering the growth of the nlp community. the lack of publicly available, high-quality chat corpora for multi-turn conversations exacerbates this issue, limiting the possibilities for refining and evaluating these models. in this paper, we propose a novel pipeline (shown in figure 1 ) to address these challenges by leveraging the capabilities of chatgpt to automatically generate a high-quality multi-turn chat corpus. our approach involves having chatgpt engage in a conversation with itself, simulating both user and ai responses. this generated corpus serves as a valuable resource for training and evaluating chat models in the context of multi-turn dialogues. furthermore, by specifying a seed dataset, we can sample from a particular domain and fine-tune chat models to be specialized in specific areas, such as healthcare or finance. to fine-tune large language models in a lowresource setting, we utilize a parameter-efficient tuning approach that effectively leverages the limited computational resources available. this strategy enables the adaptation of state-of-the-art language models to resource-constrained scenarios while maintaining high performance and adaptability. our primary focus is on improving an opensource large language model, llama @xcite , which we believe holds promise as an accessible alternative to proprietary chat models. by fine-tuning llama with our generated chat corpus, we create a new model, named baize (bái zé, a mythical creature in chinese folklore, who speaks human languages and knows everything). moreover, we propose self-distillation with feedback (sdf) as an alternative to reinforcement learning with human feedback @xcite , to further improve the performance of baize. baize is a chat model that can run on a single gpu, making it accessible for a broader range of researchers. to summarize, our main contributions in this paper are as follows: ## related work language models for chat since the success of gpt-2 @xcite , there have been many language models for chatting with humans. as an initial trial, dialogpt @xcite uses reddit data to fine-tune gpt-2 for open-domain dialogue. meena @xcite ) is a multi-turn open-domain chatbot with 2.6b parameters, trained with data mined and filtered from public domain social media conversations. following meena, lamda @xcite is a chat model with 137b parameters, pretrained on 1.56t words of public dialog data and web text. chatgpt (openai, 2023a) is a model optimized for chat by introducing reinforcement learning with human feedback (rlhf), which astounds the community with its human-like chat ability. @xcite is an improvement to chatgpt with newly added reasoning and multi-modal capability. @xcite use in-context learning with gpt-3 to augment a dialogue dataset. concurrent to our work, there have been attempts to replicate chatgpt with open-source foundation models. stanford alpaca @xcite uses self-instruct @xcite to collect data from gpt-3.5 in instruction learning format. then, the collected dataset is used to finetune llama @xcite . vicuna @xcite is a fine-tuned llama model trained on a chatgpt dialogue corpus crawled from sharegpt.com, a website for sharing chat-gpt dialogues. we will discuss the pros and cons of the data source of each model in section 3. parameter-efficient tuning conventional finetuning requires training all parameters in a large model, which can be inefficient as the numbers of parameters grows. adapter @xcite adds a tunable transformer layer while freezing the original layers. bitfit @xcite only tunes bias terms in the linear layers. diffpruning @xcite learns sparse weights that can be added to the original weights of the language model. prefix tuning @xcite fine-tunes prefix tokens inserted before the input. lora @xcite inserts tunable low-rank matrices into attention layers; lora achieves superior performance compared with conventional fine-tuning on gpt-3. concurrent to our work, there are attempts to use lora @xcite to fine-tune llama. alpaca-lora foot_1 follows the same recipe as alpaca while using lora for higher efficiency. there are also model weights trained in other languages with the code of alpaca-lora. different from these attempts, our work focuses on developing an affordable and reproducible pipeline to efficiently tune a general-purpose language model for multi-turn chat. ## data collection via self-chat in this section, we detail the methodology employed for generating a high-quality multi-turn chat corpus by leveraging chatgpt (gpt-3.5-turbo) to engage in a conversation with itself. this process, named self-chat, serves as the foundation of seed how do you fix a google play store account that isn't working? human: my google play store account is not working properly. how can i fix it? ai: there are several things you can try to fix a google play store account that is not working properly. first, check your internet connection and make sure it is working properly. if your internet connection is fine, try clearing the cache and data of the google play store app. if that doesn't work, try uninstalling and reinstalling the app. human: how do i clear the cache and data of the google play store app? ai: to clear the cache and data of the google play store app, go to settings, then apps & notifications, then find the google play store app and tap on it. from there, tap on storage, then clear cache and clear data. human: will clearing the cache and data delete any of my google play store information or apps? ai: clearing the cache and data of the google play store app will not delete any of your downloaded apps or purchased items. however, it may remove your preferences and settings within the app. human: what if clearing the cache and data doesn't work? ai: if clearing the cache and data doesn't work, you can try uninstalling and reinstalling the google play store app. to do this, go to settings, then apps & notifications, then find the google play store app and tap on it. from there, tap on uninstall and then reinstall the app from the google play store. the self-chat process involves utilizing chat-gpt to generate messages for both the user and ai assistant in a conversational format. we apply a template (shown in appendix a) to define the format and requirements, allowing the chatgpt api to continuously generate transcripts for both sides of the dialogue until a natural stopping point is reached. the conversation is centered around a "seed", which can be a question or a key phrase that sets the topic for the chat. in our own training of baize, we use questions from quora foot_2 and stack overflow foot_3 as seeds. a dialogue example generated with self-chat is shown in comparison with other data sources stanford alpaca @xcite uses self-instruct @xcite to collect data in instruction learning format. however, their instruction-input-output model base model type param. trainable param. gpu hrs data baize-v1-7b llama-7b sft 7b 17.9m 9 quora, stack overflow, alpaca baize-v1-13b llama-13b sft 13b 28.0m 16 quora, stack overflow, alpaca baize-v1-30b llama-30b sft 30b 54.6m 36 quora, stack overflow, alpaca baize-healthcare llama-7b sft 7b 17.9m 5 quora, medquad baize-v1.5-7b llama-7b sft 7b 17.9m 32 quora v2, stack overflow v2 baize-v1.5-13b llama-13b sft 13b 28.0m 64 quora v2, stack overflow v2 baize-v2-7b baize-v1.5-7b sdf 7b 17.9m 38 quora baize-v2-13b baize-v1.5-13b sdf 13b 28.0m 76 quora ## model training parameter-efficient supervised fine-tuning standard fine-tuning often requires vast amounts of computational resources, as well as high-quality and extensive datasets. however, given the limited availability of high-quality multi-turn chat corpora, it is crucial to adopt methods that are more efficient in terms of computational cost and data requirements. parameter-efficient tuning methods @xcite help achieve this goal by making better use of the available data and minimizing the need for extensive resource allocation. ## model settings during the training phase, we set the maximum length of the input sequence to 512/1024 for baize v1/v2 and the rank k in lora to 8. we initialize the llama checkpoints with the 8-bit integer format (int8) parameters released by @xcite , which remain fixed during training, thus reducing human: explain why this joke is funny: i'm a big fan of whiteboards. i find them quite re-markable. ## evaluation gpt-4 score we evaluate the performance of baize following vicuna's pipeline that uses @xcite to compare and score dialogue models. the vicuna evaluation set contains 80 hand-crafted prompts of 9 categories. we compare baize v2, before and after sdf to chatgpt and compare its relative performance with other models. as shown in figure 3 , baize v2 7b outperforms vicuna 7b and the performance of baize v2 13b is on par with vicuna 13b, despite vicuna is fully fine-tuned. note that we observe a positional bias in vicuna's evaluation pipeline. gpt-4 has a preference for the first answer than the second. to be human: ## conclusion and future work in this paper, we propose a pipeline that automatically samples seeds from specific datasets and collect high-quality dialogue corpus by leveraging chatgpt to chat with itself. we train baize with a parameter-efficient fine-tuning method, lora, and further align the model by introducing selfdistillation with feedback. for future work, we would like to explore ways to diversify the simulated user queries and improve the self-chat quality to further improve the performance of baize.