My own chatbot by fine-tuning GPT-2
Tags: machinelearning, python, deeplearning
Published: 2022-02-19 14:29:10

(Updated at 20, February, 2022)

# Introduction
In this post, I will fine-tune GPT-2, especially rinna's, which are one of the Japanese GPT-2 models. I am Japanese and most of my chat histories are in Japanese. Because of that, I will fine-tune "Japanese" GPT-2.

GPT-2 stands for Generative pre-trained transformer 2 and it generates sentences as the name shows. We could build a chatbot by fine-tuning a pre-trained model with tiny training data.

I will not go through GPT-2 in detail. I highly recommend the article [How to Build an AI Text Generator: Text Generation with a GPT-2 Model](https://dev.to/oursky/how-to-build-an-ai-text-generator-text-generation-with-a-gpt-2-model-4346) on dev.to to understand what is GPT-2 and what is a language model.

git repository: [chatbot_with_gpt2](https://github.com/ksk0629/chatbot_with_gpt2)

I would appreciate the author of the following two articles.
- [GPT-2で友達を再現して対話してみた](https://qiita.com/Yokohide/items/e74254f334e1335cd502)
- [How to Build an AI Text Generator: Text Generation with a GPT-2 Model](https://dev.to/oursky/how-to-build-an-ai-text-generator-text-generation-with-a-gpt-2-model-4346)

Thanks to the first author, I could build my chatbot model. The sources in my git repository are almost constructed with his codes. I just summarized them. Thanks to the second author, I could go through GPT-2.

# What is rinna
rinna is a conversational pre-trained model given from rinna Co., Ltd. and five pre-trained models are available on hugging face [[rinna Co., Ltd.](https://huggingface.co/rinna)] on 19, February 2022. rinna is a bit famous in Japanese because they published rinna AI on LINE, one of the most popular SNS apps in Japan. She is a junior high school girl. We could take conversations on LINE.

I am not sure when the models are published on hugging face, but anyways, the models are available now. I will fine-tune `rinna/japanese-gpt2-small` whose number of parameters is small. By the way, I wanted to use `rinna/japanese-gpt-1b` whose number of parameters is around one billion, but I couldn't because of the memory capacity on google colab.

# Process
I will suppose you have a google and git account and can use google colab.

Furthermore, I will use a chat history on LINE. If you have no account on the app, it is okay. All you have to do is prepare a chat history and modify the data. I know these processes are the hardest and most bothering things though. If you have the account, the following processes would work. Note that, if your LINE setting language is Japanese, you should change it to English until exporting a chat history because the following processes are supposing the setting language (not message language) is English.

## Prepare the environment
At the end of this process, your google drive is constructed as follows.
```
MyDrive ---- chatbot_with_gpt2.ipynb
           |
           |- config
           |    |- general_config.yaml
           |
           |- data
                |- chat_history.txt
```

- 1: Clone [chatbot_with_gpt2](https://github.com/ksk0629/chatbot_with_gpt2) repository on your local machine.

It is accomplished by running the following command on the git bash.
```
git clone https://github.com/ksk0629/chatbot_with_gpt2
```

- 2: Upload `chatbot_with_gpt2/chatbot_with_gpt2.ipynb` to the google drive.

- 3: Make a directory named `config` on your google drive and create `general_config.yaml` in the config folder.

`general_config.yaml` is as follows.
```yaml
github:
  username: your_github_username
  email: your_email
  token: your_access_token
ngrok:
  token: anything
```

The `ngrok` block is needless, but it is needed to avoid an error below.

- 4: Get a chat history from LINE.

We can get the history by following the official announcement [[Help centre - Chat history](https://help.line.me/line/android/?contentId=20007388#:~:text=1.,want%20to%20send%20the%20file.)].

- 5: Make a directory named `data` on your google drive and move the chat history to the directory.
 
## Prepare training data and build the model
- 1: Open `chatbot_with_gpt2.ipynb` on google colaboratory.

- 2: Run the cells in Preparation block.

The environment is prepared to get training data and build the model by running the cells.

- 3: Change `chatbot_with_gpt2/pre_processor_config.yaml`.

The initial yaml file is as follows.
```yaml
line:
  initial:
    input_username: "input_username"
    output_username: "output_username"
    target_year_list: "[2016,2017,2018,2019,2020,2021,2022]"
  path:
    input_path: "/content/gdrive/MyDrive/data/chat_history.txt"
    output_path: "chat_history_cleaned.pk"
```
You have to change at least initial block. The meaning of each line is as follows.

- input_username: a username of messages that you want to input into the model
- output_username: a username of messages that you want the model to output
- target_year_list: years that you want to use to train the model
- input_path: path to the raw chat history
- output_path: path to the cleaned data that is obtained by the following process

Note that, if you do not change output_path, then your training data would not be available after closing the notebook. Of course, it is available whilst the notebook is working.

- 4: Run the cell in Preprocessing data block.

The data is cleaned in the cell.

- 5: Change `chatbot_with_gpt2/model_config.yaml`.

The initial yaml file is as follows.
```yaml
general:
  basemodel: "rinna/japanese-gpt2-xsmall"
dataset:
  input_path: "chat_history_cleaned.pk"
  output_path: "gpt2_train_data.txt"
train:
  epochs: 10
  save_steps: 10000
  save_total_limit: 3
  per_device_train_batch_size: 1
  per_device_eval_batch_size: 1
  output_dir: "model/default"
  use_fast_tokenizer: False
```

You have to change input_path in dataset block to the path to the cleaned data, which is specified in `pre_processor_config.yaml`. You can change basemodel to rinna/japanese-gpt2-small, but others (medium and 1b) would not work because of a lack of GPU memory as I mentioned in What is rinna section.

- 6: Run the cells in Training data preparation and Building model block.

That is all! After running this cell, all you have to do is wait for a while. You would see your model file in the directory that is specified in `model_config.yaml`.

# Let's talk to the model
Again, all you have to do is run the only one cell in Talking with the model block. Then, the source code is running and you could talk with the model, like the following.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sais8av6ghwium6oz0pa.png)

# Conclusion
I fine-tuned GPT-2 with my chat history on LINE. I certainly did it, but there are the following problems as you could see in Let's talk to the model section.

- There is unnecessary line `Setting 'pad_token_id' to 'eos_token_id':2 for open-end generation.` in each conversation.
- There are some tokens, like `<br:`, `[<unk>hoto]<br///`, and `<br/ゥ>`, that disturb coherence sentence.
- The model did not reply well.

The first response
```
帰ったんか
おつかれさま!
```
looks quite good because "おっす" means "Hey" and the response means "You are home. You’ve got to be exhausted". Something like these. But the others look wrong. To improve the model, I could clean training data more and I need to understand GPT-2 and the source codes.

If you have any suggestions, comments, or questions about this article, please comment below. I'd appreciate it.