Application of Mix-Up Method in Document Classification Task Using BERT (2021)

Abstract: The mix-up method (Zhang et al., 2017), one of the methods for data augmentation, is known to be easy to implement and highly effective. Although the mix-up method is intended for image identification, it can also be applied to natural language processing. In this paper, we attempt to apply the mix-up method to a document classification task using bidirectional encoder representations from transformers (BERT) (Devlin et al., 2018). Since BERT allows for two-sentence input, we concatenated word sequences from two documents with different labels and used the multi-class output as the supervised data with a one-hot vector. In an experiment using the livedoor news corpus, which is Japanese, we compared the accuracy of document classification using two methods for selecting documents to be concatenated with that of ordinary document classification. As a result, we found that the proposed method is better than the normal classification when the documents with labels shortages are mixed preferentially. This indicates that how to choose documents for mix-up has a significant impact on the results.

## introduction the high cost of constructing training data is always a problem when solving natural language processing (nlp) tasks using machine learning approaches. several attempts have been made to solve this problem. one of the most recent methods for constructing training data is data augmentation @xcite . data augmentation methods can be divided into two types: processing and generation. for image identification, even if an image in the training data is flipped or cropped, the image label does not change. this means that the training data can be increased by adding such processed images to the training data. alternatively, the method for generating artificial data using generative adversarial network can be considered as a type of data augmentation. the mix-up method is one of the methods for generating data augmentation. it is highly effective and can easily be implemented. although the mix-up method is used for image identification, it can also be useed for nlp. ## proposed method if we adopt the mix-up method of the previous study for bert, we will have a problem. in the methods of the previous study @xcite , it is necessary to create feature vectors of the documents before learning the neural network (nn) (figure 2 ). however, document classification using bert obtains the feature vectors of the documents during the nn learning process (figure 1 ), which is a different order from the methods used in the previous study. if we were to adopt the method of the previous study, the calculation of feature vectors by bert would be done outside the learning process. therefore, high classification accuracy cannot be expected. figure 2: when to use prior research methods in bert in this paper, we propose the following method. ## consideration given the result, we obtained that the selection method of the documents to be mixed has a great influence on the accuracy. in this experiment, it is effective to prioritize mixing documents with labels shortages. we would try different methods and conclude. in this experiment, we used two documents with equivalent values (ratio 0.5 : 0.5). however, we think that it is worthwhile to try a method for varying the length of the concatenated words. mixup method is easier to implement than other data augmentation methods in nlp, and its accuracy has been improved. it is expected to become a mainstream method in the future. ## conclusion in this paper, we applied the mix-up method to a document classification task using bert. since bert allows for two-sentence input, two documents with different labels were combined and input. then, the labels were mixed by creating two 0.5 elements in a one-hot vector. in an experiment using the livedoor news corpus, which is japanese, we found that the proposed method is better than the normal classification when the documents with labels shortages are mixed preferentially. therefore, it indicates that the accuracy varies depending on the method of selecting documents to be mixed.