Devil’s Advocate: Novel Boosting Ensemble Method from Psychological Findings for Text Classification (2021)

Abstract: We present a new form of ensemble method–Devil’s Advocate, which uses a deliberately dissenting model to force other submodels within the ensemble to better collaborate. Our method consists of two different training settings: one follows the conventional training process (Norm), and the other is trained by artificially generated labels (DevAdv). After training the models, Norm models are fine-tuned through an additional loss function, which uses the DevAdv model as a constraint. In making a final decision, the proposed ensemble model sums the scores of Norm models and then subtracts the score of the DevAdv model. The DevAdv model improves the overall performance of the other models within the ensemble. In addition to our ensemble framework being based on psychological background, it also shows comparable or improved performance on 5 text classification tasks when compared to conventional ensemble methods.

## introduction ensemble modeling is a technique that combines several submodels into a composite model. by diminishing model bias, and variance, ensemble techniques can improve overall model performance @xcite . in addition, ensemble techniques are also used to get confidence scores of model predictions for explainable models @xcite @xcite . for these advantages, ensemble has been used as the de facto standard for many classification tasks. ensemble methods such as soft-voting, hardvoting (hansen and @xcite , bagging @xcite , and boosting @xcite attempt to build submodels which have different views on the same data, which produces more robust predictions. research in psychology has shown that a high level of cohesion and group thinking can lead to poor decisions and premature solutions @xcite @xcite . people tend to follow majority in decision making even if the decisions are not reasonable. they are also more likely to rush to judgment and alternatives preferred by the majority @xcite . as @xcite put, 35% of the responses agreed with the majority and nearly everyone followed the incorrect majority at least once. when it comes to group decision making, groups often fall into ideas that are sub-optimal rather than take advantages of using all of the ideas. parallels can be drawn between this psychological phenomenon and some ensemble methods, especially in cases where the submodels all have similar architectures. devil's advocate is one of the most prominent methods used for fostering healthy dissent in human group decision making @xcite . it involves taking a position counter to the majority position. that is, devil's advocate takes an alternative position from the norms taken for granted in order to deepen the discussion through reasonable opposition. by doing so, the dissenter can increase independence of individuals' thoughts @xcite . by leveraging this principle from human decision making, we attempt to model the settings of devil's advocate and to improve the quality of decision making (in the computational model) and performance. the contributions of the present study can be summarized as follows foot_0 : ## experiment data. we use glove @xcite as pretrained embeddings. to increase model performance, we apply a word vector post-processing method called extrofitting @xcite . we prepare 3 topic classification datasets; dbpedia ontology (dbpedia) @xcite , yahooanswers (yahoo) @xcite , ag-news. we also prepared 2 sentiment classification datasets; yelp reviews (yelp) @xcite , imdb @xcite . the data information is presented in classifier. we choose textcnn @xcite as the submodel architecture of our proposed ensemble method. the model has two convolutional layers with 32 channels and 16 channels, respectively. we adopt multiple sizes of kernels-2, 3, 4, and 5, followed by relu activation @xcite and max-pooling. we concatenate the output after every max-pooling layer. we optimize the model parameters using adam (kingma and ba, 2014) with a 1e-3 learning rate. we use 1 devadv model and 2 norm models as a default. baseline implementation. soft-voting is implemented by averaging the model prediction scores. hard-voting is implemented by selecting the majority predictions. the sampling rate of bagging is 70% of training data with replacement, ensuring all the data being used at least once. boosting cannot be compared as a baseline because our method consist of a single model architecture. in order to show the difference with @xcite , we report the performance with the embedding perturbation. ## result the performance of our proposed ensemble methods is presented in we present the performance of the ensemble models on each dataset in figure 1 . with the help of devadv, norm 1 and norm 2 models perform much better than single model on most of the datasets. that is, the scores of the devadv model force the other classifiers to be improved in order to counteract this noise. this principle is also the core idea behind boosting. on the imdb dataset, devadv model does not augment the performance of the norm models. since imdb has only 2 classes, the training process of the devadv model is not different from a conventional training process. ## related work although our boosting method is inspired by the psychological background, devil's advocate, its implementation is related to data augmentation (in particular, negative sampling @xcite ), and adversarial training in terms of training devadv and fine-tuning, respectively. data augmentation is used in many machine learning tasks to artificially enlarge the size of the training set. in the text domain, using synonyms @xcite , back translation @xcite , and paraphrasing @xcite have been proposed. however, these methods are only moderately effective since the meaning of words are sensitive to modification. instead, we use a model trained through negative sampling. our method can then be compared to adversarial training, which uses a negative model to make other models more robust towards adversarial examples. however, as far as we know, @xcite is the only work using an adversarial training framework for text classification. they used an adversarial training process at embedding-level, from the beginning of model training. in contrast, our proposed method utilizes a pretrained negative model to fine-tune other models. furthermore, our negative model contributes to the final prediction, resulting in further improvements (see ## ablation studies training and inference the false labels generated artificially serve to augment the data used for training the devadv model, which is trained using exclusively false labels. by limiting the number of false labels to 1, we confirm the effect of data augmentation. next, we remove the group discussion stage, which fine-tunes the norm models interactively. by this ablation, we can see the effect of adversarial training, which trains a model in an unconventional way by using a negative model. the group discussion process (adversarial training) shows positive effects on performance except for yelp. however, the performance gap is within the error range. we also see that our method can be used with ## conclusion in this paper, we propose a novel boosting ensemble approach, inspired by the devil's advocate. in addition to the implementation of the psychological background, the framework is designed to make submodels better collaborate with each other. we first train a model with incorrect labels in order to make the model serves as devil's advocate (devadv), and the devadv interacts with the other conventionally trained models. in the experiments, we show devadv model improves performance of the other conventionally trained models. although the proposed models' performance does not significantly outperform other ensemble methods, we believe that our new ensemble approach makes valuable contributions to the future research: the use of negative model by taking advantages of data augmentation and adversarial training to provide different views of the same dataset, and the implementation of psychologicalmotivated idea can be properly applied to the nlp field/machine learning domain.