Emotional R ob BERT and Insensitive BERT je: Combining Transformers and Affect Lexica for D utch Emotion Detection (2021)

Abstract: In a first step towards improving Dutch emotion detection, we try to combine the Dutch transformer models BERTje and RobBERT with lexicon-based methods. We propose two architectures: one in which lexicon information is directly injected into the transformer model and a meta-learning approach where predictions from transformers are combined with lexicon features. The models are tested on 1,000 Dutch tweets and 1,000 captions from TV-shows which have been manually annotated with emotion categories and dimensions. We find that RobBERT clearly outperforms BERTje, but that directly adding lexicon information to transformers does not improve performance. In the meta-learning approach, lexicon information does have a positive effect on BERTje, but not on RobBERT. This suggests that more emotional information is already contained within this latter language model.

## introduction computational analysis of affect in dutch texts is mostly restricted to polarity analysis (negative/positive/neutral), for which we know a tradition of lexicon-based approaches. recently, a bert-based model, bertje @xcite , and a roberta-based model, robbert @xcite , have been created for dutch, and they have achieved promising results on the task of sentiment analysis. for emotion detection, however, these models have not yet been evaluated. in a first step towards improving emotion detection for dutch, we will evaluate bertje and robbert on the task of emotion detection. instead of casting aside the many efforts that have been made in the creation of dutch sentiment and emotion lexica, we will investigate whether transformers and affect lexica can complement each other. attempts of combining bert models with additional features have already been successful for tasks like abusive language and sarcasm detection @xcite . we consider two architectures. in the first one, we inject lexicon information in the transformer model before the prediction layer. we do this by concatenating the [cls] token of the target sentence (which bert and roberta models use as input for prediction) with a lexicon vector obtained from seven dutch affect lexica. in the second approach, we employ a meta-learning architecture and use a support vector machine (svm) that learns from the transformer model's output. the predictions from the transformer are concatenated with the lexicon vector and used as input for the svm. we evaluate our models on 1,000 dutch tweets and 1,000 transcribed utterances from flemish tvshows. as multiple researchers have emphasized the need of studying emotions not only in terms of basic emotions, but based on dimensions like valence, arousal and dominance as well @xcite , the data has been annotated in a bi-representational design: both with categorical annotations for anger, joy, fear, love, sadness or neutral, and scores for the dimensions valence, arousal and dominance. first we will discuss related work on dutch emotion detection and other attempts on combining transformer models with additional features in section 2. in section 3, we will describe the methodology of our experiments and in section 4, we report the results. we end with a conclusion in section 5. ## related work although most emotion detection research deals with the english language, recent studies have shown interest in other languages as well, e.g. the recent work of @xcite for hindi or the corpus text example categorical dimensional v a d tweets @transavia jaaah volgende vakantie barcelona en na het zomerseizoen naar de algarve joy 0.689 0.491 0.622 en: @transavia yeah next holiday barcelona and after summer season to the algarve captions ik zou liever sterven dan hier te wonen, denk ik. sadness 0.156 0.384 0.301 en: i'd rather die than live here, i think. semeval-2018 task on affect in tweets, for which in addition to english data, arabic and spanish datasets were released @xcite . moreover, research on multilingual emotion detection @xcite and sentiment analysis @xcite is emerging. concerning the automatic modelling of affect in dutch, the main focus still is on sentiment analysis instead of fine-grained emotions. many studies have used sentiment lexica for this purpose (e.g. @xcite @xcite . recently, transformer models like bertje (de vries et al., 2019) and robbert @xcite have been used to classify reviews from the dutch book reviews dataset as either positive or negative, in which robbert came out as best (accuracy of 95% versus 93%). the only publicly available dataset for dutch emotion detection is delearyous @xcite . it consists of 740 dutch sentences from conversations, annotated according to leary's rose @xcite . @xcite found that classifying sentences from delearyous into the quadrants or octants of leary's rose was difficult for machine learners, most likely because of the sparseness of the data and low interannotator agreement. however, after this study, no further studies on dutch emotion detection were published. this clearly shows the need of new data and methods for this task. some researchers already revealed that combining bert models with handcrafted features can have a positive effect on performance, e.g. for the task of essay scoring (accuracy of 71% versus 80%) @xcite and sarcasm detection (f1score of 78% versus 80%) @xcite . lexicon features have been combined with bert as well, e.g. for abusive language detection (koufakou et al., 2020), for which the authors found an improvement on four out of six datasets compared to a plain bert model (maximum improve-ment of 3%). in all of these studies, the handcrafted features or lexicon features were injected into the transformer architecture by concatenating them with the bert representation before the prediction layer. ## conclusion as the recently developed transformer models for dutch, bertje and robbert, have not yet been tested on emotion detection tasks, we evaluated them on 1,000 dutch tweets and 1,000 captions for an emotion classification and regression task. we found that robbert outperformed bertje in almost all cases. further, we investigated whether these models could be enhanced with lexicon features and proposed two methods: one in which a lexicon vector was concatenated with the transformer's [cls] token before prediction, and a meta-learning approach. in the first method, adding lexicon information did not seem beneficial. for the second approach, however, we found that the meta-learner had a positive effect in half of the cases, and especially on the models relying on bertje, whereas the meta-learner had almost no impact on robbert. this is probably because more emotional information is already contained within this language model.