Learning Emotion from 100 Observations: Unexpected Robustness of Deep Learning under Strong Data Limitations (2020)

Abstract: One of the major downsides of Deep Learning is its supposed need for vast amounts of training data. As such, these techniques appear ill-suited for NLP areas where annotated data is limited, such as less-resourced languages or emotion analysis, with its many nuanced and hard-to-acquire annotation formats. We conduct a questionnaire study indicating that indeed the vast majority of researchers in emotion analysis deems neural models inferior to traditional machine learning when training data is limited. In stark contrast to those survey results, we provide empirical evidence for English, Polish, and Portuguese that commonly used neural architectures can be trained on surprisingly few observations, outperforming n-gram based ridge regression on only 100 data points. Our analysis suggests that high-quality, pre-trained word embeddings are a main factor for achieving those results.

## introduction deep learning (dl) has radically changed the rules of the game in nlp by boosting performance figures in almost all application areas. yet in contrast to more conventional techniques, such as n-gram based linear models, neural methodologies seem to rely on vast amounts of training data, as is obvious in areas such as machine translation @xcite or representation learning for individual words @xcite or contextualized word sequences @xcite @xcite . with this profile, dl seems ill-suited for many prediction tasks in sentiment and subjectivity analysis @xcite . for the widely studied problem of polarity prediction (distinguishing only between positive and negative emotion), training data is relatively abundant especially for the social media domain @xcite . however, in recent years, there has been a growing interest in more nuanced and informative annotation formats for affective states @xcite . such annotation schemes often follow distinct psychological theories such as the dimensional approach to emotion representation @xcite or basic emotions @xcite ). yet, annotating for more complex representations of affective states seems to be significantly harder in terms of both time consumption and inter-annotator agreement (iaa) @xcite . adding even more complexity, computational work following this trend often uses numerical scores as target variables making, emotion analysis a regression, rather than a classification problem @xcite . what makes this situation even worse is that, first, we currently have a situation where there is no community-wide consensus on how emotion should be represented. that is, different ways of annotating emotion (see, e.g., ## survey we conducted a short questionnaire study asking the research community about their beliefs regarding data requirements of deep learning in the context of emotion analysis. we included two questions, one asking for the number of training examples "necessary for deep learning to provide results in line with traditional discriminative learning" (question 1), the other asking for the number of examples necessary for "deep learning approaches to provide a clear benefit over traditional discriminative learning" (question 2). the full text of the questionnaire is given in figure 1 . participants were instructed to focus on non-contextualized word representations (in line with our latter experiments; the use of contextualized word embeddings is left for future work) being used as input to the, from their view, most suitable model corpus language size annotation emb. alg. dims. emb. data emb. data size se07 english 1000 be6 [1, 100] word2vec 300 google news 100b tokens wassa @xmath0, 1] word2vec 400 twitter 400m tweets anpst polish 718 vad [1, 9] fasttext 300 wikipedia 4b tokens mas portuguese 192 vad [1, 9] + be5 [1, 5] fasttext 300 wikipedia 4b tokens to recruit participants, we queried the acl anthology for papers from between 2016 and 2018 using the keyword "emotion", we collected all email addresses in the author field of all retrieved pdfs (166 papers in total). invitations to participate in the survey were sent to the resulting 391 email addresses on february 28, 2019. we received 26 responses within four weeks (6% response rate). one response (stating in the optional comment field that no numeric answer could be given) was excluded. figure 2 shows the distribution of the 25 remaining responses on a logarithmic scale. as can be seen, the responses to both questions clearly support the intuition that deep learning is thought of as being dependent on large amounts of training data by the scientific community. in both cases, the median response was 10,000. perhaps surprisingly, a total of 5 participants stated that fewer than 100 instances are necessary for deep learning to show clear improvements over more traditional methods (compared to 20 who believed the opposite), whereas only 2 believed that less than 100 instances are enough to show results "in line with" traditional methods. inspecting the individual responses, we found that this discrepancy stems from a minority of participants (4 of out 25) who indicated that traditional learning performs worse than dl on small datasets but may catch up as dataset sizes grow. another interesting, most likely related outcome is that the responses to question 2 show a bi-modal distribution: while a minority of 5 participants believed that dl approaches are superior below 100 observations, the vast majority of participants (20) states that 1,000 or more instances are necessary for that. yet, no one responded with a number between 100 and 1,000. while we do not validate the claim of this minority, the remainder of the paper provides strong evidence that the majority of the participants largely overestimated the data requirements of deep learning. ## data for the following study, we selected four small (< 3000 instances) and typologically diverse datasets described below. pre-trained, publicly available word2vec @xcite and fasttext vectors @xcite of matching language and target domain were used as model input. se07: the test set of semeval 2007 task 14 @xcite comprises 1000 english news headlines that are annotated according to six basic emotions, joy, anger, sadness, fear, disgust, corpus text val aro dom joy ang sad fea dis sur se07 inter milan set serie a win record ---50 2 0 0 0 9 tbs to pay $2m fine for ad campaign bomb scare ---11 25 28 45 32 43 wassa @thereval please tell us why 'protesting' injustice requires #burning #beating and #looting terrible optics #toussaintromain is true leader! ----.73 ----@taudeltaphidk thank you for my obama cut out!!!!!! i am elated that he's back home ---.83 -----anpst decyzje podjęte w przeszłość i kształtuj ą nasz ą teraźniejszość. 'decisions made in the past shape our present.' 4.9 4.1 5.8 ------dopóki walczysz i podejmujesz starania, jesteś zwycięzc ą. 'as long as you fight and keep trying, you are a winner.' 7.3 5.3 7.4 ------mas a praia é espetacular. 'the beach is spectacular.' 8.0 4.0 6.7 4.2 1.0 1.0 1.0 1.0 -a tinta é azul. 'the ink is blue.' 5.0 3.9 5.5 1.3 1.0 1.0 1.0 1.0 - and surprise on a [0, 100]-scale (be6 annotation format). the news headlines are quite short and rather objective, being written by professional journalists. however, they may still elicit strong emotional reactions in readers as illustrated in anpst: the affective norms for polish short texts (imbir, 2017) is a dataset designed as stimulus in psychological experiments. it is annotated according to valence, arousal, and dominance on a [1, 9]scale (vad). anpst comprises sentences of various genres (such as proverbs, jokes, literature quotes, or newswire material) from a wide range of sources @xcite . the resulting selection of raw data seems often quite complex and ambiguous in terms of the elicited emotion (see examples in mas: like anpst, the minho affective sentences @xcite is a dataset designed by psychologists, also being annotated according to valence, arousal, and dominance on a [1, 9]-scale (vad). yet, additionally, mas is also annotated according to the first five basic emotions (omitting 'surprise') on a [1, 5]-scale (be5). it consists of very short situation descriptions in the third person in european portuguese @xcite . those sentences were purposefully constructed by psychologists to be simple in language and familiar in content for a large proportion of the population. this was done to make the dataset more widely applicable as experimental stimulus, yet also resulted in a slightly ar- model description ridgengram n-gram features with @xmath1, 2, 3}; feature normalization; automatically chosen regularization coefficient from {10 -4 , 10 -3 , .., 10 4 } ridgebv bag of vectors-features; regularization coefficient chosen as in 'ridgengram' ffn bag of vectors-features; two dense layers (256 and 128 units) cnn one conv. layer (filter size 3, 128 channels), max-pooling layer with .5 dropout; dense layer (128 units) gru recurrent layer (128 units, unidirectional); last timestep receives .5 vertical dropout and is fed into a dense layer (128 units) lstm identical to 'gru' cnn-lstm conv. layer as in 'cnn'; max-pooling layer (pool size 2, stride size 1) with .5 dropout; lstm identical to 'gru' ## methods we provide two distinct linear baseline models which both rely on ridge regression, an ℓ 2 -regularized version of linear regression. the first one, ridge ngram , is based on n-gram features where we use @xmath2, 2, 3}. the second one, ridge bv uses bag-of-vectors features, i.e., the pointwise mean of the embeddings of the words in a text. regarding the deep learning approaches, we compare feed-forward networks (ffn), gated recurrent unit networks (gru), long short-term memory networks (lstm), convolutional neural networks (cnn), as well as a combination of the latter two (cnn-lstm) @xcite @xcite . since holding out a dev set from the already limited training data does not seem feasible for some of the datasets (see the input to the dl models is based on pre-trained word vectors. relu activation was used everywhere except in recurrent layers. dropout is used for regularization with a probability of .2 for embedding layers and .5 for dense layers following the recommendations by @xcite . we use .5 dropout also on other types of layers where it would conventionally be considered too high (e.g. on max pooling layers). our models are trained for 200 epochs using the adam optimizer (kingma and ba, 2015) with a fixed learning rate of .001 and a batch size of 32. word embeddings were not updated during training. since, in compliance with our gold data, we treat emotion analysis as a regression problem @xcite , the output layers of our models consist of an affine transformation, i.e., a dense layer without non-linearity. to reduce the risk of overfitting on such small data sets, we used relatively simple models both in terms of the number of layers and units in them (mostly 2 and 128, respectively). an overview of our models and details about their individual hyperparameter settings are provided in ## conclusion annotating emotion is necessarily subjective thus making gold data in this area particularly rare. as such, applying dl may seem ill-advised since supposedly large amounts of training data are required. but is this really the case? we started our investigation by conducting a survey among researchers in emotion analysis. 80% of the respondents believed that dl is superior to traditional machine learning techniques only when at least 1,000 training examples are available. half of the participants even believed that 10,000 or more examples are necessary. putting this popular notion to the test, we provided the first examination of neural emotion analysis under severe data constraints, featuring five distinct neural architectures and three typologically diverse languages. in stark contrast to the survey results, we found that all architectures could be fitted on datasets comprising as little as 200 observations, cnns and ffns even being robust on 100 observations. a subsequent analysis indicated that high-quality, pre-trained word embeddings are a key factor in achieving those results. in the future, we would like to extend this work to contextualized word representations, e.g., by elmo or bert @xcite .