A Comprehensive Analysis of Preprocessing for Word Representation Learning in Affective Tasks (2020)

Abstract: Affective tasks such as sentiment analysis, emotion classification, and sarcasm detection have been popular in recent years due to an abundance of user-generated data, accurate computational linguistic models, and a broad range of relevant applications in various domains. At the same time, many studies have highlighted the importance of text preprocessing, as an integral step to any natural language processing prediction model and downstream task. While preprocessing in affective systems is well-studied, preprocessing in word vector-based models applied to affective systems, is not. To address this limitation, we conduct a comprehensive analysis of the role of preprocessing techniques in affective analysis based on word vector models. Our analysis is the first of its kind and provides useful insights of the importance of each preprocessing technique when applied at the training phase, commonly ignored in pretrained word vector models, and/or at the downstream task phase.

## introduction affective tasks such as sentiment analysis, emotion classification and sarcasm detection have enjoyed great popularity in recent years. this success can be largely attributed to the fundamental and straightforward nature of the methods employed, the availability of vast amounts of user-generated natural language data, and the wide range of useful applications, spanning from hate speech detection to monitoring the sentiment of financial markets and news recommendation @xcite . most early models of affect analysis employed pretrained word embeddings that have been obtained under the assumption of the distributional hypothesis @xcite . the distributional hypothesis suggests that two words occurring frequently in similar linguistic contexts tend to be more semantically similar, and therefore should be represented closer to one another in the embedding space. however, while such embeddings are useful for several natural language processing (nlp) downstream tasks, they are known to be less suitable for affective tasks in particular @xcite . although some authors claim that there is a need for post-processing word embeddings for affective tasks, others find that off-theshelf vectors are very powerful for affective lexicon learning @xcite . for example, word2vec @xcite estimates the pair of words 'happy' and 'sad' to be more similar than the pair of words 'happy' and 'joy', which is counterintuitive, and might affect the accuracy performance of the models that depend on it. to address the limitations of traditional word embeddings, several techniques have been proposed, including task-specific fine-tuning @xcite , retrofitting @xcite , representing emotion with vectors using a multi-task training framework @xcite and generating affective word embeddings @xcite , to name a few. other attempts to overcome the limitation of word vectors include optimization of hyperparameters @xcite , as well as fine-tuned preprocessing strategies tailored to different nlp tasks. while these strategies have demonstrated evidence of improving the accuracy performance in tasks such as word similarity, word analogy, and others @xcite , their effect in affective tasks has not received considerable attention and remains less explored. our work is motivated by the observation that preprocessing factors such as stemming, stopwords removal and many others make up an integral part of nearly every improved text classification model, and affective systems in particular @xcite . however, little work has been done towards understanding the role of preprocessing techniques applied to word embeddings in different stages of affective systems. to address this limitation, the overarching goal of this research, is to perform an extensive and systematic assessment of the effect of a range of linguistic preprocessing factors pertaining to three affective tasks, including sentiment analysis, emotion classification and sarcasm detection. towards that end, we systematically analyze the effectiveness of applying preprocessing to large training corpora before learning word embeddings, an approach that has largely been overlooked by the community. we investigate the following research questions: (i) what is the effect of integrating preprocessing techniques earlier into word embedding models, instead of later on in a downstream classification models? (ii) which preprocessing techniques yield the most benefit in affective tasks? (iii) does preprocessing of word embeddings provide any improvement over stateof-the-art pretrained word embeddings? and if yes, how much? figure 1 illustrates the difference between a) preprocessing word embeddings pipeline (pre) vs. b) preprocessing classification dataset pipeline (post), where preprocessing techniques in (a) are applied to the training corpus of the model and in (b) only to the classification dataset. in brief, the main contributions of our work are as follows: the rest of the paper is organized as follows: section 2 presents an overview of the related work. section 3 elaborates on the preprocessing techniques employed in the evaluation of models. section 4 describes the experimental evaluation framework. in section 5 a comprehensive analysis of the results is provided. section 6 concludes the paper with key insights of the research. ## related work in this section, we present an overview of related work on preprocessing classification datasets and preprocessing word embeddings, and how our work aims to bridge the gap between those efforts. ## preprocessing in affective systems this section describes the preprocessing factors applied to the training corpus that is then used to generate word representations and the order of the preprocessing factors which we need to follow when applying on the corpus. ## discussion and analysis we analyze the impact of preprocessing techniques in word representation learning on affect analysis. ## conclusions we systematically examined the role of preprocessing training corpora used to induce word representations for affect analysis. while all preprocessing techniques improved performance to a certain ex-21 https://github.com/bfelbo/deepmoji 22 https://www.dropbox.com/s/wr5ovupf7yl282x/ewe uni.txt tent, our analysis suggests that the most noticeable increase is obtained through negation processing (neg). the overall best performance is achieved by applying all the preprocessing techniques, except stopwords removal (all-stop). interestingly, incorporating preprocessing into word representations appears to be far more beneficial than applying it in a downstream task to classification datasets. moreover, while all the three affective tasks (sentiment analysis, sarcasm detection and emotion classification) benefit from our proposed preprocessing framework, our analysis reveals that the multiclass emotion classification task benefits the most. exploring the space of subsets of our preprocessing factors might yield more interesting combinations; we leave this for future work.