  Emotion recognition as a key component of high-stake downstream applications
has been shown to be effective, such as classroom engagement or mental health
assessments. These systems are generally trained on small datasets collected in
single laboratory environments, and hence falter when tested on data that has
different noise characteristics. Multiple noise-based data augmentation
approaches have been proposed to counteract this challenge in other speech
domains. But, unlike speech recognition and speaker verification, in emotion
recognition, noise-based data augmentation may change the underlying label of
the original emotional sample. In this work, we generate realistic noisy
samples of a well known emotion dataset (IEMOCAP) using multiple categories of
environmental and synthetic noise. We evaluate how both human and machine
emotion perception changes when noise is introduced. We find that some commonly
used augmentation techniques for emotion recognition significantly change human
perception, which may lead to unreliable evaluation metrics such as evaluating
efficiency of adversarial attack. We also find that the trained
state-of-the-art emotion recognition models fail to classify unseen
noise-augmented samples, even when trained on noise augmented datasets. This
finding demonstrates the brittleness of these systems in real-world conditions.
We propose a set of recommendations for noise-based augmentation of emotion
datasets and for how to deploy these emotion recognition systems "in the wild".
