I MAGE
E MOTION R ECOGNITION USING C OGNITIVE C ONTEXTUAL S UMMARIZATION F RAMEWORK
Anonymous authors
Paper under double-blind review
A BSTRACT
Estimating the perceived emotion to visual stimuli has gained significant traction in the recent years.
The existing frameworks rely either on a person’s presence in the image or are based on object feature extraction and low-level image features.
By focusing on the person/object in the image, the existing frameworks fail to capture the context or the interaction between multiple elements in the image.
Also, what if an image does not have a human subject or an object?
We address this drawback by building a Cognitive Contextual Summarization (CCS) model based on an One-For-All (OFA) backbone trained on multiple tasks, including image captioning.
The ability of the backbone to recognize elements in the image and generate captions helps us capture interactions through captions, which we decode using BERT for contextual understanding.
The end-to-end fusion of the OFA and the BERT features helps us predict continuous human emotion (Valence, Arousal) from an image.
We train our framework on the Building Emotional Machines dataset in the literature, and the experiments show that our model outperforms the
State-of-the-art.
I NTRODUCTION
With the penetration of modern devices, humans interact more with the computers than with one another.
These interactions are largely influenced by emotions, which are evoked as the first response to any content being viewed (Joshi et al., 2011; Lang, 1979; Lang et al., 1998).
The paradigm of affective computing has evolved for proactive estimation of emotion from a visual stimuli.
In the literature, emotion models/datasets are of two types viz.
categorical and dimensional.
Categorical emotion models consider few basic emotion categories such as happy , fear , sad , etc.
as given in Ekman (1992) and Mikels et al.
(2005b) emotion models.
The dimensional emotion models represent emotions in the 2D space of Valence-Arousal (VA), where valence measures the amount of pleasantness and unpleasantness and arousal is the intensity of the emotion from being very passive to very active.
Human emotions are complex and not discrete, and hence, it is realistic to measure emotions in a continuous dimensional space (Posner et al., 2005).
Assigning a continuous score to the emotion evoked or elicited in the observer by an image is a very complex problem.
In order to estimate emotion one must take into account all elements in the image which in turn depends on how well the different elements (foreground, background, objects, etc.) in the image are recognized.
As opposed to this, emotion recognition of natural language is comparatively easier (Acheampong et al., 2020).
In literature, image emotion estimation has been addressed by Kim et al.
They propose a dataset wherein each image has a corresponding valence and arousal value. They also propose a framework to estimate image emotions by considering the object and background features in the image.
Although the work is very promising the framework for estimating image emotion depends only on the presence and detection of objects in the image which limits its application.
In order to rectify this, we propose a novel emotion prediction framework which takes an entire image as a input without the need for additional guidance/annotation and predicts the valence/arousal of the whole image.
In our experiments we use the Building Emotional Machines (BE) dataset (Kim et al., 2018) in the literature.
We leverage the recent advancements in cross-modal deep learning by exploiting both the image feature space and the language feature space (Wang et al., 2022).
Our framework is inspired by the fact that contextual interpretation/perception of the image information during cognition leads to the elicited emotion (Greenaway et al., 2018).
Therefore, the proposed framework utilizes the image captioning network called One-For-All (OFA)
- a Task-Agnostic and Modality-Agnostic framework (Wang et al., 2022) trained on multiple tasks, and the Bidirectional Encoder Representations from Transformers (BERT) model for context interpretation (Devlin et al., 2018) to predict the emotions. Image caption networks (ICN) are superior to object detectors because image captions not only capture the objects but also the relationship between the objects present in it. Therefore, resulting in a better contextual understanding.
To summarise, in this paper, we have described the following contributions 1) Building a continuous emotion estimation framework for images in-the-wild i.e. with no requirement of additional information (eg.
Person detection, Object detection, etc.)
We propose a model composed of
OFA (Wang et al., 2022) and BERT.
OFA summarizes the visual information in the image, which is passed to BERT for contextual understanding (Devlin et al., 2018) to predict the evoked emotion.
The Image Captioning OFA backbone not only captures objects in an image (visual cognition) but also captures the relationship between these objects (contextual interpretation/perception of the image).
Hence, this model has been named Cognitive Contextual Summarization Model.
We train and test our framework on BE dataset and report a new benchmark for the image emotion estimation task.
In Figure 1, images in a row have the same valence, which indicates that images in the same row evoke similar degree of pleasantness or unpleasantness in the mind of the viewer.
Within a row the arousal of the images gradually increases from the left to right.
What this means is the left most images are indicative of extreme passiveness while the right most images are indicative of extreme activity or intensity.
In between there is a gradual transition.
As one moves from the top row to the bottom row the valence increases, indicating that the pleasantness evoked in the viewer’s mind by the images increases.
In Figure 2, images in a row have the same arousal, which indicates that images in the same row evoke similar extents of passiveness or activity in the viewer’s mind.
However, within a row the valence of the images gradually increases from left to right.
What this means is the left most images are extremely unpleasant while the right most images are very pleasant to look at.
As one moves from the top row to the bottom row the arousal increases, indicating that the activity or the intensity evoked by the image rises.
R ELATED W ORK 2. B UILDING E MOTIONAL M ACHINES (BE) D ATASET
Several efforts in the past to collect emotion estimation datasets have been made (Kosti et al., 2017;
Mikels et al., 2005a;
Lang et al., 1997; 2005; Kim et al., 2018).
Table 1 describes all the existing datasets used for determining image emotion on a continuous scale i.e. Valence and Arousal.
As observed, BE is both large in scale and is a social dataset i.e. the images in the BE dataset are collated from social media.
For annotation, large number of annotators annotate each image with a
Valence and Arousal score on a scale of 1-9.
The score for each image is a mean of scores across annotators.
Figures 1 and 2 show sample images from the BE dataset with corresponding Valence and Arousal ground truth.
As observed from the images not all images have human subjects and objects and hence there is a need for an image emotion estimation framework which could cater to all kinds of images.
It is worth noting the variability of images in the same scale of Valence or
Arousal values.
Given the variation of images in a specific range, the complexity of image emotion prediction becomes evident.
Given the nature (image variations) and scale (no. of image and annotators) of BE dataset, we test and train our framework on the BE dataset.
[Table]
V:2.0,A:2. V:4.0,A:2. V:6.0,A:2. V:8.0,A:2.
V:2.0,A:2.
V:4.0,A:2.
V:6.0,A:2.
V:8.0,A:2.
V:2.0,A:3.
V:4.0,A:4.
V:6.0,A:3.
V:8.0,A:4.
V:2.0,A:5.
V:4.0,A:6.
V:6.0,A:6.
V:8.0,A:6.
V:2.0,A:8.
V:4.0,A:7.
V:6.0,A:7. V:8.0,A:7. 2. E MOTION E STIMATION F RAMEWORKS Human emotions studied through physiological signals (Subramanian et al., 2018) are invasive and fail to achieve the scale and diversity of modern requirements.
Therefore, the studies have moved towards crowd sourcing and non-invasive ground truth collection to understand the mapping between human emotions and visual stimuli.
Emotions have been estimated in the past using color statistics, texture and also some mid level features (Kim et al., 2018) like Scale Invariant Feature Transform (SIFT) (Lowe, 2004), bag of words, etc.
These models failed to capture the complexity of the emotions from the images and hence, deep learning methods gained significant traction.
Most of the classical affect recognition methods focus on facial expressions (Kollias & Zafeiriou, 2018; Khaireddin & Chen, 2021) and sometimes leverage additional cues, such as body pose (Schindler et al., 2008).
All of these aforementioned works focus only on images that contain people, which is a small subset of all possible images, and thus fail to account for diversity.
Therefore, in this paper we introduce a framework which works for any type of image and does not mandate the presence of any type of content in the image for emotion recognition.
C OGNITIVE C ONTEXTUAL S UMMARIZATION F RAMEWORK
The Cognitive Contextual Summarization (CCS) framework is a fusion network with two branches which regresses over an input image to output a valence or arousal score.
Both the branches are built upon a common One-For-All (OFA) encoder (Wang et al., 2022).
Image embeddings in the first V:1.83,A:2. V:2.0,A:4. V:1.8,A:6.
V:2.4,A:8.
V:2.67,A:2.
V:2.6,A:4.
V:2.0,A:6.
V:3.6,A:8.
V:4.33,A:2.
V:4.6,A:4.
V:6.2,A:6.
V:6.2,A:8.
V:6.0,A:2.
V:6.2,A:4.
V:6.2,A:6.
V:6.83,A:8.
V:7.83,A:2.
V:8.6,A:4.
V:7.2,A:6.
V:8.4,A:8.
branch are obtained from the OFA encoder, while in the other branch the OFA decoder, followed by the BERT, are stacked to get the image caption embeddings of the input image.
The weighted fusion of both the embeddings viz.
OFA encoder embedding and the BERT (image caption) embedding is then regressed to obtain the valence/arousal scores for the input image.
To better understand the framework, we first briefly review OFA and the significance of BERT in the emotional recognition context and then introduce CCS in detail.
3. OFA
OFA is a sequence-to-sequence framework which is task-agnostic and modality-agnostic.
OFA can be used for vision-only, language-only, and vision and language tasks.
It can achieve this by projecting data of various modalities to a unified space, i.e. by discretizing the text, image and object and representing them with tokens in a unified vocabulary.
OFA has an encoder-decoder architecture where both encoders and decoders are a stack of transformer layers.
A transformer layer in the encoder consists of self-attention and a feed-forward network (FFN).
On the other hand, a transformer layer in the decoder comprises a self-attention, a FFN and cross-attention for building the connection between the decoder and the encoder output representations.
OFA is pre-trained for cross-modal tasks, including visual grounding, grounded captioning, image-text matching, image captioning, and visual question answering.
The task-specific pre-training equips OFA with the ability to localize different components in the image for the corresponding natural language input and vice versa.
This training paradigm helps OFA capture local feature signatures across an image, and the finetuning on tasks like image captioning helps the model learn the relationship between different visual components, i.e. summarizing the context.
Hence, the output from the OFA encoder forms one branch in our CCS framework.
3. BERT - CAPTURING LANGUAGE CONTEXT
As discussed above, the OFA pre-training incorporates vision-language pairs.
Given an input image for a VQA task, OFA’s encoder and decoder help map the vision features to the text features and generate captions.
These captions are generated with the help of a unified vocabulary.
Since the generated captions come from the same vocabulary used for images, the OFA decoder’s feature representation is an extension of the OFA encoder embeddings.
Hence, to capture the language context, we do not directly extract OFA decoder features but instead stack BERT after the OFA decoder, which takes the caption as input and gives feature embeddings for the input caption.
Given the huge corpus of language-only data that the BERT is trained on, we expect it to capture language context better.
The same is validated through our experiments in the ablation study section.
3. CCS A RCHITECTURE
Figure 3 shows the architecture of the Cognitive Contextual Summarization model.
At first an image is sent as input to the OFA network. The caption generated by the OFA network is further passed into BERT and the embedding obtained from BERT which is of size 768x1 is the output of the first branch as shown in 3.
At the caption generation step, the modality changes from image to text and hence, there could be some loss of information. Therefore, to capture image features the encoding obtained from the last layer of the last transformer in the OFA encoder which is of size 968x1024 is extracted. This is the output of the second branch as shown in 3.
We weight the fusion of the outputs from both the branches to assign weightage to the contribution made by each branch.
The weighted fusion results into a 1D vector which is then fed to Fully-Connected (FC) layers.
After each FC layer we stack BatchNorm layers.
To avoid overfitting we also introduce two dropout layers each after the 2nd FC layer and the 4th FC layer.
The choice of the number of FC layers has been justified in an ablation study in the supplementary material.
E XPERIMENTS & A BLATION S TUDY
To validate our CCS architecture’s ability to learn the contextual understanding of the image and hence, predict emotions has been verified by extensive experimentation carried out on the BE dataset.
At first, we evaluate the BE model by training and testing on BE dataset.
Figure 4 shows the scatter plots of ground truth and predicted valence values across training (4(a), 4(b) & 4(c)) and validation (4(d), 4(e) & 4(f)) epochs.
It is clearly seen from the scatter plots and the correlation coefficients between the ground truth and the predicted values that the BE model moderately learns valence representations across images.
In other words, it is able to predict the valence of the image fairly or in other words the goodness of the linear fit (learning ability) increases across epochs while training and the model performs comparatively less during validation. [Figure]
[Figure] Similarly, Figure 5 shows the scatter plots of ground truth and predicted arousal values across training (5(a), 5(b) & 5(c)) and validation (5(d), 5(e) & 5(f)) epochs.
It is clearly seen from the scatter plots and the correlation coefficients between the ground truth and predicted values that the BE model learns the Arousal representation very minimally and the validation scatter plots depict poor performance.
We then train our CCS model and test it on BE dataset.
Figure 6 shows the scatter plots of ground truth and predicted valence values across training (6(a), 6(b) & 6(c)) and validation (6(d), 6(e) & 6(f)) epochs.
It is clearly seen that the CCS model learns the valence representation across images.
In other words, it is able to predict the valence of the image very well and the goodness of the linear fit (learning ability) increases across epochs during both training and validation.
[Figure]
Similarly, Figure 7 shows the scatter plots of ground truth and predicted arousal values across training (7(a), 7(b) & 7(c)) and validation (7(d), 7(e) & 7(f)) epochs.
It is clearly seen that the CCS model learns the Arousal representation across images much better than the BE model shown in Figure 5.
[Figure]
We measured the constant mean prediction MSEs for Valence and Arousal to be 2.59 and 1.91 respectively for the BE dataset.
From the Table 2, the performance of the BE model over the BE dataset is better by 36% and 23% than the constant mean prediction for Valence and Arousal respectively, indicating that the model is learning.
However, the learning of the BE model is not sufficient to capture the complexity in the emotion recognition domain.
This has been proven from the Table which clearly depicts that the CCS model which captures the contextual understanding outperforms the BE model for both Valence and Arousal estimation.
[Table]
[Figure]
The training and validation loss plots and correlation coefficient plots in the Figures 8 and 9 further validate the relevance of the CCS model in the emotion recognition domain.
The convergence across epochs is much better for the CCS model than the BE model.
The huge gap in the training and validation plots for BE shows that the model attains saturation beyond which point rendering it incapable of learning the valence and arousal representations across images.
As described in Section 3.3, Cognitive Contextual Summarization (CCS) model comprises of two branches viz.
the OFA encoder module and the OFA decoder and BERT module.
To understand the contribution of each module with respect to the Valence and Arousal prediction on the BE dataset, we perform an ablation study.
In these experiments, we present regression results of valence and arousal on OFA only module of the CCS model and the Fusion module.
Table 3 contains MSE performance for valence and arousal prediction of OFA encoder compared with the CCS model.
The performance of OFA encoder drops indicating that just the encoder is not enough to capture image emotion features and also validates contribution of BERT and the fusion towards the final results. [Figure]
[Table]
C ONCLUSION
Estimating the emotions of an image is the first step towards building models for emoting through visual stimuli.
In this work, we propose a novel image emotion estimation framework that, given any image, outputs the valence and arousal of the image.
Our framework does away with the need for any additional annotation, unlike those used by the previous works in the literature.
Untying emotion estimation from the presence of objects or humans in the image has large scale and widespread applications.
It opens endless possibilities for estimating emotions from any visual content
[Table]