Emotional expressions form a key part of user behavior on today's digital platforms. While multimodal emotion recognition techniques are gaining research attention, there is a lack of deeper understanding on how visual and non-visual features can be used in better recognizing emotions for certain contexts, but not others. This study analyzes the interplay between the effects of multimodal emotion features derived from facial expressions, tone and text in conjunction with two key contextual factors: 1 ) the gender of the speaker, and 2 ) the duration of the emotional episode. Using a large dataset of more than 2500 manually annotated videos from YouTube ,  we found that while multimodal features consistently outperformed bimodal and unimodal features, their performances varied significantly for different emotions, gender and duration contexts. Multimodal features were found to perform particularly better for male than female speakers in recognizing most emotions except for fear . Furthermore, multimodal features performed particularly better for shorter than for longer videos in recognizing neutral , happiness, and surprise, but not sadness , anger, disgust and fear . These findings offer new insights towards the development of more context-aware emotion recognition and empathetic systems.
Emotional expressions form a key part of user behavior on today's digital platforms. While multimodal emotion recognition techniques are gaining research attention, there is a lack of deeper understanding on how visual and non-visual features can be used to better recognize emotions in certain contexts, but not others. This study analyzes the interplay between the effects of multimodal emotion features derived from facial expressions, tone and text in conjunction with two key contextual factors: i ) the gender of the speaker, and ii ) the duration of the emotional episode. Using a large public dataset of 2 , 176 manually annotated YouTube videos, we found that while multimodal features consistently outperformed bimodal and unimodal features, their performance varied significantly across different emotions, gender and duration contexts. Multimodal features performed particularly better for male speakers in recognizing most emotions . Furthermore, multimodal features performed particularly better for shorter than for longer videos in recognizing neutral and happiness, but not sadness and anger . These findings offer new insights towards the development of more context-aware emotion recognition and empathetic systems.
R
in better recognizing emotions for
to better recognize emotions in
R
1
i
R
2
ii
R
dataset of more than 2500 manually annotated videos from YouTube
public dataset of 2
A
176 manually annotated YouTube videos,
R
performances varied significantly for
performance varied significantly across
R
were found to perform
performed
D
than female
D
except for fear
R
, happiness, and surprise,
and happiness,
R
, anger, disgust and fear
and anger