Abstract-The automatic assessment of affect is mostly based on feature-level approaches, such as distances between facial points or prosodic and spectral information when it comes to audiovisual analysis. However, it is known and intuitive that behavioural events such as smiles, head shakes or laughter and sighs also bear highly relevant information regarding a subject's affective display. Accordingly, we propose a novel string-based prediction approach to fuse such events and to predict human affect in a continuous dimensional space. Extensive analysis and evaluation has been conducted using the newly released SEMAINE database of human-to-agent communication. For a thorough understanding of the obtained results, we provide additional benchmarks by more conventional feature-level modelling, and compare these and the stringbased approach to fusion of signal-based features and stringbased events. Our experimental results show that the proposed string-based approach is the best performing approach for automatic prediction of Valence and Expectation dimensions, and improves prediction performance for the other dimensions when combined with at least acoustic signal-based features.
Eyben et al. REF proposed a string-based prediction model and multimodel fusion of verbal and nonverbal behavioral events for the automatic prediction of human affect in a dimensional space.
String-based audiovisual fusion of behavioural events for the assessment of dimensional affect
Face and Gesture 2011
Face and Gesture 2011
Computer Science