In our natural world, a face is usually encountered not as an isolated object but as an integrated part of a whole body. The face and the body both normally contribute in conveying the emotional state of the individual. Here we show that observers judging a facial expression are strongly influenced by emotional body language. Photographs of fearful and angry faces and bodies were used to create face-body compound images, with either matched or mismatched emotional expressions. When face and body convey conflicting emotional information, judgment of facial expression is hampered and becomes biased toward the emotion expressed by the body. Electrical brain activity was recorded from the scalp while subjects attended to the face and judged its emotional expression. An enhancement of the occipital P1 component as early as 115 ms after presentation onset points to the existence of a rapid neural mechanism sensitive to the degree of agreement between simultaneously presented facial and bodily emotional expressions, even when the latter are unattended. emotion communication ͉ event-related potentials ͉ visual perception T he face and the body both normally contribute in conveying the emotional state of the individual. Darwin (1) was the first to describe in detail the specific facial and bodily expressions associated with emotions in animals and humans and regarded these expressions as part of emotion-specific adaptive actions. From this vantage point, face and body are part of an integrated whole. Indeed, in our natural world, faces are usually encountered not as isolated objects but as an integrated part of a whole body. Rapid detection of inconsistencies between them is beneficial when rapid adaptive action is required from the observer. To date, there has been no systematic investigation into how facial expressions and emotional body language interact in human observers, and the underlying neural mechanisms are unknown. Here, we investigate the inf luence of emotional body language on the perception of facial expression. We collected behavioral data and simultaneously measured electrical eventrelated potentials (ERP) from the scalp to explore the time course of neuronal processing with a resolution in the order of milliseconds. We predicted that recognition of facial expressions is inf luenced by concurrently presented emotional body language, and that affective information from the face and the body start to interact rapidly. In view of the important adaptive function of perceiving emotional states and previous findings about rapid recognition of emotional signals, we deemed it unlikely that integrated perception of face-body images results from relatively late and slow semantic processes. We hypothesize that the integration of affective information from a facial expression and the accompanying emotional body language is a mandatory automatic process occurring early in the processing stream, which does not require selective attention, thorough visual analysis of individual features, or conscious deliberate evaluation. Photographs of fearful and angry faces and bodies were used to create realistic-looking face-body compound images with either matched or mismatched emotional expressions. Electroencephalogram (EEG) signals were recorded while subjects attended to the face and judged the emotional expression. A short stimulus-presentation time was used (200 ms), requiring observers to judge the faces on the basis of a ''first impression'' and to rely on global processing rather than on extensive analysis of separate facial features. The emotions fear and anger were selected because they are both emotions with a negative valence, and each is associated with evolutionarily relevant threat situations. Although subcortical circuitry may be capable of some qualitative differentiation between fear and anger (2), their differentiation in terms of cortical responses is as minimal as possible. Differential ERP responses have not been reported to occur before Ϸ170 ms (3). This will make observed differences between electrophysiological correlates of congruent and incongruent stimuli easier to interpret. To test our automatic-processing hypothesis, we focused on the analysis of early electrophysiological components that can be readily identified in the visual ERP waveform, i.e., the P1 and the N170 component. The N170 is typically found in face perception studies as a prominent negative brain potential peaking between 140 and 230 ms after stimulus onset at lateral occipito-temporal sites. It shows a strong face sensitivity putatively pointing to specialized neuronal processing routines for faces as compared with objects. It is commonly considered as an index of the early stages of encoding of facial features and configurations (e.g., ref. 4). The waveform shows a robust face-selective ''inversion'' effect indicative of configural processing; i.e., it is enhanced and delayed to faces that are presented upside down but not to inverted objects (5, 6). In general, it has been found to be insensitive to aspects of the face other than the overall structure, like expression or familiarity. Of interest, a typical but slightly faster N170 component showing the typical inversion effect commonly obtained for faces was also found for the perception of human bodies (6). Recent studies have challenged the N170 as the earliest marker of selective face processing and drawn attention to an earlier component. The P1 component is a positive deflection recognizable at occipital electrodes with an onset latency between 65 and 80 ms and peaking around 100-130 ms. There is evidence that this component is mainly generated in ''early'' extrastriate visual areas (e.g., refs. 7-10), and it is commonly thought that it reflects processing of the low-level features of a stimulus. Its amplitude is modulated by spatial attention, but selected attention to nonspatial features does not affect the component (e.g., ref. 11). However, a few recent studies suggest that when it comes to faces, at least some form of higher-order processing can already occur at this early stage. Using magnetoencephalography, a face-selective component has already been identified around 100-120 ms after stimulus onset that was
In a study by Hanneke et al. REF , images of illustrated characters showing emotions of fear and anger were manipulated to congruent and incongruent stimuli by exchanging the faces in the images.
Rapid perceptual integration of facial expression and emotional body language
Proceedings of the National Academy of Sciences of the United States of America
Proceedings of the National Academy of Sciences of the United States of America
Psychology Medicine