Are Gestures Worth a Thousand Words? An Analysis of Interviews in the Political Domain (2021)

Abstract: Speaker gestures are semantically co-expressive with speech and serve different pragmatic functions to accompany oral modality. Therefore, gestures are an inseparable part of the language system: they may add clarity to discourse, can be employed to facilitate lexical retrieval and retain a turn in conversations, assist in verbalizing semantic content and facilitate speakers in coming up with the words they intend to say. This aspect is particularly relevant in political discourse, where speakers try to apply communication strategies that are both clear and persuasive using verbal and non-verbal cues. In this paper we investigate the co-speech gestures of several Italian politicians during face-to-face interviews using a multimodal linguistic approach. We first enrich an existing corpus with a novel annotation layer capturing the function of hand movements. Then, we perform an analysis of the corpus, focusing in particular on the relationship between hand movements and other information layers such as the political party or non-lexical and semi-lexical tags. We observe that the recorded differences pertain more to single politicians than to the party they belong to, and that hand movements tend to occur frequently with semi-lexical phenomena, supporting the lexical retrieval hypothesis.

## introduction a bodily gesture is a visible action of any body part, when it is used as an utterance, or as part of an utterance @xcite . if such actions are produced while speaking, we can talk about cospeech gestures. their occurrence, simultaneous or concomitant to speech, has led to different views regarding their role in communication @xcite . some authors @xcite ) have considered gestures as an integrative, inseparable part of the language system. indeed gestures may provide important information or significance to the accompanying speech and add clarity to the children's narrative @xcite ; they can be employed to facilitate lexical retrieval and retain a turn in conversations stam2008gesture and assist in verbalizing semantic content @xcite . from this point of view, gestures facilitate speakers in coming up with the words they intend to say by sustaining the activation of a target word's semantic feature, long enough for the process of word production to take place @xcite . gestures can also convey semantic meanings. for example, @xcite discuss the principles of meaning creation and the simultaneous and linear structures of gesture forms. in this framework, they propose individual aspects of a "grammar" of gestures and conclude that in gestures we can find the seeds of language or the embodied potential of hand-movements for developing linguistic structures. as pointed out by @xcite the link between speech and gesture can be explained by two gesture-speech characteristics: semantic coherence, i.e. combining gesture with meaningful and related speech, and temporal synchrony, i.e. producing gesture in synchrony with speech @xcite . the role of synchronization is particularly relevant for the creation of multimodal resources @xcite , because it allows researchers to overcome one of the historical limits of traditional corpora that are in one modality (either written or spoken): presenting data in a single format offers limited opportunities for exploring non-verbal, gestural features of discourse, while they are important aspects to understand intercultural face-to-face interaction @xcite . nevertheless, @xcite have shown that most of the time gestures are produced before the linguistic item to which they are related, defining this phenomenon "temporal asynchrony". also @xcite presented some empirical evidence to prove that temporal asynchrony between gestures and speech was more common in spontaneous speech and that hand gestures were associated with low-frequency unpredictable lexical items, i.e., lexical items that were more difficult for speakers to reach in the course of language production @xcite . their conclusion was the following: "gestures are products of lexical preplanning processes, and seem to indicate that the speaker knows in advance the semantic specification of the words he will utter, and in some cases has to delay if he has to search for a relatively unavailable item" @xcite . research on spoken interaction has suggested that non-verbal communication is currently the least understood and analyzed aspect of communication, despite recognizing its equal importance @xcite . for this reason, we believe it is very important to carry out studies on gesture-talk interaction and develop multimodal corpora. our study focuses in particular on the relationships between the co-occurrence of speech and gesture in italian in the specific case of political interviews since: i) television interviews are inherently multimodal and multisemiotic texts, in which meaning is created through the co-presence of visual elements, verbal language, gestures, and other semiotic cues @xcite ; ii) linguistic studies in the political domain can be of interest also beyond the nlp community, for example in political science and communication studies, and iii) the italian political scene has been little studied from this perspective. in particular, in the following sections we address research questions such as: our examination of the co-occurrence of speech and gesture will shed light into how the two communication models interact. we also release the corpus of political interviews with the new annotation layer encoding the functions of hand movements at https://github.com/dhfbk/ inmezzoradataset . ## political and multimodal corpora in italian in recent years, political language has received increasing attention, especially in english, since it is possible to have free access to speech transcriptions from uk and us government portals and personal foundation websites such as the white house portal, william j. clinton foundation, margaret thatcher foundation. this has fostered research on political and media communication and persuasion strategies @xcite . as for italian, which is the language of interest for this study, only few corpora in the political domain are available. one of the first experiments was the corpusb @xcite composed by 111 speeches by the former prime minister silvio berlusconifor a total of 325,000 tokens -created to study the evolution of berlusconi's political language from the moment when he started his political career in 1994 until the last programmatic speech of his third government in 2006. subsequently, the work of salvati and pettorino (2010) analyzed diachronically some of the suprasegmental aspects of berlusconi's speeches from 1994 to 2010, including an analysis of the length of logical chains, the number of syllables per chain, the maximum and minimum pitch and frequency of speech, average duration of empty pauses, fluency and tonal range. among the most recent corpora made available in italian, the largest one includes around 3,000 public documents by alcide de gasperi @xcite that has been mainly used to study the evolution of political language over time @xcite . all the corpora cited above are monomodal and none of them takes into account gestural traits. indeed, corpora that include only one modality have a long tradition in the history of linguistics. according to @xcite ) "the construction and use of multimodal corpora is still in its relative infancy. despite this, work using multimodal corpora has already proven invaluable for answering a variety of linguistic research questions that are otherwise difficult to consider". this is also confirmed by the fact that -to date -exist 286 multimodal resources certified for all languages by the lre map foot_0 but only one is in italian, i.e. imagact a corpus-based ontology of action concepts, derived from english and italian spontaneous speech @xcite . so both from the political and the multimodal point of view, this language is not well represented. in an attempt to fill this gap, we first developed the polimodal corpus @xcite , containing the transcripts of 56 tv faceto-face interviews of 14 hours, taken from the italian political talk show "in mezz'ora in più" broadcast between 2017 and 2018, for a total of 100,870 tokens. the annotation has been done using xml as markup language and following the tei standard for speech transcripts in terms of utterances to keep track of so-called "speech constants" @xcite . in particular, the corpus contains the annotation of the following hesitation phenomena: the corpus includes also the annotation of facial displays, hand gesture and body posture, which we carried out using the mumin coding scheme annotation @xcite and anvil @xcite a tool for the automatic annotation of audiovisual material containing multimodal dialogue. this corpus is considered as a starting point for our study. ## annotation of hand movements starting from polimodal corpus described in section 2, we manually add a new level of annotation that takes into account the semantic functions covered by one of the gestures already tagged in the corpus: hand movements. this is because the gestural movements of the hands and arms, i.e. spontaneous communicative movements that accompany speech @xcite , are probably the most studied co-speech gestures @xcite . based on the seminal works by @xcite about the relationship between body motion and speech on the one hand, and about gesticulation and speech in the process of utterance on the other, they are usually separated into several gestural phases: initial/rest position, preparation, (pre-stroke) hold, stroke, (post-stroke) hold, retraction/recovery and rest position @xcite @xcite . note that all gestures are not necessarily constituted by all these phases and that some phases may also be duplicated. in polimodal the hand movement trajectory tag indicates only the start and end of the movement in terms of time and the trajectory of the gesture, in particular up, down, sideways, complex. in order to keep track also of the semantic function covered by the tag, we manually added an additional information layer to those already present -following the classification proposed by @xcite adapting @xcite and @xcite 2 -which attributes five functions to hand movements: our annotation follows the selection criterion highlighted by @xcite , claiming that annotators are expected to select gestures to be annotated only if they have a communicative function. following this principle, each annotator looked at the portion of the video in which the hand movements were occurring and depending on the meaning that he/she thought the gesture had in that particular context of utterance, attributed the corresponding semantic function. however, as @xcite points out gestures can be functionally ambiguous and thus have multiple semantic functions simultaneously. according to @xcite , the source of these multiple functions often lies in the sequential environment of the conversation in which the utterance occurs. to simplify the task, annotators are therefore asked to assign a single semantic function to the gestures under investigation, choosing the function that they consider prevalent in the context of use. when the gesture-speech relationship appears too vague, it is good practice to conduct interviews with speakers to confirm the interpretation of gesture meanings. in fact, as suggested by @xcite , through multiple methods of data analysis, such as triangulation, we can test whether interpretations of the results were consistent and internally coherent. in our case, since such checking is not possible, we try to ensure a high-quality and consistent annotation by computing inter-annotator agreement. specifically we perform a double annotation of the semantic functions listed above on three of the interviews considered (matteo renzi, luigi di maio, matteo salvini) for a total of about 2 hours of interviews. both annotators (one male and one female) are expert linguists. macro-averaged f1 computed on exact matches amounts to 0.83, which corresponds to an almost perfect agreement. this result confirms that the task is well-defined and that the corresponding annotation guidelines are clear. figure 1 shows an example annotation with the new information layer specified with the tag 'semantic function'. for each observed gesture, the polimodal corpus already contained: i) the start and end point in the video in terms of milliseconds; ii) the type of gesture observed; iii) the movement trajectory. we add to this the semantic function covered by the gesture in the context. ## description of gesture-speech annotation a summary of the hand gesture annotations in the corpus is reported in over the years, studies have shown that the production of gestures is influenced by the syntax of the language itself and by the socio-cultural context of the language. as explained in a 2015 study by @xcite -focused on co-speech gesture production in children's narratives -language syntax influences gesture production. for example -as known -some languages require an explicit subject (i.e. english, french, etc.), whereas others (i.e. italian, spanish, etc.) are null-subject languages. this characteristic requires distinct marking of referential continuity in the textual use of language, with less need to repeat anaphora in the latter case @xcite . another key factor influencing the communication is culture as a set of values and norms that helps shape the social behavior of individuals who belong to a cultural group as well as social interaction between them. very well known is the study in @xcite , showing that italians use a great number of gestures when communicating. so -since some socio-cultural factors seem to influence the production of gestures -we wonder whether political party affiliation is among them. then as a next step, we investigate whether there is a significant relationship between the political party of affiliation of interviewees in the dataset and the type of gestures used, and thus whether the political party of affiliation affects the choice of gesture categories. the political parties considered are the ones reported in a qualitative analysis of the single interviews shows interesting differences in attitude and communication style, which pertain to single politicians rather than to party positions. matteo renzi, for example, uses gestures very frequently to accompany his speech. we report an example of 'integration' below: matteo renzi:"quello che sta accadendo invece in queste settimane, in questi mesi, conferma che c'è una grande distanza tra la politica dei palazzi e la politica della quotidianità @xcite ." (eng. "instead what is happening in these weeks, in these months, confirms that there is a great distance between the politics of the palaces and the politics of everyday life.") renzi underlines that the distance between politics made by elites, detached from the real problems of the country ("politics of the palaces"), and "politics of everyday life", that is, attentive to reality and to citizens, is increasingly evident. a gesture is used to stress this difference: the speaker's open right hand points away from his torso in correspondence with the metaphorical expression "politics of the palaces", almost as if to indicate that it is something in which he does not recognize himself. his right hand then immediately rejoins his left hand and points downwards at the moment in which the expression "politics of everyday life" is pronounced, as if to indicate a politics that is instead attentive to relevant and concrete things. concerning the reinforcing type of gesturespeech relationship, it is mainly used to reiterate a concept already expressed linguistically, and it is not very used, probably because it may seem redundant. angelino alfano turns out to be the interviewee who makes most use of this type of gesture. in this example, alfano, talking about the consensus obtained by one of his political opponent matteo salvini, claims that this consensus was obtained at his expense. so, in saying "contro di me" (against me), the open hands are close to his bust. angelino alfano:"quindi la sfida di salvini, avendo aggregato consenso -contro di me peraltro [reinforcing] -sull'immigrazione, è incanalarlo su un regime di legislazione democratica." as mentioned above, supplementary gestures are used with a very low frequency. one of the few examples in the corpus is present in simone di stefano's interview, where he is asked to clarify the alleged relations of the party with a convicted member of the mafia. the interviewee tries to provide an explanation, but the interviewer continues to put him under pressure. at this point the interviewee lowers his gaze and moves his open right hand away from his torso while saying "but i don't want to avoid [your question]", as if to implicitly ask the journalist to stop her suppositions and let him explain his position. complementary gestures bring a necessary complement to the incomplete linguistic information provided by the verbal message. they are frequently used by the respondents in the corpus under analysis, in most cases to disambiguate the message or simply some linguistic elements. this indicates the speaker's intention to be as clear as possible. for example, at the beginning of the interview with carlo calenda, he is shown a photo that portrays him wearing a worker's helmet. the interviewee refers to the photo by pointing with his left hand away from his torso to the screen where the photo is displayed, making it easier for viewers to understand what he was referring to: carlo calenda:"benché gli operai non si sentiranno, come posso dire, contenti dopo aver visto la mia foto con quel caschetto @xcite in cui sembravo un totale ebete." (eng. "although the workers won't feel, how can i say, happy after seeing the picture of me in that helmet where i looked like a total idiot.") as noted above, a residual category has been added to the tags. the other category includes all the gestures that annotators were not able to classify with the above mentioned semantic labels. this problem was found most frequently in the interviews with pier carlo padoan and carlo calenda. these gestures are different from the others because they show a batonic value, that is, they are used to mark the rhythm of the enunciation, for example by tapping a finger on the table. ## lexical retrieval hypothesis many studies have suggested that gestures, especially representational gestures (krauss and hadar, 1999) play a direct role in speech production by priming the lexical retrieval of words. this view has been termed the lexical retrieval hypothesis. the hypothesis is based on research arguing that (1) gesturing occurs during hesitation pauses or in pauses before words indicating problems with lexical retrieval @xcite , and (2) that the inability to gesture can cause verbal disfluencies @xcite . in addition -as (krauss, 1998) pointed out -speakers were more dysfluent overall in constrained-speech conditions than in natural conditions. since the corpus used as the object of study presents a level of annotation that takes into account some hesitation pauses and verbal disfluencies, we decided to verify this hypothesis in the political domain, where speakers usually have to control well their communication and be persuasive. we compute weighted mutual information @xcite between hand movements and each of the speech disfluencies reported in among the politicians included in this dataset, the one that most accompanies his speech with hand gestures is matteo salvini (lega) considering both interviews, followed by carlo calenda (pd) and angelino alfano (il popolo della libertà). their belonging to different political parties suggests that the use of hand movements is more an individual trait than a feature characterising specific political positions. weighted mutual information (wmi) is computed between hand movements and tags reported in figure 2 : wmi values between hand movements and tags reported on the x-axis for each interviewee on the y-axis overall, hand movements tend to have a higher association with semi-lexical traits and pauses, which would confirm the assumptions of lexical retrieval hypothesis according to which gesturing occurs during hesitation pauses or in pauses before words indicating problems with lexical retrieval @xcite . this effect is however not present for some politicians, such as di battista and alfano, while it is evident for some others such as bersani and salvini. therefore, our findings are not generally applicable to all interviewees in our corpus. evident differences in gesturing behaviour among the considered politicians. for instance, although carlo calenda and angelino alfano present a high incidence of hand movements, they do not seem to be associated with specific tags. matteo renzi, instead, shows a gesturing behaviour that is unique compared to all the other interviees, with hand gestures that are almost always used in association with other speech phenomena. in the interviews, we observe also the presence of negative values for wmi obtained in relation to false-starts (-0.11), repetitions (-0.1 and -0.6) and truncations (-0.8), suggesting that hand movements are less likely to be accompanied by such linguistic phenomena. ## conclusions in this work, we investigate co-speech gestures of several italian politicians during face-to-face interviews. to this purpose, we enrich an existing corpus with labels describing the semantic type of the different hand movements. concerning gesturespeech relationship, the results obtained suggest that hand movements are mainly used with an integrative and complementary functions. so, the information provided by such gestures adds precision and emphasis to spoken information. we also show that party affiliation does not significantly influence the gesture-speech relationship. finally we test the lexical retrieval hypothesis by computing the association between hand movements produced by each interviewee and speech disfluen-cies using weighted mutual information. results show that hand movements tend to co-occur with full pauses (i.e. repetition) and empty pauses (i.e. pause) and more frequently with interjections (i.e. semi-lexical), suggesting that gesticulating may represent an attempt at lexical retrieval. in the future we plan to conduct further analyses aimed at understanding whether such gestures cooccur with specific types of words (e.g. copulative verbs, predicative verbs, etc.) and whether other linguistic or socio-linguistic variables such as language complexity or age influence the use of hand movements and their semantic functions.