Towards a Methodology Supporting Semiautomatic Annotation of H ead M ovements in Video-recorded Conversations (2021)

Abstract: We present a method to support the annotation of head movements in video-recorded conversations. Head movement segments from annotated multimodal data are used to train a model to detect head movements in unseen data. The resulting predicted movement sequences are uploaded to the ANVIL tool for post-annotation editing. The automatically identified head movements and the original annotations are compared to assess the overlap between the two. This analysis showed that movement onsets were more easily detected than offsets, and pointed at a number of patterns in the mismatches between original annotations and model predictions that could be dealt with in general terms in post-annotation guidelines.

## introduction communicative gestures, such as head movements, gaze and hand gestures, are important in face-toface communication, where they provide relevant and non redundant information that must be processed together with speech @xcite . the automatic annotation of gestures is still problematic while their manual annotation is challenging and resource intensive. head movements are the most frequent gesture type and their importance as feedback and turn management signals has been recognised in numerous studies @xcite @xcite @xcite . for this reason, the automatic identification of the most frequent types of head movement has often been addressed as a component in multimodal systems where humans communicate with software agents or robots @xcite @xcite . many factors influence the frequency and type of gestures produced by speakers, e.g. the language @xcite and the communicative set-ting. therefore, it is important to have reliably annotated data of conversations in different languages and contexts. there are a wealth of video-recorded monologues and conversation on the internet that could be used for multimodal analysis. however, although several projects have developed multimodal data annotated with gestural information (for a discussion, see @xcite , few of these efforts have yet resulted in freely available annotated corpora for multiple languages such as in @xcite . in parallel with this work, the linguistics community interested in co-speech gesture has created and annotated many multimodal data collections, but typically for experimental purposes and thus in very specific contexts @xcite . in the nlp community there is a growing interest for multimodal sentiment analysis, in which visual and acoustic features are used to analyse sentiment and emotion in video data @xcite . the data collections used for this task are consequently annotated with sentiment and emotion labels, not with gestural information. a recent example is the cmu-mosei corpus @xcite . this paper describes work aimed to support the annotation of head movements in video-recorded spontaneous conversations. more specifically, we propose a method in which automatically identified head movement segments from dyadic conversations are uploaded to the anvil multimodal annotation tool @xcite , from which human annotators will be able to correct them. to assess how demanding it would be for human annotators to work with the system's suggestions, we compare the automatically identified head movements with manually annotated ones from the same corpus, and present both quantitative and qualitative analyses of this comparison. finally we discuss possible improvements of the system and future work. in section 2 we present background and related studies; in section 3 we describe the corpus, particularly the head movement annotations; in section 4 we give a succinct account of the method used to train models that predict head movements; in section 5 we analyse the results in terms of how well they match what the human annotators did; finally, in section 6 we summarise our results and suggest some feature perspectives. ## related studies the automatic identification of head movements can be based on the use of different tracking systems, e.g. looking at the coordinates provided by kinect @xcite or by various eye-tracking systems @xcite . a number of researchers have worked with automatic detection of head movements in video-recordings of human-human and human-robot conversations, and the results of these studies have been evaluated using the manual annotations of the relevant movements. two main approaches have been used. in the first one, computer vision techniques have been applied @xcite , but most of these attempts have posed certain requirements to the quality, light and settings of the videos. the second approach concerns training various classifiers on automatically extracted visual and, possibly, audio features. for the visual features the two freely available systems opencv foot_0 and openpose foot_1 have been used, while audio features have been extracted via different systems. the focus of the majority of these studies has been that of determining the best visual and auditory features and the most effective classifiers for the task @xcite @xcite @xcite @xcite . only a couple of studies have addressed the integration of automatically identified head movements in annotation tools. @xcite integrated the opencv facetracker into anvil to support head movement annotation for users of that system. the max planck institute for psycholinguistics and the fraunhofer institutes hhi and iais developed avatech @xcite , a gesture recogniser which is integrated into the elan multimodal annotation system @xcite . the recogniser identifies head movements and hand gestures in videos. the system is based on skin recognition and other computer vision techniques, and therefore only works when videos follow specific requirements with respect to video quality, background colour, position of the speaker(s) and light. the system works best in cases when only one speaker is recorded and both face and hands can clearly been seen. when successful, avatech generates a video where the gestures are marked, and elan tracks with the head and gesture annotations given certain specifications. the system is not being further-developed, but it is still an integrated part of elan. we tested the recogniser on our data, but it failed to process the videos because of their quality, and in some cases, their format. given this background, additional efforts are needed to provide the research community with more and better automatic support for the creation of gestural annotation of conversational data. ## the corpus the corpus used in our study is the danish nomco corpus of first encounter dialogues @xcite , which consists of twelve spontaneous conversations between young people (six males and six females) of about five minutes each, for a total recording time of approximately one hour. each participant took part in two dialogues with persons of different genders. during the encounters, the participants were standing facing each other and were recorded by three cameras. for this study we used recordings in which two frontal views of the participants are joined as shown in figure 1 . coding scheme for formal and functional annotation of gestural behaviour @xcite . this study is concerned with the formal annotation of the speakers' head movements, for which the coding scheme provides the following labels: nod, up-nod, shake, headbackward, headforward, sideturn, tilt, waggle and headother. a head movement is annotated by defining start and end point of the movement on the tool's annotation board, and by picking one of the labels. the linguistic context was taken into account in order to select and annotate only those head movements that were judged to have a communicative function. to familiarise themselves with the annotation scheme and guidelines, the three annotators involved went through an initial training exercise in which they annotated the gestures in one video together, and discussed all disagreements with one another and with two expert researchers. after that, they annotated a second video independently and their mutual agreement was measured. the inter-coder agreement for identification and classification of head movements reached a cohen's kappa score in the range 0.72-0.8 @xcite . to produce the final version of the annotations, all disagreements were adjudicated by involving an expert annotator. the annotations have been used in numerous studies addressing aspects such as the identification of feedback and turn-taking signals given through head movements and facial expressions @xcite . the manual annotations of head movements, as well as of other gesture types, are available for research foot_2 . ## automatic detection of head movement in this section we summarise the method we use for the automatic detection of head movements. a more detailed explanation and discussion is provided in @xcite . the manual annotations of head movements were used as a basis for training a classifier to detect the presence of head movement in a frame-wise fashion. to create a speaker-independent model, the classifier had to detect movement for each speaker after having being trained on the data from the other eleven participants following a leave-one-out cross validation strategy. the training data consisted of visual, acoustic and textual features. openpose @xcite was used to extract nose tip positions from the data. for each of these positions we include both the cartesian (x and y) and polar (radius and angle) coordinates, thus, four different scalar values. we then calculate velocity, acceleration and jerk of these positions by computing the first, second and third order derivatives, using the previous 9, 11 and 13 frames, respectively. we include these three orders of derivatives, resulting in a total of twelve visual features. intensity and pitch measurements were extracted from the audio files using praat @xcite . finally, head movement labels, as well as word information, were extracted from the annotations. as a result, for each frame in each video a vector was created with labels expressing presence/absence of movement and the head movement class; velocity, acceleration and jerk features; pitch and intensity values referring to the gesturer; and finally a binary feature expressing whether the same gesturer is speaking or not. several classifiers were tested on the data. the best performing one was a multilayer perceptron (mlp) with 4 layers, which achieved an average accuracy across the twelve speakers of 0.730 and an f1 score of 0.684 (macro average). the frame-wise head movement predictions were combined into sequences by conjoining directly adjacent frames for which a positive movement value had been assigned by the predictor and mapped onto the xml format necessary to read them into the anvil tool as independent tracks. in this way, a comparison could be carried out between the manual and the automatic annotations. figure 2 is a screenshot of the anvil annotation board visualising an excerpt of the annotation with tracks for one of the speakers (speaker a). there are eight tracks corresponding to i) the speech transcription, ii-iv) the manual annotations of facial expressions, head movements and body posture, v) the head movement annotations detected by the mlp classifier, and vi-viii) the values for movement velocity, acceleration and jerk. the two tracks of interest here are the ones containing the manually annotated head movements, a sequence of which has been highlighted in red, and the corresponding automatically derived head movement elements, marked in green. in this example, we see that three head movements in the former track roughly correspond to three larger and two smaller elements in the latter. there are also, however, two additional predictions in the middle of the sequence which do not overlap with any annotated movement. in a scenario in which a new annotation task has to be produced, only the elements suggested by the model would be available, making it possible for an annotator to revise these suggestions as needed. to get an impression of how much and what type of revision may become necessary, in the next section we look more closely at how well the two annotation tracks correspond to one other. ## conclusion and future work in this paper we have presented work aimed at creating support for the annotation of head movements in video-recorded data, an annotation which is a necessary first step in any analysis of the role played by head movements in the semantics and pragmatics of face-to-face conversations. the first step in our methodology is to train machine learning models to detect head movements based on visual and acoustic features that are obtained from video-recorded materials without specific demands on the audio-visual quality. the second step is to make the automatically derived movement sequences available in the anvil multimodal annotation tool. the evaluation of the results showed a good coverage in that 86% of annotated movements were predicted with various degrees of overlap. however, a high number of non-existing movements were also incorrectly predicted. subsequent analysis showed that movement onsets were more easily detected than offsets, and pointed at a number of patterns in the mismatches between original annotations and model predictions that could be dealt with in general terms in postannotation guidelines. more work is needed to understand and possibly reduce the impact of false positives. the sequencegeneration process may also be a source of errors, thus alternative methods to combine frames into sequences should be explored. another obvious extension of this work concerns the automatic classification of head movement in different types. moreover, we plan to work on adapting our results to make the predicted movement sequences compatible with the elan tool. finally, we need to determine how useful the automatic prediction is in a concrete annotation exercise. it is important to evaluate not only whether the annotators would save time post-editing the predicted movements rather than creating new ones from scratch, but also whether the same accuracy can be reached and what kinds of error the annotators would make in the post-editing scenario compared to the traditional one. we plan to conduct such a study in connection with the development of a new corpus of online group discussions that is being collected by the international network on gestures and head movements in language (gehm) 4 amir zadeh, minghai chen, soujanya poria, erik cambria, and louis-philippe morency. 2017. tensor fusion network for multimodal sentiment analysis. arxiv preprint arxiv:1707.07250. amir zadeh, paul pu liang, soujanya poria, prateek vij, erik cambria, and louis-philippe morency. 2018. multi-attention recurrent network for human communication comprehension. in thirty-second aaai conference on artificial intelligence.