A Graph Autoencoder Approach for Gesture Classification with Gesture AMR (2025)

Abstract: We present a novel graph autoencoder (GAE) architecture for classifying gestures using Gesture Abstract Meaning Representation (GAMR), a structured semantic annotation framework for gestures in collaborative tasks. We leverage the inherent graphical structure of GAMR by employing Graph Neural Networks (GNNs), specifically an Edge-aware Graph Attention Network (EdgeGAT), to learn embeddings of gesture semantic representations. Using the EGGNOG dataset, which captures diverse physical gesture forms expressing similar semantics, we evaluate our GAE on a multi-label classification task for gestural actions. Results indicate that our approach significantly outperforms naive baselines and is competitive with specialized Transformer-based models like AMRBART, despite using considerably fewer parameters and no pretraining. This work highlights the effectiveness of structured graphical representations in modeling multimodal semantics, offering a scalable and efficient approach to gesture interpretation in situated human-agent collaborative scenarios.

## introduction in-person situated communication involves not just language, but non-verbal behavior like actions and, importantly, gestures. however, automated gesture interpretation is complicated by how the same gestural semantics may be represented by very different physical forms. fig. 1 shows an instance of this: two people use entirely distinct iconic gesture shapes to denote the same concept-block. this points to the need for higher levels of abstraction to adequately model the relationship between physical form and gestural meaning, particularly in collaborative dialogue. abstract meaning representation (amr; @xcite ) is a popular choice in the computational semantics community for its clarity and expressiveness, and brutti figure 1 : example from the eggnog dataset @xcite showing different gesture shapes expressing the same gesture semantics. both are iconic gestures @xcite denoting blocks, articulated differently: the physical label of the left is rh: into closed, left; that of the right is arms: move, up; hands: into facing, into open. in this paper, we observe that amr/gamr's natural graphical structure lends itself to graph neural network (gnn)-based approaches for automated processing, and propose a graph autoencoder (gae) that learns mappings between gesture semantics represented in gamr annotation and the physical forms of the associated gestures. experiments on eggnog @xcite , a challenging audio-visual dataset, show that our approach both outperforms naive baselines, and beats or competes with strong transformers on gesture shape prediction, despite having significantly fewer parameters and faster inference time, making our method suitable for gesture classification in lowresource and edge environments. ## related work early work on gesture semantics followed traditions viewing gesture as simulated action @xcite or a general mode of reference @xcite @xcite . following mcneill's work, @xcite posited a division of gestures into deictic and iconic, creating a typing system continued in gamr @xcite . @xcite , @xcite , and @xcite further developed the grammar, semantics, and pragmatics of gesture on which gamr is based. related coding schemes for multimodal or non-verbal behavior include @xcite ; @xcite ; @xcite ; @xcite . abstract meaning representation (amr; @xcite ) is well-known for abstracting away from specific syntax using rooted, directed acyclic graphs (dags) and for applications to diverse tasks such as translation and nlu. graphbased learning approaches using amr include amr-to-sequence learning @xcite ) and text generation @xcite @xcite . our primary experimental dataset eggnog @xcite , containing natural gestures elicited during a collaborative task. eggnog has been used to train gesture recognizers for multimodal interactive agents such as @xcite @xcite and @xcite . @xcite annotated a subset of eggnog with gesture and speech amr, as well as coreference relations within and across the two modalities. ## methodology the eggnog dataset @xcite contains 360 videos of pairs of participants engaged in a collaborative task. one person, the actor, is given a set of wooden blocks, while the other, the signaler, is shown an image of a block structure. the signaler uses gesture, sometimes together with speech, to instruct the actor how to build the structure. gestures are labeled according to both a physical description (e.g., rh: thumbs, up) and the signaler's intent (e.g., yes); this work focuses on the former. each eggnog physical gesture label refers to one or more body parts, which include the head, arms, hands, and upper body. each body part is then described with one or more aspects, including various types of motions (of body parts in space, such as rotate and shake), relations (of body parts to each other, such as crossed and facing), and poses (hand positions, such as claw and point). finally, aspects have optional orientations: up, down, left, right, front, or back. see fig. 1 for an example. for simplicity, we focus on the aspects within each label. @xcite annotated 21 of the eggnog videos with gesture amr. because this was done separately from the physical gesture labels, a single gamr can overlap with multiple labels. we link each gamr with each overlapping label, and, in turn, with each aspect occurring in those labels, making this a multi-label classification problem. in total, the dataset contains 319 gamrs (167 unique), associated with 889 aspects (33 unique). we split the data into an 80:20 train/test split. ## results and discussion while amrbart achieves the best f1 score overall, our gae embeddings achieve competitive performance despite using orders of magnitude fewer parameters ( fig. 3 shows the f1 scores for the 10 mostfrequently occurring physical gesture classes according to the binary-union strategy, across all finally, since the gae has substantially fewer parameters than the competitor methods, we performed an inference-time experiment to quantify the speed advantage. fig. 4 shows the cumulative time required to process increasing numbers of gamrs by each method. we see that the gae boasts a nearly 50% improvement in processing time over amrbart despite amrbart's extremely modest classification advantage, and that the gae remains about 20% faster than roberta at all input sizes despite outperforming it nearly globally. ## conclusion we presented a novel approach to gesture classification using gesture amr and graph autoencoders. our approach achieves competitive classification accuracy with sota transformer approaches at significantly less computational overhead with faster inference speed. we also explored the effects of different label aggregation strategies, based on the premise that in real world data, the same semantics may have different physical forms attached to them. our results can inform the choice of classification technique for downstream tasks that use gesture information with different requirements, such as epistemic position classification as in @xcite . our efficient gae method is suitable for real-time (e.g., vanderhoeven et al. ( 2025 )) or gpu-less systems.