Can Vision Language Models Understand Mimed Actions? (2025)

Abstract: Non-verbal communication (NVC) is an integral part of human language, but it has been overlooked in natural language processing research. Studying NVC in general is challenging because of its high variance in interpretation among individuals and cultures, but mime—the theatrical technique of suggesting intent using only gesture, expression, and movement—is a subset of NVC with much lower human interpretation variance. As a gateway for evaluating vision-language models on their understanding of NVC, we propose Mime Identification-based Multimodal Evaluation (MIME), a gesture recognition task built upon a novel corpus of mimed activity comprising 86 unique gestures with a variety of perturbations applied to the avatar, background, and viewpoint for evaluating recognition robustness. We find that both open-weight and API-based vision-language models perform significantly worse than humans at identifying mimed gestures in MIME, motivating the need for increased research for instilling more robust understanding of human actions for VLMs.

## introduction nonverbal communication (nvc) -the use of nonverbal cues such as gestures, facial expressions, and body language to convey messages -is an instrumental part of human language @xcite @xcite . nvc not only serves as a crucial substitute to communication when verbal modes are limited @xcite @xcite @xcite , but also makes interaction engaging and natural @xcite @xcite , and may even betray true intent that contradicts what is verbally expressed @xcite shown with a few frames from a video of a 3d male character miming a basketball shot in a living room. humans achieve almost perfect accuracy on identifying mimed actions regardless of evaluation format, adversarial perturbations, and the absence of salient context (e.g., basketball, court, basketball outfit), while vlms struggle without salient context. @xcite . therefore, ai systems need to establish a thorough understanding of nvc for them to become more accessible and effective assistants to humans @xcite . unfortunately, this is an overwhelming undertaking considering the broad scope of nvc @xcite , variability in how individuals interpret and exhibit nonverbal cues @xcite , and the limited capabilities of current visionlanguage models (vlms) @xcite @xcite @xcite @xcite . despite impressive achievements of vlms on action recognition benchmarks @xcite @xcite , we find that they cannot even reliably identify a subset of nvc (2) then, a 3d character is retargeted to our motion capture data in blender, a computer graphics software. (3) next, we render frames of the animation with a transparent background. (4) with frames rendered with transparent backgrounds, we can easily overlay them over images of our choice. to this end, we address the following research questions: (i) can vlms reliably recognize mimed actions? and (ii) if not, can we improve a vlm's performance on identifying mimed actions? for the first research question, we construct mime identification multimodal evaluation (mime), 2 a novel video-based question answering benchmark comprising of 86 mimed actions. we create mime using motion capture data and computer graphics software, which enables us to create variations of each action with perturbations applied to the character, background, and viewpoint for evaluating recognition robustness (see figure 1 for a sample of mime and corresponding human and vlm predictions). on mime, humans achieve almost 100% accuracy, regardless of adversarial perturbations and evaluation format. however, vlms, openweight models and api-based black-box models alike, only achieve at most 52.3% accuracy in a multiple choice format where contextual informa-1 a neurological disorder that disrupts the ability to plan and execute purposeful movements, despite having the physical ability to do so. to answer the second research question, we conduct a preliminary exploration into whether existing methods can bridge this shortcoming. specifically, we experiment with chain of thought @xcite , few-shot in-context learning, and finetuning with a subset of mime. we find that the only method that consistently improves model performance over zero-shot is few-shot in-context learning for api-based black-box models, but their results remain significantly worse than human performance. a manual inspection into the descriptions of the mimed actions generated by using chain of thought with gemini 1.5 flash reveal that the majority of failure cases is due to incorrect observations of the demonstrated gestures (80%) and a smaller portion is from incorrectly interpreting correctly generated descriptions (15%). in conclusion, our findings with mime motivate research that instills a more robust understanding of human gestures in vlms for establishing an essential foundation for nvc comprehension. ## mime in this section, we describe the data collection pipeline for mime. an overview is shown in figure 2 . mime is a video-based question answering benchmark that comprises of animations of 86 mimed actions, each with ten variants that are shown in figure 3 , resulting in a total of 860 evaluation samples. the videos are rendered with 3d graphics software by combining digital assets with motion capture data of actors miming various actions. this setup is advantageous to alternative methods foot_0 for conducting a systematic study of recognition robustness with regards to various components that comprise an action as each action can be post-processed and remixed with different backgrounds, characters, and camera angles. ## improving on mime given the poor performance of vlms on mime, we are interested in whether simple methods can surface vlms' potential to understand mimed actions. in this section, we discuss our attempts to improve their performance via such methods. ## conclusion we introduce mime, a novel video-based question answering benchmark that consist of animations of 86 mimed actions created with motion capture data and 3d graphics software. mime contains systematic perturbations in character, background, and viewpoint to assess the robustness of vlms' understanding of full-body gestures. while humans demonstrate almost perfect accuracy and remain highly robust to all modifications in mime, both open-weight and api-based vlms struggle, particularly in the free-form format where recognition accuracy approaches zero under adversarial perturbations. these results highlight the need for further research in enhancing vlms with a robust understanding of human gestures to establish a crucial bedrock for nvc comprehension.