How Much Do Large Language Models Know about Human Motion? A Case Study in 3 D Avatar Control (2025)

Abstract: We explore the human motion knowledge of Large Language Models (LLMs) through 3D avatar control. Given a motion instruction, we prompt LLMs to first generate a high-level movement plan with consecutive steps (**High-level Planning**), then specify body part positions in each step (**Low-level Planning**), which we linearly interpolate into avatar animations. Using 20 representative motion instructions that cover fundamental movements and balance body part usage, we conduct comprehensive evaluations, including human and automatic scoring of both high-level movement plans and generated animations, as well as automatic comparison with oracle positions in low-level planning. Our findings show that LLMs are strong at interpreting high-level body movements but struggle with precise body part positioning. While decomposing motion queries into atomic components improves planning, LLMs face challenges in multi-step movements involving high-degree-of-freedom body parts. Furthermore, LLMs provide reasonable approximations for general spatial descriptions, but fall short in handling precise spatial specifications. Notably, LLMs demonstrate promise in conceptualizing creative motions and distinguishing culturally specific motion patterns.

## introduction recent approaches in text-conditioned human motion generation attempt to improve the generalization to unseen instructions by leveraging large language models (llms) to extract motion-relevant information, such as active body parts (athanasiou et al., 2023), detailed body part descriptions (huang et al., 2024), and keyframe coordinates (huang et al., 2023). however, these methods only utilize llms as auxiliary components, leaving the extent of their human motion knowledge largely unexplored. figure 1: our motion knowledge grounding pipeline converts the instruction into a high-level movement plan, followed by body part position predictions, which are mapped to animation codes and rendered in unity. in this paper, we explore llms' knowledge of human motion through their capabilities to drive a 3d human avatar. following the natural hierarchy from action sequences to body part movements (flash and hochner, 2005), our approach (figure 1 ) consists of two stages: (1) high-level planning, where an llm generates step-by-step body part movements in natural language from the input motion instruction, and (2) low-level planning, where the llm selects a position for every body part from a predefined set of poses within each step (e.g., neutral, bent_in_90_degrees for left elbow). these predictions are then converted into animation codes for the skinned multi-person linear (smpl; loper et al., 2015) 3d human model in unity, foot_1 using predefined rules that map body part positions to smpl joint rotations (e.g., bending left elbow to 90 degrees maps to m_avg_l_elbow rotation from (0, 0, 0) to (0, 90, 0)). the rendered animations from linearly interpolated llmselected poses provide a clear verification lens for human evaluators. we carefully design 20 representative motion instructions with full coverage of basic movement primitives and balanced body part usage, and evaluate both commercial (e.g., claude 3.5 sonnet) and open-source (e.g., llama-3.1-70b) llms through three complementary approaches: (1) human and gpt-4.1-based evaluation of high-level planning feasibility and key movement completeness, assessing the conceptual understanding of motion; (2) quantitative comparison of low-level body part positioning against oracle annotations for scalability and reproducibility; and (3) human and gemini 2.5 pro-based judgement of animation quality for both complete motion and individual body parts, capturing multiple valid motion variations with holistic feedback on naturalness. firstly, we find that llms demonstrate high competence in generating high-level plans with physically proper key movements. however, they struggle with precise body part positioning, especially for multi-step motions involving high-degreeof-freedom body parts (e.g., upper arm). these positioning errors often accumulate across multiple body parts, resulting in low-quality animations. secondly, breaking down motion queries into atomic components enhances performance compared to single-round generation. for high-level planning, iterative querying of individual motion aspects (e.g., body movements, states) proves more effective. for low-level planning, hierarchical decomposition and position-by-position selection consistently yield superior results. finally, while llms provide reasonable approximations for general spatial descriptions (e.g., the bending motion for wiping a one-meter high table), they fail to handle precise spatial specifications in text (e.g., pick up the object by foot), and fall short in generating accurate spatial and temporal parameters for avatar control. however, llms show promise in conceptualizing creative motions (e.g., strut like a peacock showing off feathers) and distinguishing culturally-specific motion patterns (e.g., differentiating normal kneel to bow and kneel to perform a japanese bow), suggesting their potential to provide enhanced semantic understanding when combined with high-quality low-level motion generators from traditional supervised approaches. ## related work contemporary generative models show remarkable progress in synthesizing realistic human body movements from natural language instructions (guo et al., 2022; tevet et al., 2023; zhang et al., 2024a; guo et al., 2024). however, these models often fail on novel motion instructions out of the limited training datasets, such as compositional instructions, rare activities, or nuanced movement descriptions. to address this generalization problem, recent work uses llms to extract specific motion-relevant information, indicating that llms might contain rich human motion knowledge. athanasiou et al. (2023) use llms to identify relevant body parts for action labels like "stroll", showing llms' understanding of the anatomical requirements for different movements. however, they only focus on simple action-to-body-part mapping without exploring complex motion reasoning capabilities. later research further prompts llms to decompose abstract motion descriptions into sequential, step-by-step movement specifications (li and feng, 2024). more advanced approaches leverage llms for hierarchical motion planning and control. for instance, como (huang et al., 2024) and fg-t2m++ (wang et al., 2025) employ llms to parse ambiguous instructions into structured descriptions targeting specific body parts, enabling fine-grained control over motion generation. more comprehensively, fan et al. (2024) propose a framework that converts instructions into atomic motion plans organized by predefined body segments, such as spine, left upper limb, etc. instead of using llms for motion-related text generation, recent work also shows that llms can directly generate keyframe coordinates to be interpolated as motions (zhang et al., 2024b; huang et al., 2023). the aforementioned approaches focus on leveraging llms as auxiliary tools to optimize text-tomotion systems. however, they overlook the fundamental question of what motion knowledge llms actually possess and how accurately they understand human movement principles. we address this research gap by grounding llm responses into 3d avatar animations, and probing their motion knowledge across multiple levels of abstraction. ## methodology this section presents our comprehensive methodology for motion knowledge grounding and evaluation. we introduce a hierarchical pipeline that converts natural language instructions into unity animations ( §3.1). within the pipeline, we develop systematic querying strategies that guide llms through high-level motion decomposition and lowlevel body part specification ( §3.2). finally, we design a novel evaluation framework to assess llm capabilities at multiple levels of the pipeline ( §3.3). ## results and analysis using the designed motion instructions, we run the motion knowledge grounding pipeline on both commercial and open-source llms including claude 3.5 sonnet, gpt-4o, gpt-4o-mini, gpt-3.5-turbo and llama-3.1-70b. @xcite nine evaluators with artificial intelligence research backgrounds participate in the human evaluation, where we calculate the inter-annotator agreement. for hps and wbs using likert scales, we calculate the pairwise weighted kappa (fleiss and cohen, 1973). for category-based bpq, we apply average pairwise agreement, calculated as the mean percentage of matching categories between evaluator pairs. this section presents our evaluation results and analysis across three key components of the motion knowledge grounding process: high-level planning that decomposes motion descriptions into sequential steps ( §4.1), low-level planning that predicts precise body part positions ( §4.2), and complete animation generation that synthesizes animations from the instructions ( §4.3). ## discussion to reveal both the capabilities and limitations of llms in each stage of motion knowledge grounding in more details, we analyze the performance patterns in high-level planning, low-level planning, and complete animation generation through quantitative error analysis and representative case studies. ## conclusion through our hierarchical framework for 3d avatar control, we evaluate llms' human motion understanding on carefully designed representative instructions. our findings reveal that llms possess substantial motion-related semantic knowledge despite limitations in precise spatial understanding, particularly for multi-step motions involving highdegree-of-freedom body parts. breaking down movements into atomic components improves both high-level conceptual planning and low-level body part coordination, yet llms fall short in handling precise spatial specifications and generating accurate spatial-temporal parameters for avatar control. notably, llms show promise in conceptualizing creative motions and distinguishing culturallyspecific motion patterns. practical value & future directions. our work suggests llms could enhance natural language interfaces for avatar control and culturally-aware, creative motion synthesis. to address spatial precision limitations, we propose a hybrid approach: using llms for high-level planning and motion decomposition, while delegating low-level execution to specialized motion models (tevet et al., 2023; guo et al., 2024) that handle joint dynamics (e.g., velocity, acceleration) and physical constraints (e.g., foot contact, collision). this leverages llms' semantic strengths with motion models' spatial precision, enabling interpretable language-based control with enhanced motion realism. while implementing such a system exceeds our current scope of evaluating llms' human motion knowledge, we hope our findings provide insights for future development.