MotionGlot: A Multi-Embodied Motion Generation Model

Sudarshan Harithas and Srinath Sridhar.

Paper · Project · Code

Recovered research checkpoint at step 29,200, including its exact shared tokenizer, human and robot VQ-VAE models, and matching human normalization. The GPT weights are identical to the local online-demo checkpoint. Optimizer state and training data are not included.

Contents

• gpt/: GPT-2 causal model and unified 51,694-token tokenizer.
• human_vq.pth: 512-code human tokenizer and decoder for 263-dimensional features.
• robot_vq.pth: 128-code quadruped tokenizer and decoder for three-dimensional trajectories.
• normalization.npz: human mean and standard deviation associated with the VQ checkpoint.
• manifest.json: file hashes, model dimensions and recovery metadata.

Use

Install the linked MotionGlot source repository, then download the bundle:

from huggingface_hub import snapshot_download
path = snapshot_download("sudarshanharithas/MotionGlot", revision="v0.1.0")

Run a human-motion example:

motionglot generate --task text-to-human --model BUNDLE/gpt \
  --human-vq BUNDLE/human_vq.pth --normalization BUNDLE/normalization.npz \
  --text 'A person walks forward.' --output outputs/human --device cpu

Replace BUNDLE with the downloaded directory. The model emits embodiment-specific motion codes; use the corresponding VQ decoder to recover trajectories. The source package supports six tasks: human and robot generation, human and robot captioning, goal reaching, and question-conditioned human generation.

Evaluation and limitations

The released artifacts passed strict VQ state loading and single-example inference checks for all six tasks. Full retraining and independent reproduction of the paper's benchmark results were not performed. This is a recovered research checkpoint; its original training split differs from the release code's source-isolated splits. Robot outputs are trajectories for a separate controller, not commands sent to hardware by this package.

The code and weights follow Apache-2.0, with attribution to T2M-GPT for the human VQ-VAE. See the code repository's third-party notices. Dataset and body-model files are not included.

@article{harithas2024motionglot,
 title={MotionGlot: A Multi-Embodied Motion Generation Model},
 author={Harithas, Sudarshan and Sridhar, Srinath},
 journal={arXiv preprint arXiv:2410.16623},
 year={2024}
}

Online browser demo

Try MotionGlot online: the three human-motion tasks run in your browser, with mjswan visualization and physics disabled. No installation or login is required. Explore examples immediately; live generation downloads approximately 293 MB on first use.

Browser ONNX exports are in browser/ at the immutable browser-v0.1.0 tag. GPT weights use float16 storage with float32 computation; VQ exports use float32. See browser documentation for conversion, checksums and numerical validation. The original PyTorch bundle remains pinned at v0.1.0.