JARVIS - VLA : Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mouse (2025)

Abstract: Recently, action-based decision-making in open-world environments has gained significant attention. Visual Language Action (VLA) models, pretrained on large-scale web datasets, have shown promise in decision-making tasks. However, previous work has primarily focused on action post-training, often neglecting enhancements to the foundation model itself. In response, we introduce Act from Visual Language Post-Training (ActVLP), a novel training paradigm. ActVLP distinctively enhances the foundation model prior to action-specific tuning by first post-training it on a curated set of environment-specific visual and linguistic tasks using self-supervised learning. This initial stage significantly improves the model’s capabilities in world knowledge, visual recognition, and spatial grounding. Subsequently, this strengthened VLM undergoes action post-training via imitation learning on trajectory datasets.Following this paradigm, we develop JARVIS-VLA, the first VLA model in Minecraft that can follow human instructions on over 1k different atomic tasks, including crafting, smelting, cooking, mining, and killing. Our experiments demonstrate that our ActVLP paradigm leads to a significant 40% improvement over the best agent baseline on a diverse set of atomic tasks. Furthermore, JARVIS-VLA surpasses traditional imitation learning-based policies in Minecraft, achieving state-of-the-art performance. We have open-sourced the code, models, and datasets to foster further research.The project page can be found at https://craftjarvis.github.io/JarvisVLA .

## introduction pretraining foundation models on large-scale, noisy internet datasets has become a mainstream approach in nlp and vision (dosovitskiy, 2020; * equal contribution † corresponding author. email: yitaol@pku.edu.cn @xcite @xcite . the success of models like gpt and llama @xcite has shown that large, capable language models can infer and execute tasks described by language prompts. however, this paradigm has yet to achieve similar success in the decision-making domain @xcite . in particular, while openai's video pre-training (vpt) model @xcite has attempted to apply a similar approach in minecraft, it still relies heavily on imitation learning (il) after collecting large-scale youtube videos of human play. vpt's approach of pretraining with imitation learning, followed by downstream supervised fine-tuning and reinforcement learning, made significant strides-culminating in the successful obtaindiamond, a key challenge in minecraft foot_0 . despite this success, the reliance on next-action prediction in imitation learning limits the development of robust, multi-task decision-making abilities @xcite @xcite . moreover, this pretraining paradigm struggles to generalize to unseen environments or tasks due to the intricacies of the interactions between observations and behavior, whereas language tokens are more standardized. to overcome these challenges, a new approach has emerged that leverages pretrained vision language models (vlms) for decisionmaking. these models, known as vision language action models (vlas), integrate language understanding with action generation and can be further enhanced through post-training on visual-language tasks @xcite . a more detailed discussion can be found in figure 1 (left) and section 2.2. however, much like traditional imitation learning, current vla approaches predominantly focus on action post-training. in these models, the learning objective is to generate correct actions based on large-scale cross-task imitation data. we propose that, in addition to action generation, understanding the environment and incorporating taskrelated knowledge could be equally important for achieving more flexible and generalizable decisionmaking. to this end, we introduce a novel training paradigm-vision language post-training (actvlp)-which integrates visual-language tasks into the post-training phase of vla models. following the above paradigms, we obtain the first vla models in minecraft that can follow human instructions on over 1k different atomic tasks, including crafting, smelting, cooking, mining, and killing. our contributions are as follows: (1) we pioneer the use of vla in the open-world environment of minecraft by introducing jarvis-vla, a powerful model achieving state-of-the-art performance in action-based decision-making. @xcite we introduce the concept of vision language post-training and identify key visual-language guidance strategies that enhance decision-making. @xcite we investigate the scaling laws of vla models, demonstrating that expanding the scale of non-trajectory visionlanguage tasks during post-training leads to significant improvements in downstream task performance. @xcite we open-source the code, models, and datasets to support further research in this area. ## learning to act from vision language post-training in this section, we present a detailed introduction to actvlp, a new paradigm for training vla models. one of the most significant improvements is that we investigate a post-training stage prior to imitation learning. specifically, we instantiate this paradigm in our proposed model, jarvis-vla. we begin by discussing the architecture for jarvis-vla in section 2.1, followed by an explanation of the training pipeline in section 2.2 and the datasets used in section 2.3. ## experiments our experiments (starting from subsection 4.2) aim to address the following questions: q1: how do jarvis-vla compare to sota openworld agents and imitation learning methods? q2: is vision language post-training the true cause of the performance improvement? q3: whether vlas exhibit scaling laws and how actvlp influences them? q4: is actvlp sensitive to different vlm backbones? due to space constraints, we quickly respond with an affirmative no, detailed experiment discussion deferred to appendix f. ## conclusions we present actvlp, a novel training framework for visual-language-action models that leverages vision-language post-training to enhance decisionmaking capabilities in dynamic environments. our experiments demonstrate that post-training on nontrajectory tasks significantly enhances foundation models' ability to understand complex environments, resulting in substantial improvements in downstream imitation learning on trajectory data. the effectiveness of this model is validated across multiple vlm architectures, providing strong evidence of its broad applicability and potential for visual-language-action model training, as exemplified by our state-of-the-art model, jarvis-vla.