Pretrained Image-Text Models are Secretly Video Captioners (2025)

Abstract: Developing video captioning models is computationally expensive. The dynamic nature of video also complicates the design of multimodal models that can effectively caption these sequences. However, we find that by using minimal computational resources and without complex modifications to address video dynamics, an image-based model can be repurposed to outperform several specialised video captioning systems. Our adapted model demonstrates top-tier performance on major benchmarks, ranking 2nd on MSR-VTT and MSVD, and 3rd on VATEX. We transform it into a competitive video captioner by post-training a typical image captioning model BLIP-2 with only 6,000 video-text pairs and simply concatenating frames—significantly fewer data than other methods, which use 2.5 to 144 million pairs. From a resource optimization perspective, this video captioning study focuses on three fundamental factors: optimizing model scale, maximizing data efficiency, and incorporating reinforcement learning. This extensive study demonstrates that a lightweight, image-based adaptation strategy can rival state-of-the-art video captioning systems, offering a practical solution for low-resource scenarios.

## introduction vision-language pretraining significantly advances multimodal tasks such as captioning, question answering, retrieval and broader video understanding @xcite @xcite @xcite @xcite @xcite @xcite @xcite . among these, video captioning stands out as it narrates visual concepts and their temporal interactions, reflecting the intricate multimodal processes as humans to perceive and articulate dynamic visual experiences. current video-text methods often incorporate intricate designs tailored to video inputs. for instance, some models extend existing frameworks by integrating frame samplers to capture temporal dynamics @xcite @xcite . other approaches, such as al-pro @xcite and violet @xcite , propose end-to-end models that are meticulously trained on large-scale video-text datasets sourced from the web @xcite . despite their success, video captioning models remain highly resource-intensive, often hitting performance bottlenecks when (i) computational resources are constrained, or (ii) the task requires specialized priors without clear guidance for model design and training. this raises a critical question: for simplicity and efficiency, how can we repurpose existing image captioning models for video captioning, without relying on complex, hand-crafted video-specific designs? to address this, we revisit fundamental factors in training-model scale, data efficiency, and supervision-that critically influence video captioning while being agnostic to the variants of videospecific designs: first, we find that moderate-sized language models (lms) when fine-tuned for specific tasks, can meet the demands of video captioning efficiently. this challenges the common belief that larger models are always superior, demonstrating that targeted optimization can outperform sheer model size. second, using extensive pretraining on image-text pairs, as demonstrated with blip-2, is transferable to video tasks. this allows the model to achieve high performance with minimal video usage, offering an efficient alternative to training from scratch. third, instead of relying on traditional cross-entropy loss, we optimize directly for nondifferentiable cider with reinforcement learning, ensuring that the generated captions better align with human-standard video descriptions. by bypassing complex, specialized video input designs, our experiments demonstrate that blip-2 straightforwardly derived from image captioning, can be effectively optimized to deliver competitive video captioning performance. this study underscores the potential of simplicity and efficiency in advancing multimodal video captioning, providing a streamlined yet stable solution. the codes are released: https://github.com/ chunhuizng/mllm-video-captioner . ## recycling blip-for video captioning as shown in fig. 1 , we adapt blip-2, a typical image-text model (details in app. b), for video captioning without any additional parameters. each video frame is encoded by vit, which generates visual tokens that are concatenated to form a unified representation (e.g., an 8-frame video produces a token sequence of size 8×256). this unified token sequence is then processed by the q-former and passed to the lm to generate captions. ## training recipes: model, data, and supervision according to tab. 1, our solution has top-level performance on important benchmarks (particularly on the cider metric-the primary ranking measure on paperswithcode), ranking 2nd on msr-vtt and msvd, and 3rd on vatex, among models with publicly available code. more importantly, it proves to be highly efficient without any video architecture design, using only 6k video-text pairssignificantly less than the million-level datasets required by competing baselines. additional background is in app. a. the settings are detailed in app. c, and further experiments (ablations, other datasets, and other video tasks) supporting the following analysis are in app. d. ## discussion and conclusion this study stands out from existing video captioning research by identifying three factors-model scale, data efficiency, and training supervisionthat are critical for effectively adapting image captioning models to video tasks. by using these insights to reuse the image-based blip-2 model for video tasks, our solution with minimal resource usage ranks 2nd, 2nd, and 3rd on msr-vtt, msvd, and vatex. this open-source guide provides a foundation for future research aimed at optimizing resource allocation in video captioning and refining post-training techniques.