arxiv:2502.14834

LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in Vision-Language Models

Published on Feb 20

· Submitted by

tsq2000 on Feb 21

Upvote

Authors:

Shangqing Tu ,

Yushi Bai ,

Zhiyuan Liu ,

Juanzi Li

Abstract

Existing Large Vision-Language Models (LVLMs) can process inputs with context lengths up to 128k visual and text tokens, yet they struggle to generate coherent outputs beyond 1,000 words. We find that the primary limitation is the absence of long output examples during supervised fine-tuning (SFT). To tackle this issue, we introduce LongWriter-V-22k, a SFT dataset comprising 22,158 examples, each with multiple input images, an instruction, and corresponding outputs ranging from 0 to 10,000 words. Moreover, to achieve long outputs that maintain high-fidelity to the input images, we employ Direct Preference Optimization (DPO) to the SFT model. Given the high cost of collecting human feedback for lengthy outputs (e.g., 3,000 words), we propose IterDPO, which breaks long outputs into segments and uses iterative corrections to form preference pairs with the original outputs. Additionally, we develop MMLongBench-Write, a benchmark featuring six tasks to evaluate the long-generation capabilities of VLMs. Our 7B parameter model, trained with LongWriter-V-22k and IterDPO, achieves impressive performance on this benchmark, outperforming larger proprietary models like GPT-4o. Code and data: https://github.com/THU-KEG/LongWriter-V

View arXiv page View PDF Add to collection

Community

tsq2000

Paper author Paper submitter 2 days ago

Existing Large Vision-Language Models (LVLMs) struggle to generate coherent outputs beyond 1,000 words. We train a series of models to generate Ultra-Long and High-Fidelity texts for long-output visual instructions.

⚙️ LongWriter-V Deployment

We open-source three models: LongWriter-V-7B and LongWriter-V-7B-DPO, trained based on Qwen2.5-VL-7B-Instruct and LongWriter-V-72B, trained based on Qwen2.5-VL-72B-Instruct.

🖥️ Model Training

You can download and save the LongWriter-V-22K data through the Hugging Face datasets (🤗 HF Repo).

You can train the model with LLaMA-Factory, we used the official Qwen2_VL training script for training.

📊 Evaluation

We introduce two evaluation benchmarks: MMLongBench-Write and LongWrite-V-Ruler. MMLongBench-Write focuses more on measuring the long output quality as well as the output length, while LongWrite-V-Ruler is designed as a light-weight stress test of the model's maximum output length.