Improve model card: add prominent links, comprehensive guides, and project details (#2)

- Improve model card: add prominent links, comprehensive guides, and project details (ff264893f327294b99bd0c59896d42002f48150d)


Co-authored-by: Niels Rogge <nielsr@users.noreply.huggingface.co>

README.md

@@ -1,67 +1,68 @@
 ---
-license: apache-2.0
-datasets:
-- lmms-lab/LLaVA-One-Vision-1.5-Mid-Training-85M
-- lmms-lab/LLaVA-OneVision-1.5-Insturct-Data
 base_model:
 - DeepGlint-AI/rice-vit-large-patch14-560
 - Qwen/Qwen3-4B-Instruct-2507
-pipeline_tag: image-text-to-text
+datasets:
+- lmms-lab/LLaVA-One-Vision-1.5-Mid-Training-85M
+- lmms-lab/LLaVA-OneVision-1.5-Insturct-Data
 library_name: transformers
+license: apache-2.0
+pipeline_tag: image-text-to-text
 ---
+
 # LLaVA-OneVision-1.5: Fully Open-Source State-of-the-Art VLM Model
 
-# ✨ Key Features
+This is the official Hugging Face model card for **LLaVA-OneVision-1.5**, a novel family of Large Multimodal Models (LMMs) presented in the paper [LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training](https://huggingface.co/papers/2509.23661).
 
-**LLaVA-OneVision-1.5** introduces a novel family of **fully open-source** Large Multimodal Models (LMMs) that achieves **state-of-the-art performance**  with substantially **lower cost** through training on **native resolution** images.
+📚 [Paper](https://huggingface.co/papers/2509.23661) | 💻 [Code](https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5) | 🏠 [Project Page](https://huggingface.co/collections/lmms-lab/llava-onevision-15-68d385fe73b50bd22de23713) | 🚀 [Demo](https://huggingface.co/spaces/lmms-lab/LLaVA-OneVision-1.5)
 
-1. **Superior Performance**
-A family of fully open-source large multimodal models demonstrating **superior performance** across multiple multimodal benchmarks, **outperforming Qwen2.5-VL** in most evaluation tasks.
+# ✨ Key Features
 
-2. **High-Quality Data at Scale**
-Meticulously curated **mid-training and SFT data** with rigorous filtering and quality control.
-- Concept-balanced, highly diverse, high-quality caption data
-- Comprehensive instruction fine-tuning data covering a wide range of tasks
+**LLaVA-OneVision-1.5** introduces a novel family of **fully open-source** Large Multimodal Models (LMMs) that achieves **state-of-the-art performance** with substantially **lower cost** through training on **native resolution** images.
 
-3. **Ultra-Efficient Training Framework**
-Complete end-to-end training framework designed for maximum efficiency:
-- **$16K total budget** for full model training
-- **45% HFU efficiency** on A100 GPUs ($0.6 per GPU/Hour)
-- Built on **MegatronLM** with support for **MoE**, **FP8**, and **long sequence parallelization**
-- Optimized codebase for cost-effective scaling
+1.  **Superior Performance**
+    A family of fully open-source large multimodal models demonstrating **superior performance** across multiple multimodal benchmarks, **outperforming Qwen2.5-VL** in most evaluation tasks.
 
-4. **Fully Open Framework** for community access and reproducibility:
-- ✅ High-quality mid-training & SFT data
-- ✅ Complete training framework & code
-- ✅ Training recipes & configurations
-- ✅ Base & instruct model checkpoints
-- ✅ Comprehensive training logs & metrics
+2.  **High-Quality Data at Scale**
+    Meticulously curated **mid-training and SFT data** with rigorous filtering and quality control.
+    - Concept-balanced, highly diverse, high-quality caption data
+    - Comprehensive instruction fine-tuning data covering a wide range of tasks
 
-## Code
-This model is trained using a fully open-source, end-to-end training framework, with all code available at [EvolvingLMMs-Lab/LLaVA-OneVision-1.5](https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5).
+3.  **Ultra-Efficient Training Framework**
+    Complete end-to-end training framework designed for maximum efficiency:
+    - **$16K total budget** for full model training
+    - **45% HFU efficiency** on A100 GPUs ($0.6 per GPU/Hour)
+    - Built on **MegatronLM** with support for **MoE**, **FP8**, and **long sequence parallelization**
+    - Optimized codebase for cost-effective scaling
 
+4.  **Fully Open Framework** for community access and reproducibility:
+    - ✅ High-quality mid-training & SFT data
+    - ✅ Complete training framework & code
+    - ✅ Training recipes & configurations
+    - ✅ Base & instruct model checkpoints
+    - ✅ Comprehensive training logs & metrics
 
 ## Models
 
-| Model                    | HF Link                                                                                      | Training Log |
-|--------------------------|--------------------------------------------------------------------------------------------------------|-------------|
-| LLaVA-OV-1.5-4B-Instruct | [🤗 HF / 4B-Instruct](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-4B-Instruct)                | [📈 Tensorboard](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-4B-Instruct/tensorboard) |
-| LLaVA-OV-1.5-8B-Instruct | [🤗 HF / 8B-Instruct](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-8B-Instruct)                | [📈 Tensorboard](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-8B-Instruct/tensorboard) |
-
+| Model | HF Link | Training Log |
+|---|---|---|
+| LLaVA-OV-1.5-4B-Instruct | [🤗 HF / 4B-Instruct](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-4B-Instruct) | [📈 Tensorboard](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-4B-Instruct/tensorboard) |
+| LLaVA-OV-1.5-8B-Instruct | [🤗 HF / 8B-Instruct](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-8B-Instruct) | [📈 Tensorboard](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-8B-Instruct/tensorboard) |
 
 ## Dataset
+
 | Description | Link |
-|-------------|------|
+|---|---|
 | Mid-training data for LLaVA-OneVision-1.5 | [🤗 Download (Uploading!)](https://huggingface.co/datasets/lmms-lab/LLaVA-One-Vision-1.5-Mid-Training-85M) |
 | SFT data for LLaVA-OneVision-1.5 | [🤗 Download (Uploading!)](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-1.5-Insturct-Data) |
 
 ## Evaluation Results
 All evaluations were conducted using [lmms_eval](https://github.com/EvolvingLMMs-Lab/lmms-eval).
 
-
 ![image](https://cdn-uploads.huggingface.co/production/uploads/655c70d331c4978366d4b2e6/J8oBYmQkTOC6pBNLgJn9d.png)
 
-### Using 🤗  Transformers to Chat
+## Quick Start with HuggingFace
+
 Here we show a code snippet to show you how to use the chat model with `transformers` and `qwen_vl_utils`:
 
 ```python
@@ -115,20 +116,260 @@ output_text = processor.batch_decode(
 print(output_text)
 ```
 
+## Evaluation
+```
+# pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git  
+
+accelerate launch --num_processes=8 --main_process_port 12399 -m lmms_eval \
+    --model=llava_onevision1_5 \
+    --model_args=pretrained=lmms-lab/LLaVA-OneVision-1.5-8B-Instruct,attn_implementation=flash_attention_2,max_pixels=3240000 \
+    --tasks=mmmu_val,mmmu_pro_standard,mmbench_en_test,mmerealworld,mmerealworld_cn,ai2d,ai2d_no_mask,vstar_bench,chartqa,charxiv,docvqa_test,mathvista_testmini,mmstar,scienceqa \
+    --batch_size=1
+```
+
+## Quick Start Guide
+
+### 1.🐳 Docker (Recommended)
+
+We strongly recommend using the docker environment for a seamless experience. The following instructions are tailored for the A100 80GB GPU environment.
+
+
+```bash
+# Clone repository
+git clone https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5.git
+cd LLaVA-OneVision-1.5
+
+docker build -t llava_megatron:25.04 .
+
+# Run container with -w to set working directory directly to the mounted volume
+docker run -it --gpus all \
+    --ipc host --net host --privileged --cap-add IPC_LOCK \
+    --ulimit memlock=-1 --ulimit stack=67108864 --rm \
+    -v $(pwd):/workspace/LLaVA-OneVision-1.5 \
+    -w /workspace/LLaVA-OneVision-1.5 \
+    --name "llava_megatron_container" \
+    llava_megatron:25.04 /bin/bash
+```
+
+### 2. Checkpoint and Format Conversion
+
+You have two options to get started with LLaVA-OneVision-1.5-stage-0:
+
+#### Option 1: Download pre-trained model from HuggingFace
+Download our `LLaVA-OneVision-1.5-4B-stage0` model directly from [HuggingFace](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-4B-stage0).
+
+#### Option 2: Merge initial weights yourself
+Alternatively, you can merge the initial weights from the original ViT and LLM:
+```bash
+python ds/merge_model.py \
+--vit_path DeepGlint-AI/rice-vit-large-patch14-560 \
+--llm_path Qwen/Qwen3-4B-Instruct-2507 \
+--output LLaVA-OneVision-1.5-4B-stage0
+```
+Note: When merging weights, the adapter component will be initialized with default values.
+
+Convert the model from HuggingFace format to Megatron format:
+
+```bash
+AIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-1.5 bash examples/llava_ov_1_5/convert/convert_4b_hf_to_mcore.sh \
+LLaVA-OneVision-1.5-4B-stage0 \
+LLaVA-OneVision-1.5-4B-stage0_mcore_tp1_pp1 \
+1 1
+```
+
+### 3. Stage 1 Alignment-Training
+
+Download LLaVA from [LLaVA-558K-Webdataset](https://huggingface.co/datasets/lmms-lab/LLaVA-558K-Webdataset).
+
+
+```bash
+# ============================================================
+# Required environment variables:
+#   AIAK_TRAINING_PATH  Root directory of the AIAK-Training-LLM project
+#   DATA_PATH           Directory with WebDataset shards (.tar) for pretraining
+#   TOKENIZER_PATH      Hugging Face tokenizer directory
+#   CHECKPOINT_PATH     Megatron-formatted checkpoint directory (e.g., mcore TP1/PP1)
+#   SAVE_CKPT_PATH      Output directory for saving training checkpoints
+AIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-1.5 \
+DATA_PATH=LLaVA-558K-Webdataset \
+TOKENIZER_PATH=LLaVA-OneVision-1.5-4B-stage0 \
+CHECKPOINT_PATH=LLaVA-OneVision-1.5-4B-stage0_mcore_tp1_pp1 \
+bash examples/llava_ov_1_5/quick_start/stage_1_alignment_llava_ov_4b.sh
+```
+
+### 4. Stage 1.5 Mid-Training 
+
+Download our lightweight packed subset from [LLaVA-OneVision-1.5-Mid-Training-Quick-Start-3M-Webdataset](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-1.5-Mid-Training-Webdataset-Quick-Start-3M).
+
+```bash
+# ============================================================
+# Convert model to release format
+bash examples/llava_ov_1_5/convert/convert_4b_mcore_to_release.sh \
+stage_1_alignment_llava_ov_4b/iter_0002500/ \
+stage_1_alignment_llava_ov_4b_release 1 1
+# ============================================================
+# Launch
+AIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-1.5 \
+DATA_PATH=LLaVA-OneVision-1.5-Mid-Training-Quick-Start-3M-Webdataset \
+TOKENIZER_PATH=LLaVA-OneVision-1.5-4B-stage0 \
+CHECKPOINT_PATH=stage_1_alignment_llava_ov_4b_release \
+bash examples/llava_ov_1_5/quick_start/stage_1.5_mid_training_llava_ov_4b.sh
+```
+
+
+### 5. Stage 2 Instruct-Training
+
+Download LLaVA-NeXT-780k-webdataset at [LLaVA-NeXT-780K Dataset](https://huggingface.co/datasets/lmms-lab/LLaVA-NeXT-780k-webdataset).
+
+```bash
+# ============================================================
+# Convert model to release format
+bash examples/llava_ov_1_5/convert/convert_4b_mcore_to_release.sh \
+stage_1.5_mid_training_llava_ov_4b/iter_0020000/ \
+stage_1.5_mid_training_llava_ov_4b_release 1 1
+# ============================================================
+# # Launch
+AIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-1.5 \
+DATA_PATH=LLaVA-NeXT-780k-Webdataset \
+TOKENIZER_PATH=LLaVA-OneVision-1.5-4B-stage0 \
+CHECKPOINT_PATH=stage_1.5_mid_training_llava_ov_4b_release \
+bash examples/llava_ov_1_5/quick_start/stage_2_instruct_llava_ov_4b.sh
+```
+
+
+### 6. Convert mcore to huggingface
+```bash
+AIAK_TRAINING_PATH=/workspace/LLaVA-OneVision-1.5 \
+bash examples/llava_ov_1_5/convert/convert_4b_mcore_to_hf.sh \
+stage_2_instruct_llava_ov_4b/iter_0003500 \
+LLaVA-OneVision-1.5-4B-3M-Mid-Training-780K-Instruct \
+1 1
+# Copy non-model files (e.g., tokenizer config) to the new directory
+find LLaVA-OneVision-1.5-4B-stage0/ -type f -not -iname '*safetensors*' -exec cp {}  LLaVA-OneVision-1.5-4B-3M-Mid-Training-780K-Instruct/ ';'
+```
+
+### 7. Evaluation
+```bash
+# pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git
+CUDA_VISIBLE_DEVICES=4,5,6,7 accelerate launch \
+--num_processes=4 --main_process_port 12399 -m lmms_eval --model=llava_onevision1_5 --batch_size=1 --tasks=mme \
+--model_args=pretrained=/workspace/LLaVA-OneVision-1.5/LLaVA-OneVision-1.5-4B-3M-Mid-Training-780K-Instruct,max_pixels=3240000
+```
+
+## Fully Reproducing Guide
+
+> [!TIP]
+> More detailed reproduction steps for the complete process will be provided after the dataset upload is completed.
+
+
+### Mid-Training
+
+To improve model training efficiency, we implement offline sample packing:
+
+1.  Download the [**Mid-Training-85M Dataset**](https://huggingface.co/datasets/lmms-lab/LLaVA-One-Vision-1.5-Mid-Training-85M)
+2.  Pack the data into webdataset format, refer to [**Examples offlinepacking**](examples_offline_packing) and [**Offline Padding-Free Data Packing**](examples/llava_ov_1_5/sample_packing/README.md)
+
+
+### Instruct
+1.  Download the [**LLaVA-OneVision-1.5-Insturct-Data**](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-1.5-Insturct-Data)
+2.  Convert the data into webdataset format, refer to [**Conversion for Mixed Instruction Data**](docs/sft_data_preprocessing.md)
+
+## Roadmaps
+
+Q4 2025 Key Deliverables:
+
+1.  **Ultra-efficient MoE Training**  
+2.  **Full Video Input LLM**  
+
+
+## Contributors
+Thanks so much to all of our amazing contributors!
+
+<!-- readme: collaborators,contributors,jiankangdeng/- -start -->
+<table>
+	<tbody>
+		<tr>
+            <td align="center">
+                <a href="https://github.com/fdcp">
+                    <img src="https://avatars.githubusercontent.com/u/15667917?v=4" width="80;" alt="fdcp"/>
+                    <br />
+                    <sub><b>fdcp</b></sub>
+                </a>
+            </td>
+            <td align="center">
+                <a href="https://github.com/anxiangsir">
+                    <img src="https://avatars.githubusercontent.com/u/31175974?v=4" width="80;" alt="anxiangsir"/>
+                    <br />
+                    <sub><b>anxiangsir</b></sub>
+                </a>
+            </td>
+            <td align="center">
+                <a href="https://github.com/yiyexy">
+                    <img src="https://avatars.githubusercontent.com/u/35927125?v=4" width="80;" alt="yiyexy"/>
+                    <br />
+                    <sub><b>yiyexy</b></sub>
+                </a>
+            </td>
+            <td align="center">
+                <a href="https://github.com/wideyard">
+                    <img src="https://avatars.githubusercontent.com/u/101321826?v=4" width="80;" alt="wideyard"/>
+                    <br />
+                    <sub><b>wideyard</b></sub>
+                </a>
+            </td>
+            <td align="center">
+                <a href="https://github.com/chengzheng345">
+                    <img src="https://avatars.githubusercontent.com/u/209475443?v=4" width="80;" alt="chengzheng345"/>
+                    <br />
+                    <sub><b>chengzheng345</b></sub>
+                </a>
+            </td>
+            <td align="center">
+                <a href="https://github.com/killTheHostage">
+                    <img src="https://avatars.githubusercontent.com/u/16442720?v=4" width="80;" alt="killTheHostage"/>
+                    <br />
+                    <sub><b>killTheHostage</b></sub>
+                </a>
+            </td>
+            <td align="center">
+                <a href="https://github.com/mathCrazyy">
+                    <img src="https://avatars.githubusercontent.com/u/20607153?v=4" width="80;" alt="mathCrazyy"/>
+                    <br />
+                    <sub><b>mathCrazyy</b></sub>
+                </a>
+            </td>
+            <td align="center">
+                <a href="https://github.com/yunglechao">
+                    <img src="https://avatars.githubusercontent.com/u/7631185?v=4" width="80;" alt="yunglechao"/>
+                    <br />
+                    <sub><b>yunglechao</b></sub>
+                </a>
+            </td>
+		</tr>
+		<tr>
+            <td align="center">
+                <a href="https://github.com/RobitYadda">
+                    <img src="https://avatars.githubusercontent.com/u/6811311?v=4" width="80;" alt="RobitYadda"/>
+                    <br />
+                    <sub><b>RobitYadda</b></sub>
+                </a>
+            </td>
+		</tr>
+	<tbody>
+</table>
+<!-- readme: collaborators,contributors,jiankangdeng/- -end -->
+
 ## Citation
 
 If you find *LLaVA-OneVision-1.5* useful in your research, please consider to cite the following related papers:
 
 ```
-@misc{an2025llavaonevision15fullyopenframework,
-      title={LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training}, 
-      author={Xiang An and Yin Xie and Kaicheng Yang and Wenkang Zhang and Xiuwei Zhao and Zheng Cheng and Yirui Wang and Songcen Xu and Changrui Chen and Chunsheng Wu and Huajie Tan and Chunyuan Li and Jing Yang and Jie Yu and Xiyao Wang and Bin Qin and Yumeng Wang and Zizhen Yan and Ziyong Feng and Ziwei Liu and Bo Li and Jiankang Deng},
-      year={2025},
-      eprint={2509.23661},
-      archivePrefix={arXiv},
-      primaryClass={cs.CV},
-      url={https://arxiv.org/abs/2509.23661}, 
-}
+@inproceedings{LLaVA-OneVision-1.5,
+  title={LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training},
+  author={An, Xiang and Xie, Yin and Yang, Kaicheng and Zhang, Wenkang and Zhao, Xiuwei and Cheng, Zheng and Wang, Yirui and Xu, Songcen and Chen, Changrui and Wu, Chunsheng and Tan, Huajie and Li, Chunyuan and Yang, Jing and Yu, Jie and Wang, Xiyao and Qin, Bin and Wang, Yumeng and Yan, Zizhen and Feng, Ziyong and Liu, Ziwei and Li, Bo and Deng, Jiankang},
+  booktitle={arxiv},  
+  year={2025}
+ }
 
 @inproceedings{xie2025region,
   title={Region-based Cluster Discrimination for Visual Representation Learning},
@@ -143,4 +384,21 @@ If you find *LLaVA-OneVision-1.5* useful in your research, please consider to ci
   journal={Transactions on Machine Learning Research}
   year={2024}
 }
-```
+```
+
+## Acknowledgement
+
+We extend our sincere gratitude to **AIAK team of the** [**Baige AI computing platform**](https://cloud.baidu.com/product/aihc.html) **from Baidu AI Cloud** for providing the exceptional training framework. The outstanding capabilities of AIAK-Training-LLM and AIAK-Megatron have significantly accelerated our training process with remarkable efficiency. These cutting-edge frameworks have been instrumental in achieving our research goals. `To get full AIAK support, you can contact Baidu Cloud.`
+
+
+We also thank the maintainers and contributors of the following open-source projects, whose work greatly inspired and supported our research:
+
+- LLaVA: Large Language-and-Vision Assistant — [LLaVA](https://github.com/haotian-liu/LLaVA)
+- LLaVA-NeXT: Next-generation multi-modal assistant — [LLaVA-NeXT](https://github.com/LLaVA-VL/LLaVA-NeXT)
+- lmms-eval: A standardized evaluation framework for Large Multimodal Models — [lmms-eval](https://github.com/EvolvingLMMs-Lab/lmms-eval)
+- Megatron-LM: Efficient, scalable training for large language models — [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)
+- Qwen2.5-VL: Strong vision-language foundation model — [Qwen2.5-VL](https://github.com/QwenLM/Qwen2.5-VL)
+- InternVL: Open-source large-scale vision-language foundation model — [InternVL](https://github.com/OpenGVLab/InternVL)
+- Qwen3: Next-generation Qwen LLM — [Qwen](https://github.com/QwenLM/Qwen)
+- MetaCLIP: Scalable contrastive pretraining — [MetaCLIP](https://github.com/facebookresearch/MetaCLIP)
+- FineVision: Open Data Is All You Need — [FineVision](https://huggingface.co/spaces/HuggingFaceM4/FineVision)