--- datasets: - liuhaotian/LLaVA-Pretrain - liuhaotian/LLaVA-Instruct-150K pipeline_tag: image-text-to-text library_name: xtuner ---
[![Generic badge](https://img.shields.io/badge/GitHub-%20XTuner-black.svg)](https://github.com/InternLM/xtuner)
## Model llava-internlm-7b is a LLaVA model fine-tuned from [InternLM-Chat-7B](https://huggingface.co/internlm/internlm-chat-7b) and [CLIP-ViT-Large-patch14-336](https://huggingface.co/openai/clip-vit-large-patch14-336) with [LLaVA-Pretrain](https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain) and [LLaVA-Instruct](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K) by [XTuner](https://github.com/InternLM/xtuner). ## Quickstart ### Installation ```shell pip install -U 'xtuner[deepspeed]' ``` ### Chat ```shell xtuner chat internlm/internlm-chat-7b \ --visual-encoder openai/clip-vit-large-patch14-336 \ --llava xtuner/llava-internlm-7b \ --prompt-template internlm_chat \ --image $IMAGE_PATH ``` ### Training 1. Alignment module pretraining (saved by default in `./work_dirs/`) ```shell NPROC_PER_NODE=8 xtuner train llava_internlm_chat_7b_clip_vit_large_p14_336_e1_gpu8_pretrain --deepspeed deepspeed_zero2 ``` 2. Instruction following fine-tuning (saved by default in `./work_dirs/`) ```shell NPROC_PER_NODE=8 xtuner train llava_internlm_chat_7b_qlora_clip_vit_large_p14_336_lora_e1_gpu8_finetune --deepspeed deepspeed_zero2 ``` ### MMBench Evaluation XTuner integrates the MMBench evaluation, and you can perform evaluations with the following command! ```bash xtuner mmbench internlm/internlm-chat-7b \ --visual-encoder openai/clip-vit-large-patch14-336 \ --llava xtuner/llava-internlm-7b \ --prompt-template internlm_chat \ --data-path $MMBENCH_DATA_PATH \ --work-dir $RESULT_PATH ``` After the evaluation is completed, if it's a development set, it will directly print out the results; If it's a test set, you need to submit `mmbench_result.xlsx` to the official MMBench for final evaluation to obtain precision results! ## Citation ```bibtex @misc{2023xtuner, title={XTuner: A Toolkit for Efficiently Fine-tuning LLM}, author={XTuner Contributors}, howpublished = {\url{https://github.com/InternLM/xtuner}}, year={2023} } ```