Update README.md

README.md

@@ -57,6 +57,155 @@ This model was developed using English corpora, and thus may be considered Engli
 Further, this model was developed in part using the [PMC-15M](https://aka.ms/biomedclip-paper) dataset. The figure-caption pairs that make up this dataset may contain biases reflecting the current practice of academic publication. For example, the corresponding papers may be enriched for positive findings, contain examples of extreme cases, and otherwise reflect distributions that are not representative of other sources of biomedical data. 
 
 
+## Evaluation
+
+### Medical Visual Chat (GPT-assisted Evaluation)
+
+Our GPT-assisted evaluation pipeline for multimodal modeling is provided for a comprehensive understanding of the capabilities of vision-language models.  Please see our paper for more details.
+
+1. Generate LLaVA-Med responses
+
+```Shell
+python model_vqa.py \
+    --model-name ./checkpoints/LLaVA-7B-v0 \
+    --question-file data/eval/llava_med_eval_qa50_qa.jsonl \
+    --image-folder data/images/ \
+    --answers-file /path/to/answer-file.jsonl
+```
+
+2. Evaluate the generated responses.  In our case, [`llava_med_eval_qa50_qa.jsonl`](/data/eval/llava_med_eval_qa50_qa.jsonl) contains the questions, context (captions and inline-mentions) and responses generated by text-only GPT-4 (0314), which we treat as ground truth.
+
+```Shell
+python llava/eval/eval_multimodal_chat_gpt_score.py \
+    --question_input_path data/eval/llava_med_eval_qa50_qa.jsonl \
+    --input_path /path/to/answer-file.jsonl \
+    --output_path /path/to/save/gpt4-eval-for-individual-answers.jsonl
+```
+
+3. Summarize the evaluation results
+
+```Shell
+python summarize_gpt_review.py
+```
+
+### Medical VQA
+
+Three Medical VQA datasets are considered in our experiments, including  VQA-Rad, SLAKE, Pathology-VQA. We use VQA-Rad as the running example to illustrate how LLaVA-Med is applied to a downstream scenario.
+
+#### - Prepare Data
+1. Please see VQA-Rad [repo](https://paperswithcode.com/dataset/vqa-rad) for setting up the dataset.
+2. Generate VQA-Rad dataset for LLaVA-Med conversation-style format (the same format with instruct tuning). For each dataset, we process it into three components: `train.json`, `test.json`, `images`.
+
+
+#### - Fine-tuning
+
+To achieve the higher performance for given a downstream dataset, the same full-model tuning script with instruct tuning is used to continue train LLaVA-Med.
+
+<details>
+<summary> Detailed script to fine-tune to downstream datasets: LLaVA-Med-7B, 8x A100 (40G).  Time: ~1 hour.</summary>
+
+```Shell
+torchrun --nnodes=1 --nproc_per_node=8 --master_port=25001 \
+    llava/train/train_mem.py \
+    --model_name_or_path /path/to/checkpoint_llava_med_instruct_60k_inline_mention \
+    --data_path /path/to/eval/vqa_rad/train.json \
+    --image_folder /path/to/eval/vqa_rad/images \
+    --vision_tower openai/clip-vit-large-patch14 \
+    --mm_vision_select_layer -2 \
+    --mm_use_im_start_end True \
+    --bf16 True \
+    --output_dir /path/to/checkpoint_llava_med_instruct_60k_inline_mention/eval/fine_tuned/vqa_rad \
+    --num_train_epochs 3 \
+    --per_device_train_batch_size 1 \
+    --per_device_eval_batch_size 4 \
+    --gradient_accumulation_steps 8 \
+    --evaluation_strategy "no" \
+    --save_strategy "steps" \
+    --save_steps 5000 \
+    --save_total_limit 3 \
+    --learning_rate 2e-5 \
+    --weight_decay 0. \
+    --warmup_ratio 0.03 \
+    --lr_scheduler_type "cosine" \
+    --logging_steps 1 \
+    --tf32 True \
+    --fsdp "full_shard auto_wrap" \
+    --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
+    --model_max_length 2048 \
+    --gradient_checkpointing True \
+    --lazy_preprocess True \
+    --report_to wandb
+```
+</details>
+
+## Serving
+
+| Model Delta Weights | Size |
+| --- | ---: |
+| [llava_med_in_text_60k_delta.zip](https://hanoverprod.z21.web.core.windows.net/med_llava/models/llava_med_in_text_60k_delta.zip) | 11.06 GB |
+
+The model weights above are *delta* weights. The usage of LLaVA-Med checkpoints should comply with the base LLM's model license: [LLaMA](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md).
+
+Instructions:
+
+1. Download the delta weights [llava_med_in_text_60k_delta.zip](https://hanoverprod.z21.web.core.windows.net/med_llava/models/llava_med_in_text_60k_delta.zip) and unzip.
+1. Get the original LLaMA weights in the huggingface format by following the instructions [here](https://huggingface.co/docs/transformers/main/model_doc/llama).
+1. Use the following scripts to get LLaVA-Med weights by applying our delta. In the script below, set the --delta argument to the path of the unzipped `llava_med_in_text_60k_delta` directory. It can be adapted for other delta weights by changing the `--delta` argument (and base/target accordingly).
+
+```bash
+python3 -m llava.model.apply_delta \
+    --base /path/to/llama-7b \
+    --target /output/path/to/llava_med_in_text_60k \
+    --delta path/to/llava_med_in_text_60k_delta
+```
+
+
+## - Evaluation
+
+Depending on which checkpoint is employed in evaluation, zero-shot performance is reported on medical instruct tuned checkpoint (eg, [LLaVA-Med-7B](/path/to/checkpoint_llava_med_instruct_60k_inline_mention)), and fine-tuned performance is reported on checkpoint that has been further tuned on training set of the downstream datasets (eg, [LLaVA-Med-7B-VQA-Rad](/path/to/checkpoint_llava_med_instruct_60k_inline_mention/fine_tuned/vqa_rad) ).
+
+(a) Generate LLaVA responses on ScienceQA dataset
+
+(a.1). [Option 1] Multiple-GPU inference
+You may evaluate this with multiple GPUs, and concatenate the generated jsonl files.  Please refer to our script for [batch evaluation](scripts/chunyl/finetune_on_benchmarks/eval_med_dataset_batch.sh).
+
+```Shell
+python llava/eval/run_med_datasets_eval_batch.py --num-chunks 8  --model-name /path/to/checkpoint_llava_med_instruct_60k_inline_mention/eval/fine_tuned/vqa_rad \
+    --question-file path/to/eval/vqa_rad/test.json \
+    --image-folder path/to/eval/vqa_rad/images \
+    --answers-file /path/to/checkpoint_llava_med_instruct_60k_inline_mention/eval/fine_tuned/vqa_rad/test-answer-file.jsonl
+```
+(a.2). [Option 2] Single-GPU inference
+
+```Shell
+python llava/eval/model_vqa_med.py --model-name /path/to/checkpoint_llava_med_instruct_60k_inline_mention/eval/fine_tuned/vqa_rad \
+    --question-file path/to/eval/vqa_rad/test.json \
+    --image-folder path/to/eval/vqa_rad/images \
+    --answers-file /path/to/checkpoint_llava_med_instruct_60k_inline_mention/eval/fine_tuned/vqa_rad/test-answer-file.jsonl
+```
+
+(b) Evaluate the generated responses
+
+(b.1). [Option 1] Evaluation for all three VQA datasets
+```Shell
+
+python llava/eval/run_eval_batch.py \
+    --pred_file_parent_path /path/to/llava-med \
+    --target_test_type test-answer-file
+```
+
+It collects the decoding results of all predictions files under the project path, computes the corresponding evaluation metrics, and outputs the results in "`eval_results_med_datasets.jsonl`". To analyze the score, we provdie ipython notebook [run_eval_metrics.ipynb](llava/notebook/run_eval_metrics.ipynb).
+
+(b.2). [Option 2] Evaluation for on one specific VQA dataset
+```Shell
+python llava/eval/run_eval.py \
+    --gt /path/to/eval/vqa_rad/test.json \
+    --pred /path/to/checkpoint_llava_med_instruct_60k_inline_mention/eval/fine_tuned/vqa_rad/test-answer-file.jsonl
+```
+
+Please find the LLaVA-Med performance in [llava_med_performance.md](docs/llava_med_performance.md) or in the paper.
+
+
 ## Acknowledgement
 
 - Our project is built upon [LLaVA](https://github.com/lm-sys/FastChat) and [Vicuna](https://github.com/lm-sys/FastChat): They provide our base models with the amazing multimodal and langauge capabilities, respectively!