File size: 4,347 Bytes
032e687 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 |
# VLM
Codebase of VLM projects
## Evaluation
Currently, the codebase supports evaluation on several benchmarks, including HallusionBench, ai2d, docvqa, mmbench, mme, mmstar, ocrvqa, pope, seed_bench, sqa, textvqa, and vqav2. You can modify the configuration in the config file to enable evaluation.
### Config
Please refer to
[llava_test.py](./projects/llava/configs/vicuna_7b_v15_vit_14_336/test/llava_vicuna_7b_v15_qlora_clip_vit_large_p14_336_lora_e1_gpu8.py) or
[omg_llava_test.py](./projects/omg_llava/configs/test/omg_llava_7b_finetune_8gpus.py).
1. Firstly, you need load the evaluation benchmarks from [here](https://huggingface.co/datasets/OMG-Research/VLM). And put them to `./data/`.
2. Copy the train config of your model and delete the custom_hooks.
```commandline
# remove custom_hooks
custom_hooks = []
```
3. Implement the preparing_for_generation and predict_forward for your model.
Please refer to [llava](./projects/llava/model/llava.py) or [omg_llava](./projects/omg_llava/model/omg_llava.py).
preparing_for_generation set the generation setting for the model such as template. predict_forward is the predict forward function of your method, the input is items from the test dataset (such as pixel_values and text_prompts), the output is the response dict.
2. Add these items in your config.
```commandline
test_dataset = [
dict(
type=MultipleChoiceDataset,
data_file='./data/eval/mmbench/MMBench_DEV_EN.tsv',
image_processor=image_processor,
pad_image_to_square=True,
),
dict(
type=MultipleChoiceDataset,
data_file='./data/eval/mmbench/MMBench_TEST_EN.tsv',
image_processor=image_processor,
pad_image_to_square=True,
),
dict(
type=MMEDataset,
data_file='./data/eval/mme/MME.tsv',
image_processor=image_processor,
pad_image_to_square=True,
),
dict(
type=MultipleChoiceDataset,
data_file='./data/eval/seed_bench/SEEDBench_IMG.tsv',
image_processor=image_processor,
pad_image_to_square=True,
),
dict(
type=MultipleChoiceDataset,
data_file='./data/eval/sqa/ScienceQA_VAL.tsv',
image_processor=image_processor,
pad_image_to_square=True,
),
dict(
type=MultipleChoiceDataset,
data_file='./data/eval/sqa/ScienceQA_TEST.tsv',
image_processor=image_processor,
pad_image_to_square=True,
),
dict(
type=MultipleChoiceDataset,
data_file='./data/eval/ai2d/AI2D_TEST.tsv',
image_processor=image_processor,
pad_image_to_square=True,
),
dict(
type=MultipleChoiceDataset,
data_file='./data/eval/mmstar/MMStar.tsv',
image_processor=image_processor,
pad_image_to_square=True,
),
dict(
type=HallusionDataset,
data_file='./data/eval/HallusionBench/HallusionBench.tsv',
image_processor=image_processor,
pad_image_to_square=True,
),
dict(
type=POPEDataset,
data_file=[
'./data/eval/pope/coco_pope_adversarial.json',
'./data/eval/pope/coco_pope_popular.json',
'./data/eval/pope/coco_pope_random.json',
],
coco_val_path='./data/eval/val2014/',
image_processor=image_processor,
pad_image_to_square=True,
),
]
test_dataloader = dict(
batch_size=1,
num_workers=0,
drop_last=False,
sampler=dict(type=DefaultSampler, shuffle=False),
dataset=dict(type=ConcatDataset, datasets=test_dataset),
)
test_evaluator = dict()
test_cfg = dict(type=TestLoop, select_metric='first')
```
5. Perform test.
```commandline
# example
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7,8 PYTHONPATH=. bash tools/dist.sh test projects/omg_llava/configs/test/omg_llava_7b_finetune_8gpus.py 8 --checkpoint ./pretrained/omg_llava/omg_llava_fintune_8gpus.pth
```
| model | MMbench-DEV-EN | SEEDBench | MME | ScienceQA_VAL | ScienceQA_TEST | AI2D | MMStar |
|------------------------|----------------|-----------|------|---------------|----------------|------|--------|
| llava-vicuna-7b | 68.5 | 65.9 | 1689 | 67.6 | 68.9 | 56.7 | 34.8 |
| omg-llava-internlm2-7b | 45.7 | 54.2 | 1255 | 53.5 | 55.6 | 42.3 | 34.8 |
|