| Model Open Ended VQA: % Human Rating Multiple Choice VQA: % Accuracy Hints-Multiple Choice VQA: % Accuracy Attributions-Multiple Choice VQA: % Accuracy Refernce Based-Automatic Evaluation: Accuracy of Judge Prediction Compared to Human Ratings Refernce Free-Automatic Evaluation: Accuracy of Judge Prediction Compared to Human Ratings Automatic Evaluation: % Auto-Rater Ratings Hints-Automatic Evaluation: % Auto-Rater Ratings Attributions-Automatic Evaluation: % Auto-Rater Ratings | |
| Humans 82 * * * * * 78 * * | |
| Gemini Pro 1.5 40 38 66 72 87 52 53 62 29 | |
| Gemini Pro Vision 30 41 62 * 75 38 34 47 | |
| GPT4 34 45 69 82 86 51 38 61 25 | |
| LlaVA-1.6-34B 15 24 30 * 76 43 21 16 * | |
| LlaVA-1.5-7B 13 17 29 * 70 35 19 30 * | |
| InstructBlip 13 * * * * * 20 28 * | |
| Gemini Pro 1.5 Caption _ Gemini Pro 1.5 23 * * * * * * * * | |
| Human (Oracle) Caption _ Gemini Pro 1.5 50 * * * * * * * * | |
| Claude 3.5 Sonnet * 46 45 * * * 39 * * | |
| GPT4o * 55 83 * * * 50 * * | |
| Qwen-VL-Max * 35 53 * * * 26 * * | |
| Molmo-7B * 34 42 * * * 36 * * |