Zhang Hui
commited on
Commit
•
9a424a3
1
Parent(s):
609b6ac
add more results
Browse files- .gitignore +1 -0
- MMMU.png +0 -3
- README.md +45 -6
.gitignore
ADDED
@@ -0,0 +1 @@
|
|
|
|
|
1 |
+
results/
|
MMMU.png
DELETED
Git LFS Details
|
README.md
CHANGED
@@ -24,10 +24,13 @@ A multimodal large-scale model, characterized by its open-source nature, closely
|
|
24 |
|
25 |
视觉编码部分继承自Qwen-VL-Chat,即Openclip ViT-bigG。
|
26 |
|
|
|
|
|
|
|
|
|
27 |
## Quick Start
|
28 |
|
29 |
```
|
30 |
-
|
31 |
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig
|
32 |
tokenizer = AutoTokenizer.from_pretrained(
|
33 |
pretrained_model_name_or_path="huizhang0110/CatVision",
|
@@ -55,19 +58,33 @@ response, history = model.chat(
|
|
55 |
|
56 |
## Benchmark
|
57 |
|
58 |
-
Our model achieved favorable results on the
|
59 |
|
60 |
- **[MMMU](https://eval.ai/web/challenges/challenge-page/2179/leaderboard/5377)**
|
61 |
|
62 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
63 |
|
64 |
- **[CMMMU](https://github.com/CMMMU-Benchmark/CMMMU/blob/main/README.md)**
|
65 |
|
66 |
| Model | Val (900) | Test (11K) |
|
67 |
|--------------------------------|:---------:|:------------:|
|
68 |
-
| GPT-4V(ision) (Playground) |
|
69 |
| Qwen-VL-PLUS* | 39.5 | 36.8 |
|
70 |
-
| CatVision
|
71 |
| Yi-VL-34B | 36.2 | 36.5 |
|
72 |
| Yi-VL-6B | 35.8 | 35.0 |
|
73 |
| Qwen-VL-7B-Chat | 30.7 | 31.3 |
|
@@ -81,6 +98,28 @@ Our model achieved favorable results on the [MMMU](https://eval.ai/web/challenge
|
|
81 |
| Frequent Choice | 24.1 | 26.0 |
|
82 |
| Random Choice | 21.6 | 21.6 |
|
83 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
84 |
|
85 |
- **Show Case**
|
86 |
|
@@ -101,7 +140,7 @@ Our model achieved favorable results on the [MMMU](https://eval.ai/web/challenge
|
|
101 |
```
|
102 |
@misc{CatVision,
|
103 |
author = {zhanghui@4paradigm.com},
|
104 |
-
title = {
|
105 |
year = {2024},
|
106 |
publisher = {huggingface},
|
107 |
howpublished = {\url{https://huggingface.co/huizhang0110/CatVision}}
|
|
|
24 |
|
25 |
视觉编码部分继承自Qwen-VL-Chat,即Openclip ViT-bigG。
|
26 |
|
27 |
+
- We are continuously collecting instruction data, optimizing the model, and looking forward to supporting more tasks.
|
28 |
+
|
29 |
+
我们正在持续收集指令数据,优化模型,期待能支持更多的功能。
|
30 |
+
|
31 |
## Quick Start
|
32 |
|
33 |
```
|
|
|
34 |
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig
|
35 |
tokenizer = AutoTokenizer.from_pretrained(
|
36 |
pretrained_model_name_or_path="huizhang0110/CatVision",
|
|
|
58 |
|
59 |
## Benchmark
|
60 |
|
61 |
+
Our model achieved favorable results on the many leaderboards.
|
62 |
|
63 |
- **[MMMU](https://eval.ai/web/challenges/challenge-page/2179/leaderboard/5377)**
|
64 |
|
65 |
+
| Model | Val (900) | Test (11K) |
|
66 |
+
|--------------------------------|:---------:|:------------:|
|
67 |
+
| Gemini Ultra | 59.4 | ---- |
|
68 |
+
| GPT4V | 56.8 | 55.7 |
|
69 |
+
| Gemini Pro | 47.9 | ---- |
|
70 |
+
| Yi-VL-34B | 45.9 | 41.6 |
|
71 |
+
| Qwen-VL-PLUS | 45.2 | 40.8 |
|
72 |
+
| *CatVision* | 45.9 | 40.1 |
|
73 |
+
| Macro-VL | 41.2 | 40.4 |
|
74 |
+
| InfiMM-Zephyr-7B | 39.4 | 35.5 |
|
75 |
+
| Yi-VL-6B | 39.1 | 37.8 |
|
76 |
+
| SVIT | 38.0 | 34.1 |
|
77 |
+
| LLaVA-1.5-13B | 36.4 | 33.6 |
|
78 |
+
| Emu2-Chat | 36.3 | 34.1 |
|
79 |
+
| Qwen-VL-7B-Chat | 35.9 | 32.9 |
|
80 |
|
81 |
- **[CMMMU](https://github.com/CMMMU-Benchmark/CMMMU/blob/main/README.md)**
|
82 |
|
83 |
| Model | Val (900) | Test (11K) |
|
84 |
|--------------------------------|:---------:|:------------:|
|
85 |
+
| GPT-4V(ision) (Playground) | 42.5 | 43.7 |
|
86 |
| Qwen-VL-PLUS* | 39.5 | 36.8 |
|
87 |
+
| *CatVision* | 39.6 | ---- |
|
88 |
| Yi-VL-34B | 36.2 | 36.5 |
|
89 |
| Yi-VL-6B | 35.8 | 35.0 |
|
90 |
| Qwen-VL-7B-Chat | 30.7 | 31.3 |
|
|
|
98 |
| Frequent Choice | 24.1 | 26.0 |
|
99 |
| Random Choice | 21.6 | 21.6 |
|
100 |
|
101 |
+
- **[MMBench](https://mmbench.opencompass.org.cn/leaderboard)**
|
102 |
+
|
103 |
+
| Model | mmbench_cn (test) | mmbench_cn (dev) | mmbench_en (test) | mmbench_zh (dev) | ccbench |
|
104 |
+
|---------------------|:-----------------:|:----------------:|:-----------------:|:----------------:|:-------:|
|
105 |
+
| Qwen-VL-PLUS(BASE) | 83.3 | 83.2 | 82.7 | 81.5 | 77.6 |
|
106 |
+
| GPT4v | 77.0 | 75.1 | 74.4 | 75.0 | 46.5 |
|
107 |
+
| Qwen-VL-PLUS | 67.0 | 66.2 | 70.7 | 69.6 | 55.1 |
|
108 |
+
| *CatVision* | 70.9 | 71.8 | 70.2 | 71.6 | 49.8 |
|
109 |
+
| Qwen-VL-Chat | 61.8 | 60.6 | 56.3 | 56.7 | 41.2 |
|
110 |
+
|
111 |
+
- **[MME](https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models)**
|
112 |
+
|
113 |
+
| Model | Perception | Cognition |
|
114 |
+
|---------------|:----------:|:---------:|
|
115 |
+
| GPT4v | 1409.43 | 517.14 |
|
116 |
+
| Qwen-VL-PLUS | 1681.25 | 502.14 |
|
117 |
+
| *CatVision* | 1560.90 | 366.43 |
|
118 |
+
| Qwen-VL-Chat | 1487.57 | 360.71 |
|
119 |
+
|
120 |
+
- **Open Compress**
|
121 |
+
|
122 |
+
wait
|
123 |
|
124 |
- **Show Case**
|
125 |
|
|
|
140 |
```
|
141 |
@misc{CatVision,
|
142 |
author = {zhanghui@4paradigm.com},
|
143 |
+
title = {CatVision},
|
144 |
year = {2024},
|
145 |
publisher = {huggingface},
|
146 |
howpublished = {\url{https://huggingface.co/huizhang0110/CatVision}}
|