Hugging Face's logo

openbmb
/
MiniCPM-Llama3-V-2_5

Visual Question Answering
Transformers
Safetensors
English
Chinese
minicpmv
feature-extraction
custom_code
Model card Files Community
68
finalf0 commited on
Commit
b542605
•
1 Parent(s): f17d4d9

update readme

Browse files
Files changed (2) hide show
  1. README.md +5 -311
  2. assets/MiniCPM-Llama3-V-2.5-benchmark.png +3 -0
README.md CHANGED
@@ -33,319 +33,13 @@ datasets:
33
  ### Evaluation <!-- omit in toc -->
34
 
35
  <div align="center">
36
- <img src=assets/MiniCPM-Llama3-V-2.5-peformance.png width=66% />
37
  </div>
38
- <details>
39
- <summary>Click to view results on TextVQA, DocVQA, OCRBench, OpenCompass, MME, MMBench, MMMU, MathVista, LLaVA Bench, RealWorld QA, Object HalBench. </summary>
40
- <div align="center">
41
-
42
- <table style="margin: 0px auto;">
43
- <caption>Results on OCR-specific and General Multimodal Benchmarks</caption>
44
- <thead>
45
- <tr>
46
- <th align="left">Model</th>
47
- <th>Size</th>
48
- <th>OCRBench</th>
49
- <th>TextVQA val</th>
50
- <th>DocVQA test</th>
51
- <th>Open-Compass</th>
52
- <th>MME</th>
53
- <th>MMB dev (en)</th>
54
- <th>MMB dev (zh)</th>
55
- <th>MMMU val</th>
56
- <th>Math-Vista</th>
57
- <th>LLaVA Bench</th>
58
- <th>RealWorld QA</th>
59
- <th>Object HalBench</th>
60
- </tr>
61
- </thead>
62
- <tbody align="center">
63
- <tr>
64
- <td colspan="14" align="left"><strong>Proprietary</strong></td>
65
- </tr>
66
- <tr>
67
- <td nowrap="nowrap" align="left">Gemini Pro</td>
68
- <td>-</td>
69
- <td>680</td>
70
- <td>74.6</td>
71
- <td>88.1</td>
72
- <td>63.8</td>
73
- <td>2148.9</td>
74
- <td>75.2</td>
75
- <td>74.0</td>
76
- <td>48.9</td>
77
- <td>45.8</td>
78
- <td>79.9</td>
79
- <td>60.4</td>
80
- <td>-</td>
81
- </tr>
82
- <tr>
83
- <td nowrap="nowrap" align="left">GPT-4V (2023.11.06)</td>
84
- <td>-</td>
85
- <td>645</td>
86
- <td>78.0</td>
87
- <td>88.4</td>
88
- <td>63.2</td>
89
- <td>1771.5</td>
90
- <td>75.1</td>
91
- <td>75.0</td>
92
- <td>53.8</td>
93
- <td>47.8</td>
94
- <td>93.1</td>
95
- <td>63.0</td>
96
- <td>86.4</td>
97
- </tr>
98
- <tr>
99
- <td colspan="14" align="left"><strong>Open-source</strong></td>
100
- </tr>
101
- <tr>
102
- <td nowrap="nowrap" align="left">DeepSeek-VL-1.3B</td>
103
- <td>1.7B</td>
104
- <td>413</td>
105
- <td>58.4*</td>
106
- <td>37.9*</td>
107
- <td>46.0</td>
108
- <td>1531.6</td>
109
- <td>64.0</td>
110
- <td>61.2</td>
111
- <td>33.8</td>
112
- <td>29.4</td>
113
- <td>51.1</td>
114
- <td>49.7</td>
115
- <td>-</td>
116
- </tr>
117
- <tr>
118
- <td nowrap="nowrap" align="left">Mini-Gemini</td>
119
- <td>2.2B</td>
120
- <td>-</td>
121
- <td>56.2</td>
122
- <td>34.2*</td>
123
- <td>-</td>
124
- <td>1653.0</td>
125
- <td>59.8</td>
126
- <td>-</td>
127
- <td>31.7</td>
128
- <td>-</td>
129
- <td>-</td>
130
- <td>-</td>
131
- <td>-</td>
132
- </tr>
133
- <tr>
134
- <td nowrap="nowrap" align="left">Yi-VL-6B</td>
135
- <td>6.7B</td>
136
- <td>290</td>
137
- <td>45.5*</td>
138
- <td>17.1*</td>
139
- <td>49.3</td>
140
- <td>1915.1</td>
141
- <td>68.6</td>
142
- <td>68.3</td>
143
- <td>40.3</td>
144
- <td>28.8</td>
145
- <td>51.9</td>
146
- <td>53.5</td>
147
- <td>-</td>
148
- </tr>
149
- <tr>
150
- <td nowrap="nowrap" align="left">Qwen-VL-Chat</td>
151
- <td>9.6B</td>
152
- <td>488</td>
153
- <td>61.5</td>
154
- <td>62.6</td>
155
- <td>52.1</td>
156
- <td>1860.0</td>
157
- <td>60.6</td>
158
- <td>56.7</td>
159
- <td>37.0</td>
160
- <td>33.8</td>
161
- <td>67.7</td>
162
- <td>49.3</td>
163
- <td>56.2 / 80.0</td>
164
- </tr>
165
- <tr>
166
- <td nowrap="nowrap" align="left">DeepSeek-VL-7B</td>
167
- <td>7.3B</td>
168
- <td>435</td>
169
- <td>64.7*</td>
170
- <td>47.0*</td>
171
- <td>55.6</td>
172
- <td>1765.4</td>
173
- <td>74.1</td>
174
- <td>72.8</td>
175
- <td>38.3</td>
176
- <td>36.8</td>
177
- <td>77.8</td>
178
- <td>54.2</td>
179
- <td><strong>91.5 / 95.3</strong></td>
180
- </tr>
181
- <tr>
182
- <td nowrap="nowrap" align="left">Yi-VL-34B</td>
183
- <td>34B</td>
184
- <td>290</td>
185
- <td>43.4*</td>
186
- <td>16.9*</td>
187
- <td>52.6</td>
188
- <td>2050.2</td>
189
- <td>71.1</td>
190
- <td>71.4</td>
191
- <td>45.1</td>
192
- <td>30.7</td>
193
- <td>62.3</td>
194
- <td>54.8</td>
195
- <td>79.3 / 86.0</td>
196
- </tr>
197
- <tr>
198
- <td nowrap="nowrap" align="left">CogVLM-Chat</td>
199
- <td>17.4B</td>
200
- <td>590</td>
201
- <td>70.4</td>
202
- <td>33.3*</td>
203
- <td>52.5</td>
204
- <td>1736.6</td>
205
- <td>63.7</td>
206
- <td>53.8</td>
207
- <td>37.3</td>
208
- <td>34.7</td>
209
- <td>73.9</td>
210
- <td>60.3</td>
211
- <td>73.6 / 87.4</td>
212
- </tr>
213
- <tr>
214
- <td nowrap="nowrap" align="left">TextMonkey</td>
215
- <td>9.7B</td>
216
- <td>558</td>
217
- <td>64.3</td>
218
- <td>66.7</td>
219
- <td>-</td>
220
- <td>-</td>
221
- <td>-</td>
222
- <td>-</td>
223
- <td>-</td>
224
- <td>-</td>
225
- <td>-</td>
226
- <td>-</td>
227
- <td>-</td>
228
- </tr>
229
- <tr>
230
- <td nowrap="nowrap" align="left">IDEFICS2-8B</td>
231
- <td>8.0B</td>
232
- <td>-</td>
233
- <td>-</td>
234
- <td>-</td>
235
- <td>57.2</td>
236
- <td>1847.6</td>
237
- <td>75.7</td>
238
- <td>68.6</td>
239
- <td>45.2</td>
240
- <td>52.2</td>
241
- <td>49.1</td>
242
- <td>60.7</td>
243
- <td>-</td>
244
- </tr>
245
- <tr>
246
- <td nowrap="nowrap" align="left">Bunny-LLama-3-8B</td>
247
- <td>8.4B</td>
248
- <td>-</td>
249
- <td>-</td>
250
- <td>-</td>
251
- <td>54.3</td>
252
- <td>1920.3</td>
253
- <td>77.0</td>
254
- <td><strong>73.9</strong></td>
255
- <td>41.3</td>
256
- <td>31.5</td>
257
- <td>61.2</td>
258
- <td>58.8</td>
259
- <td>-</td>
260
- </tr>
261
- <tr>
262
- <td nowrap="nowrap" align="left">XTuner-Llama-3-8B-v1.1</td>
263
- <td>8.4B</td>
264
- <td>-</td>
265
- <td>-</td>
266
- <td>-</td>
267
- <td>53.3</td>
268
- <td>1818.0</td>
269
- <td>71.7</td>
270
- <td>63.2</td>
271
- <td>39.2</td>
272
- <td>40.0</td>
273
- <td>69.2</td>
274
- <td>-</td>
275
- <td>-</td>
276
- </tr>
277
- <tr>
278
- <td nowrap="nowrap" align="left">LLaVA-NeXT Llama-3-8B</td>
279
- <td>8.4B</td>
280
- <td>-</td>
281
- <td>-</td>
282
- <td>78.2</td>
283
- <td>-</td>
284
- <td>1971.5</td>
285
- <td>72.1</td>
286
- <td>-</td>
287
- <td>41.7</td>
288
- <td>37.5</td>
289
- <td>80.1</td>
290
- <td>60.0</td>
291
- <td>-</td>
292
- </tr>
293
- <tr style="background-color: #e6f2ff;">
294
- <td nowrap="nowrap" align="left">MiniCPM-V 1.0</td>
295
- <td>2.8B</td>
296
- <td>366</td>
297
- <td>60.6</td>
298
- <td>38.2</td>
299
- <td>47.6</td>
300
- <td>1650.2</td>
301
- <td>67.9</td>
302
- <td>65.3</td>
303
- <td>38.3</td>
304
- <td>28.9</td>
305
- <td>51.3</td>
306
- <td>51.2</td>
307
- <td>78.4 / 88.5</td>
308
- </tr>
309
- <tr style="background-color: #e6f2ff;">
310
- <td nowrap="nowrap" align="left">MiniCPM-V 2.0</td>
311
- <td>2.8B</td>
312
- <td>605</td>
313
- <td>74.1</td>
314
- <td>71.9</td>
315
- <td>55.0</td>
316
- <td>1808.6</td>
317
- <td>69.6</td>
318
- <td>68.1</td>
319
- <td>38.2</td>
320
- <td>38.7</td>
321
- <td>69.2</td>
322
- <td>55.8</td>
323
- <td>85.5 / 92.2</td>
324
- </tr>
325
- <tr style="background-color: #e6f2ff;">
326
- <td nowrap="nowrap" align="left">MiniCPM-Llama3-V 2.5</td>
327
- <td>8.5B</td>
328
- <td><strong>725</strong></td>
329
- <td><strong>76.6</strong></td>
330
- <td><strong>84.8</strong></td>
331
- <td><strong>65.0</strong></td>
332
- <td><strong>2024.6</strong></td>
333
- <td><strong>76.7</strong></td>
334
- <td><strong>73.4</strong></td>
335
- <td><strong>45.8</strong></td>
336
- <td><strong>54.3</strong></td>
337
- <td><strong>86.7</strong></td>
338
- <td><strong>63.5</strong></td>
339
- <td>89.7 / 95.0</td>
340
- </tr>
341
- </tbody>
342
- </table>
343
-
344
 
 
 
 
345
  </div>
346
- * We evaluate the officially released checkpoint by ourselves.
347
-
348
- </details>
349
 
350
  ### Examples <!-- omit in toc -->
351
 
@@ -463,4 +157,4 @@ If you find our work helpful, please consider citing the following papers
463
  journal={arXiv preprint arXiv:2403.11703},
464
  year={2024}
465
  }
466
- ```
 
33
  ### Evaluation <!-- omit in toc -->
34
 
35
  <div align="center">
36
+ <img src=assets/MiniCPM-Llama3-V-2.5-peformance.png width=80% />
37
  </div>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
38
 
39
+ Results on TextVQA, DocVQA, OCRBench, OpenCompass, MME, MMBench, MMMU, MathVista, LLaVA Bench, RealWorld QA, Object HalBench.
40
+ <div align="center">
41
+ <img src=assets/MiniCPM-Llama3-V-2.5-benchmark.png width=140% />
42
  </div>
 
 
 
43
 
44
  ### Examples <!-- omit in toc -->
45
 
 
157
  journal={arXiv preprint arXiv:2403.11703},
158
  year={2024}
159
  }
160
+ ```
assets/MiniCPM-Llama3-V-2.5-benchmark.png ADDED

Git LFS Details

  • SHA256: a786704b5c5ba7ffcedb8bb828f3de71f102121972dc9430f40006960edea379
  • Pointer size: 131 Bytes
  • Size of remote file: 408 kB