Update README.md
Browse files
README.md
CHANGED
|
@@ -1,35 +1,6 @@
|
|
| 1 |
-
##
|
| 2 |
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
### Low-level Question-Answering
|
| 6 |
-
|
| 7 |
-
This model has reached 75.90\%(*13\% better than previous version*)/76.52\%(*10\% better than previous version*) on Q-Bench A1 *dev/test* (multi-choice questions).
|
| 8 |
-
|
| 9 |
-
It also outperforms the following close-source models with much larger model capacities:
|
| 10 |
-
|
| 11 |
-
| Model | *dev* | *test* |
|
| 12 |
-
| ---- | ---- | ---- |
|
| 13 |
-
| **Co-Instruct-Preview (mPLUG-Owl2) (This Model)** | **75.90\%** | **76.52\%** |
|
| 14 |
-
| \*GPT-4V-Turbo | 74.41\% | 74.10\% |
|
| 15 |
-
| \*Qwen-VL-**Max** | 73.63\% | 73.90\% |
|
| 16 |
-
| \*GPT-4V (Nov. 2023) | 71.78\% | 73.44\% |
|
| 17 |
-
| \*Gemini-Pro | 68.16\% | 69.46\% |
|
| 18 |
-
| Q-Instruct (mPLUG-Owl2, Nov. 2023) | 67.42\% | 70.43\% |
|
| 19 |
-
| \*Qwen-VL-Plus | 66.01\% | 68.93\% |
|
| 20 |
-
| mPLUG-Owl2 | 62.14\% | 62.68\% |
|
| 21 |
-
|
| 22 |
-
\*: Proprietary Models.
|
| 23 |
-
|
| 24 |
-
#### Image/Video Quality Assessment
|
| 25 |
-
|
| 26 |
-
| Model | live | agi | livec | test_spaq | csiq | test_kadid | test_koniq | konvid | maxwell_test |
|
| 27 |
-
|--------------------------|--------------|--------------|-------------|-------------|-------------|-------------|-------------|-------------|--------------|
|
| 28 |
-
|**Co-Instruct-Preview (mPLUG-Owl2) (This Model)** | **0.803/0.756** | **0.719**/0.732 | **0.827/0.835** | **0.946/0.937** | **0.711/0.727** | **0.782/0.766** | 0.886/**0.935** | **0.818/0.790** | **0.735/0.714** |
|
| 29 |
-
| Q-Instruct (mPLUG-Owl2, Nov. 2023) | 0.749/0.747 | 0.710/**0.753** | 0.781/0.791 | 0.921/0.917 | 0.693/0.723 | 0.670/0.665 | **0.904**/0.921 | 0.766/0.738 | 0.650/0.649 |
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
We are also constructing multi-image benchmark sets (image pairs, triple-quadruple images), and the results on multi-image benchmarks will be released soon!
|
| 33 |
|
| 34 |
## Load Model
|
| 35 |
|
|
@@ -37,7 +8,7 @@ We are also constructing multi-image benchmark sets (image pairs, triple-quadrup
|
|
| 37 |
import torch
|
| 38 |
from transformers import AutoModelForCausalLM
|
| 39 |
|
| 40 |
-
model = AutoModelForCausalLM.from_pretrained("q-future/co-instruct
|
| 41 |
trust_remote_code=True,
|
| 42 |
torch_dtype=torch.float16,
|
| 43 |
attn_implementation="eager",
|
|
|
|
| 1 |
+
## News
|
| 2 |
|
| 3 |
+
A technical report for this model is coming soon.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4 |
|
| 5 |
## Load Model
|
| 6 |
|
|
|
|
| 8 |
import torch
|
| 9 |
from transformers import AutoModelForCausalLM
|
| 10 |
|
| 11 |
+
model = AutoModelForCausalLM.from_pretrained("q-future/co-instruct",
|
| 12 |
trust_remote_code=True,
|
| 13 |
torch_dtype=torch.float16,
|
| 14 |
attn_implementation="eager",
|