Repository: open-compass/opencompass
Issue: [Feature]  Unsupported Model Type in vLLM/LMDeploy acceleration
Root cause: Unimplemented Feature Gap

### Describe the feature

I tried the vLLM and LMDeploy using the following command:
```
python run.py \
    --datasets humaneval_gen \
    --hf-type chat \
    --hf-path meta-llama/Meta-Llama-3-8B-Instruct \
    -a vllm
```
and
```
python run.py \
    --datasets humaneval_gen \
    --hf-type chat \
    --hf-path meta-llama/Meta-Llama-3-8B-Instruct \
    -a lmdeploy
```
Then I notice this WARNING at the console:
```
08/15 12:23:50 - OpenCompass - INFO - Loading mmlu_gen: configs/datasets/mmlu/mmlu_gen.py
08/15 12:23:50 - OpenCompass - INFO - Transforming _hf to lmdeploy (or vllm)
08/15 12:23:50 - OpenCompass - WARNING - Unsupported model type opencompass.models.huggingface_above_v4_33.HuggingFacewithChatTemplate, will keep the original model.
```
Seems the Meta-Llama-3-8B-Instruct is not supported for the acceleration load failed. But I noticed there's a model config [`hf_llama/lmdeploy_llama3_8b_instruct`](https://github.com/open-compass/opencompass/blob/main/configs/models/hf_llama/lmdeploy_llama3_8b_instruct.py) with prefix **lmdeploy**, so I also tried:
```
python run.py \
    --datasets humaneval_gen \
    --hf-type chat \
    --models lmdeploy_llama3_8b_instruct \
    -a lmdeploy
```
which also leads to the WARNING:
```
08/15 12:26:40 - OpenCompass - INFO - Loading lmdeploy_llama3_8b_instruct: configs/models/hf_llama/lmdeploy_llama3_8b_instruct.py
08/15 12:26:40 - OpenCompass - INFO - Transforming llama-3-8b-instruct-turbomind to lmdeploy
08/15 12:26:40 - OpenCompass - WARNING - Unsupported model type <class 'opencompass.models.turbomind_with_tf_above_v4_33.TurboMindModelwithChatTemplate'>, will keep the original model.
```
So I'm looking for help with using the acceleration framework on Opencompass evaluation:

* Do any misoperation I made in my commands? Is there any solution for supporting the model `Meta-Llama-3-8B-Instruct` for acceleration?
* For my private LLMs (maybe with unique architecture), how can I accelerate the evaluation with vLLM and LMDeploy?

I checked the intro [here](https://github.com/open-compass/opencompass/blob/main/docs/en/advanced_guides/accelerator_intro.md) but it doesn't help.

If we need a new issue for this question, please let me know.

Thanks for your help.

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

It seems to work by modifying L158 the following line at [`opencompass/utils/run.py`](https://github.com/open-compass/opencompass/blob/main/opencompass/utils/run.py):
```python
model = dict(type=f'{mod.__module__}.{mod.__name__}',
```
into:
```python
model = dict(type=mod,
```

Fixed in https://github.com/open-compass/opencompass/pull/1430

Thanks.

I am reopening this issue because the WARNING occurred again while I was evaluating the base model `deepseek-7b-base-hf`:
```
09/19 15:40:28 - OpenCompass - INFO - Transforming deepseek-7b-base-hf to vllm
09/19 15:40:28 - OpenCompass - WARNING - Unsupported model type <class 'opencompass.models.huggingface_above_v4_33.HuggingFaceBaseModel'>, will keep the original model.
```

I used the following command:
```
CUDA_VISIBLE_DEVICES=0 python run.py --models hf_deepseek_7b_base --datasets gsm8k_gen -a vllm
```
Thanks for your help. @liushz 

I am facing the same issue. Please help.