Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Can't find llava_onevision model
Root cause: Validation Gap

I have conducted ```pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git``` and successfully installed lmms-eval=0.2.1. 
But i got the following error:
```ValueError: Attempted to load model 'llava_onevision', but no model for this name found! Supported model names: batch_gpt4, claude, from_log, fuyu, gemini_api, gpt4v, instructblip, internvl, internvl2, llama_vid, llava, llava_hf, llava_sglang, longva, mantis, minicpm_v, phi3v, qwen_vl, qwen-vl-api, reka, srt_api, tinyllava, xcomposer2_4khd, xcomposer2d5```
It can't find `llava_onevision` model.

Hi, can you try to install [llava](https://github.com/LLaVA-VL/LLaVA-NeXT/tree/main) first and see whether the error occur again?

Yes, i have installed llava-1.7.0-dev0 and install lmms-eval=0.2.1, and still see the problem. Here shows the information by conducting `pip list`.
![image](https://github.com/user-attachments/assets/d12084a1-7e74-41f0-a818-68afc95b2c86)


Can you try 
`from lmms_eval.models.llava_onevision import LlavaOneVision` 
and see what import error occurs?

If it is not in the models folder, then you might need to build from source

Thank you for your respones. The following error happens. 
![image](https://github.com/user-attachments/assets/70c128df-af76-45fa-90e9-89da51d98ac1)
llms-eval requires transformers=4.39.2, which already contains Qwen2moe_config, but it reports the error that can't import  Qwen2moe_config.
It seems that i need to install a higher version of transformer and reinstalll lmms-eval from source.

I have resintalled llava and lmms-eval from source, and updated transformers to 4.44.0, and still see the following error,
![image](https://github.com/user-attachments/assets/56262165-19d8-4dd4-a461-98915c41a2c5)
![image](https://github.com/user-attachments/assets/2c7ff3f2-7d34-4c91-a753-a75976a35a0b)


This is how I prepare envs

```bash
cd /path/to/lmms-eval
python3 -m pip install -e .;

cd /path/to/LLaVA-NeXT;
python3 -m pip install -e ".[train]";

# This could be optional
python3 -m pip install httpx==0.23.3 protobuf==3.20;
```

Check that this could run llava onevision model with current `lmms-eval` and `LLaVA-NeXT` code. Can you try pull again both repo and install again?

If you are encountering this error

> Thank you for your respones. The following error happens. ![image](https://private-user-images.githubusercontent.com/29403866/356798782-70c128df-af76-45fa-90e9-89da51d98ac1.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjMyOTkyNzIsIm5iZiI6MTcyMzI5ODk3MiwicGF0aCI6Ii8yOTQwMzg2Ni8zNTY3OTg3ODItNzBjMTI4ZGYtYWY3Ni00NWZhLTkwZTktODlkYTUxZDk4YWMxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODEwVDE0MDkzMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTQ4MWM3NDhhZDllYzNjNzQ2ZGIzOGY4MWZjOTcyMmM5YjU4OWRkMDUxMTJjMTFiNmYyMzg3ZmE2MjM0ZGY3NDkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.ooBGslchEfO9p4y5E318etsDYeCpc-GMbFdaGh6R--M) llms-eval requires transformers=4.39.2, which already contains Qwen2moe_config, but it reports the error that can't import Qwen2moe_config. It seems that i need to install a higher version of transformer and reinstalll lmms-eval from source.

Please go to hf and get access from llama. You might to export your huggingface token also.

Thanks. I have repulled the code and reinstalled the environment following your instructions. When i directly try `from lmms_eval.models.llava_onevision import LlavaOneVision, the llava_onevision model seems to exist, and it requests me to provide a huggingface token. But when i try evaluating the llava_onevision model following the [instructions](https://github.com/LLaVA-VL/LLaVA-NeXT/blob/main/docs/LLaVA_OneVision.md), it shows the following errors. 
![image](https://github.com/user-attachments/assets/5b913204-22b7-4ff3-87dc-3a5f2a52ae6d)
It seems the llava_onevision model is not registered in the repo.

Please make sure that `from lmms_eval.models.llava_onevision import LlavaOneVision` can be executed without error, otherwise this model will be skipped and will not add to the registry. `export HF_TOKEN=xxx` can export your hf tokens in to the envs. Make sure you have been granted access to gated models

Thanks for your response. I have exported hugging face tokens. Directly importing LlavaOneVision by conducting `from lmms_eval.models.llava_onevision import LlavaOneVision` raises the following error.
![image](https://github.com/user-attachments/assets/c969a01c-34fc-4e96-8902-001ab73551da)
Llava and lmms-eval are built from source with the following versions.
![image](https://github.com/user-attachments/assets/92e3e2fb-ab7d-4691-b07a-9b9cea942fc0)


Hi, I have noticed that there is a typo in the init file. Can you repull the repo and try again?

Thanks, the model can be loaded by conducting `from lmms_eval.models.llava_onevision import Llava_OneVision`.