Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Unable to reproduce llava evaluation
Root cause: Environment Incompatibility

To Reproduce (on a linux machine with 1xA100 80GB):

1. I first create a new environment of python 3.10 with anaconda: 
> conda create --name lmms python=3.10
> conda activate lmms

2. Install lmms-eval from source (when in lmms-eval repo)
> pip install --no-deps -U -e .

3. Install llava requirements
> pip install -r llava_repr_requirements.txt
This Fails due to a conflict in transformer requirement: 

      ERROR: Cannot install -r llava_repr_requirements.txt (line 1) and transformers>=4.36.2 because these package versions have conflicting dependencies.
      
      The conflict is caused by:
          The user requested transformers>=4.36.2
          llava 1.1.3 depends on transformers==4.31.0

      To fix this you could try to:
      1. loosen the range of package versions you've specified
      2. remove package versions to allow pip attempt to solve the dependency conflict

      ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

4. I then tried manually installed transformers>=4.36.2 and retried install after removing it from llava_repr_requirements.txt
> pip install transformers
> pip install -r llava_repr_requirements.txt

5. Run main command
> accelerate launch --num_processes=1 -m lmms_eval --model llava   --model_args pretrained="liuhaotian/llava-v1.5-7b"   --tasks mme  --batch_size 1 --log_samples --log_samples_sufix reproduce --output_path ./logs/

Received error:
      `__main__.py: error: unrecognized arguments: --log_samples_sufix reproduce`

Removed unknown commands (log_samples_sufix)
> accelerate launch --num_processes=1 -m lmms_eval --model llava   --model_args pretrained="liuhaotian/llava-v1.5-7b"   --tasks mme  --batch_size 1 --log_samples --output_path ./logs/

Received ValueError:
      `ValueError: Attempted to load model 'llava', but no model for this name found! Supported model names: qwen_vl, gpt4V, instructblip, minicpm_v`



I've tried from many fresh installations. I'm am up to date with the main branch of lmms-eval (cloned it few hours ago). What should I do? 

Interesting, I observe different behavior for the log_samples_suffix reproduce (does not work) and log_samples_suffix llava_v1.5_mme_mmbenchen (I got to work)

In order to get it functioning, I also had to introduce two changes to the lmms-eval > lmms_eval > models > llava.py
1. Comment out "image_sizes=gen_kwargs["image_sizes"]," from generation args 
2. Skip padding tokens (-200)
        cont = cont.masked_fill(cont == -200, 1)
        result = self.tokenizer.batch_decode(cont, skip_special_tokens=True)
        text_outputs = [i.replace("
", "").replace("
", "").strip() for i in result]
4. Hard Code device and device_map
        self._device = torch.device("cuda:0")
        self.device_map = "cuda:0"

Would anyone be able to clarify the difference between reproduce and llava_v1.5_mme_mmbenchen? What benchmarks are supported by llava_v1.5_mme_mmbenchen?

Additionally, has anyone ever run into the errors I mentioned? Wondering why I was unable to run the code as is. 

I almost met the same series of error as @jacob-hansen. Hope can have a clean version for llava soon.

Hi, @jacob-hansen , @justlovebarbecue , thank you for spotting out the issue.

It seems like current `llava_requirement.txt` does have some issue. Here are my way to make it correct under current version.

---

## Solution

First cd into lmms-eval
```
pip install --no-deps -U -e .
```

Then cd into the LLaVA repo from https://github.com/haotian-liu/LLaVA?tab=readme-ov-file and do the same thing

```
pip install --no-deps -U -e .
```

This will build llava and lmms_eval without installing any dependency.

Then instead of using the current `llava_requirement.txt`, please use this
```
accelerate==0.21.0
datasets==2.16.1
evaluate==0.4.1
hf_transfer==0.1.6
Jinja2==3.1.3
numpy==1.26.4
openai==1.13.3
packaging==23.2
pandas==2.2.1
Pillow==10.2.0
protobuf==4.25.3
pycocoevalcap==1.2
pycocotools==2.0.7
pytablewriter==1.2.0
pytest==8.0.2
python_Levenshtein==0.25.0
pytz==2024.1
PyYAML==6.0.1
PyYAML==6.0.1
Requests==2.31.0
sacrebleu==2.4.0
scikit_learn==1.2.2
sentencepiece==0.1.99
setuptools==68.2.2
sglang==0.1.12
shortuuid==1.0.12
sqlitedict==2.1.0
tenacity==8.2.3
torch==2.0.1
tokenizers==0.15.2
tqdm==4.66.2
transformers==4.37.2
```
Save this into `llava_requirement.txt` and 

```
pip install -r llava_requirement.txt
```

The correct environment will be installed by this requirement file. Noted that this file is generated py `pipreqs` and possibly there are some redundant packages but it does work.


Then you can run
```
accelerate launch --num_processes=8 --main_process_port 12345 -m lmms_eval --model llava --model_args pretrained="liuhaotian/llava-v1.5-7b,use_flash_attention_2=False" --tasks mme --batch_size 1 --log_samples --log_samples_suffix llava_v1.5_mme --output_path ./logs/;
```
Make sure use_flash_attention_2 is set to False in model_args.


### Results

<img width="762" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/92624596/e5e20098-91db-48f1-89f6-644dea38b4be">

---

### Additional Note

If you want to use flash attention, you can install it by
```
pip install flash-attn --no-build-isolation --no-cache-dir
```

But noted that if you use flash-attn 2, there will be a slight different in the result. I got 
"mme_cognition_score": 355.7142857142857,
"mme_percetion_score": 1509.9907963185276,

Also, @jacob-hansen , there is one typo for the `log_samples_suffix ` in our example script, should be `log_samples_suffix ` instead of `log_samples_sufix`. 

When following your new protocol, I observe no differences than my work arounds for the installation. 
 
By specifying use_flash_attention_2=False, I no longer had to comment that part out from my code. 
 
But I still observe a gpu and required the following changes: 
> self.device_map = torch.device(device) # (line 74)

This might be specific to my system though

@jacob-hansen , you might also need to add `device_map=auto` or `device_map=cuda` when you use single device in the model args

This change has been merged into:

https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/24