Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Module 'exact_match' doesn't exist on the Hugging Face Hub either.
Root cause: External Dependency Breakage

install:
pip install lmms-eval

run:
accelerate launch --num_processes=8 -m lmms_eval --model llava   --model_args pretrained="liuhaotian/llava-v1.5-7b"   --tasks mme  --batch_size 1 --log_samples --log_samples_suffix llava_v1.5_mme --output_path ./logs/

bug:
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.10/dist-packages/lmms_eval/__main__.py", line 24, in <module>
    from lmms_eval import evaluator, utils
  File "/usr/local/lib/python3.10/dist-packages/lmms_eval/evaluator.py", line 17, in <module>
    import lmms_eval.api.metrics
  File "/usr/local/lib/python3.10/dist-packages/lmms_eval/api/metrics.py", line 146, in <module>
    exact_match = evaluate.load("exact_match")
  File "/usr/local/lib/python3.10/dist-packages/evaluate/loading.py", line 748, in load
    evaluation_module = evaluation_module_factory(
  File "/usr/local/lib/python3.10/dist-packages/evaluate/loading.py", line 681, in evaluation_module_factory
    raise FileNotFoundError(
FileNotFoundError: Couldn't find a module script at /exact_match/exact_match.py. Module 'exact_match' doesn't exist on the Hugging Face Hub either.


Hi, may I asked have you installed llava from github also? I did not encounter any issue using the same command. You may want to check if flash-attn and llava are both installed.

> Hi, may I asked have you installed llava from github also? I did not encounter any issue using the same command. You may want to check if flash-attn and llava are both installed.

yeah，pip install package or pip install -e . Tried both installation methods
flash-attn and llava are both installed.
What else do I need to check?
![image](https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/38122102/b12a77a3-d4ad-4d45-81a9-887b1334d77d)


Hmm... , This is interesting. Mine works perfectly fine under the same environment.

Can you provide the whole running log of your program? @xienan0326 

Maybe need to see where the error occurs. Is it before model responding or postprocessing.

Same error

/exact_match/exact_match.py. Module 'exact_match' doesn't exist on the Hugging Face Hub either.


I would suggest try test your script on more robust envirment, this is hard to debug, it evolves logic inside.

> Same error
> 
> /exact_match/exact_match.py. Module 'exact_match' doesn't exist on the Hugging Face Hub either.
> 
> I would suggest try test your script on more robust envirment, this is hard to debug, it evolves logic inside.
手动配置下可以，但是又遇到了如何加载本地数据的问题
https://huggingface.co/spaces/evaluate-metric/exact_match  

请问下，我这边也是遇到了同类问题，该如何解决，有解决方案吗


我也是同样的问题

Hi @zyp2 @liguopeng0923 , you can refer to #75 , hope it would help

Hi @kcz358 ,

I have solved it by installing the offline `extract_math` in `llms-eval`.

But I met new issues
<img width="605" alt="image" src="https://github.com/user-attachments/assets/2cae80c5-6556-4e56-8eb0-2f603073750f">



@liguopeng0923 , I think it is just nltk download issue for nextqa. I think the evaluation will still work if you are not evaluating on nextqa

Thanks @kcz358 , It may be solved by setting the offline path of datasets like https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/23

Hi @kcz358 ,

Actually, I find some other datasets also have this problem, such as `detailcaps`.

Why we need to load all `yaml` files if we only evaluate on one dataset.

Hi @liguopeng0923 , because we need to initialize all the tasks and then select the task you want to evaluate from the `TASK_REGISTRY`. If you don't want to see the error message, you can simply delete the folder or the yaml file if you won't need this task.

Thanks. But I find the reason is the following code is not robust. For example, my machine cannot download packages from the internet and I download the offline package. But the following code still requests the internet.
<img width="903" alt="image" src="https://github.com/user-attachments/assets/cd4106a2-4c33-4b62-9a3c-d77b74a698c6">