Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Tasks were not found: activitynetqa
Root cause: Environment Incompatibility

2024-09-10 16:26:20.400 | WARNING  | __main__:cli_evaluate_single:234 -  --limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2024-09-10 16:26:20.401 | INFO     | __main__:cli_evaluate_single:269 - Evaluating on 1 tasks.
2024-09-10 16:26:20.401 | ERROR    | __main__:cli_evaluate_single:275 - Tasks were not found: activitynetqa. Try `lmms-eval --tasks list` for list of available tasks
2024-09-10 16:26:20.401 | INFO     | __main__:cli_evaluate_single:280 - Selected Tasks: []
Traceback (most recent call last):
  File "/home/jiangbaoyang/lmms-eval-main/lmms_eval/__main__.py", line 202, in cli_evaluate
    results, samples = cli_evaluate_single(args)
  File "/home/jiangbaoyang/lmms-eval-main/lmms_eval/__main__.py", line 298, in cli_evaluate_single
    results = evaluator.simple_evaluate(
  File "/home/jiangbaoyang/lmms-eval-main/lmms_eval/utils.py", line 434, in _wrapper
    return fn(*args, **kwargs)
  File "/home/jiangbaoyang/lmms-eval-main/lmms_eval/evaluator.py", line 86, in simple_evaluate
    assert tasks != [], "No tasks specified, or no tasks found. Please verify the task names."
AssertionError: No tasks specified, or no tasks found. Please verify the task names.
2024-09-10 16:26:20.568 | ERROR    | __main__:cli_evaluate:216 - Error during evaluation: No tasks specified, or no tasks found. Please verify the task names.
Traceback (most recent call last):
  File "/home/jiangbaoyang/lmms-eval-main/lmms_eval/__main__.py", line 202, in cli_evaluate
    results, samples = cli_evaluate_single(args)
  File "/home/jiangbaoyang/lmms-eval-main/lmms_eval/__main__.py", line 298, in cli_evaluate_single
    results = evaluator.simple_evaluate(
  File "/home/jiangbaoyang/lmms-eval-main/lmms_eval/utils.py", line 434, in _wrapper
    return fn(*args, **kwargs)
  File "/home/jiangbaoyang/lmms-eval-main/lmms_eval/evaluator.py", line 86, in simple_evaluate
    assert tasks != [], "No tasks specified, or no tasks found. Please verify the task names."
AssertionError: No tasks specified, or no tasks found. Please verify the task names.

why？

I found that this dataset only supports single gpu testing...

Hi @jby20180901 , when a task is not found in TASK_REGISTRY, it may be caused by two reasons. First, it might because of that the config is wrongly written. Second, it might because some dependencies are missing or init lines for the task have errors that causing the task not being imported. In this setting, the error are most likely caused by the second case. You might wanna check whether you have `HF_HOME` set in your envs because we have to pick a folder to cache the unzipped videos and setting this env var explicitly is what we recommended.

To check the error messages, you can add `--verbosity DEBUG` to see the exact error messages otherwise we simply skip register this task and keep going for the evaluation.

Hi @YangYangGirl , I don't see any issue when trying to evaluate on multi-gpu, may I ask your settings?

<img width="1791" alt="image" src="https://github.com/user-attachments/assets/89cd6226-cb5e-4772-8c53-9e8aefc9ad2a">


accelerate launch --num_processes=8  --main_process_port 22345 \
    -m lmms_eval \
    --model llava_onevision \
    --model_args pretrained=lmms-lab/llava-onevision-qwen2-0.5b-ov,conv_template=qwen_1_5,video_decode_backend=decord,model_name=llava_qwen,max_frames_num=32 \
    --tasks activitynetqa \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_onevision \
    --output_path ./logs/
![image](https://github.com/user-attachments/assets/71a8ada9-587f-41f3-b020-ea2053154e86)


when I try to evaluate the model on videomme dataset using the command below, it works:
    accelerate launch --num_processes=8  --main_process_port 22345 \
    -m lmms_eval \
    --model llava_onevision \
    --model_args pretrained=lmms-lab/llava-onevision-qwen2-0.5b-ov,conv_template=qwen_1_5,video_decode_backend=decord,model_name=llava_qwen,max_frames_num=32 \
    --tasks videomme \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_onevision \
    --output_path ./logs/
![image](https://github.com/user-attachments/assets/145339d8-d152-406e-b6e3-29ab9fd5e336)



Thanks for the kind reply. But there seems to be some problems with the integration of activityqa dataset : )

> 你好@jby20180901，当一个任务在 TASK_REGISTRY 中找不到时，可能有两个原因。首先，可能是因为配置写错了。其次，可能是因为缺少一些依赖项或任务的 init 行有错误导致任务无法导入。在这个设置中，错误很可能是由第二种情况引起的。你可能要检查一下你`HF_HOME`的 envs 中是否已经设置了，因为我们必须选择一个文件夹来缓存解压后的视频，我们建议明确设置这个环境变量。
> 
> 要检查错误消息，您可以添加`--verbosity DEBUG`以查看确切的错误消息，否则我们只需跳过注册此任务并继续进行评估。

我找到问题所在了，activitynetqa的util.py里面的hf_home文件夹不存在

@YangYangGirl  Hi, my commands are similar to you but I can't reproduce the result on my side
```bash
TASK=$1
CKPT_PATH=$2

echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

accelerate launch --num_processes 8 --main_process_port 12345 -m lmms_eval \
    --model llava_onevision \
    --model_args pretrained=$CKPT_PATH,conv_template=qwen_1_5,model_name=llava_qwen,mm_spatial_pool_mode=bilinear \
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix ${TASK_SUFFIX} \
    --output_path ./logs/
```

I checked again on current main branch that activitynetqa can work with llava onevision
<img width="1792" alt="image" src="https://github.com/user-attachments/assets/4f513a1f-1646-4aa2-9ed3-d52b985f2368">

You might wanna check your environment with the accelerate or there might be other errors occur during loading the files


I delete the activityqa dataset and re-download it, then it works! Thanks for your patience.

Does the evaluation of this activitynetqa require gpt calling? It uses traditional calculation-based metrics in the paper but when running in lmms-eval, it keeps throwing gpt calling errors (yeah I didn't set the openai api key) and ends with no results.
![Screenshot 2024-11-06 at 10 40 46](https://github.com/user-attachments/assets/ab1f5ff9-3ac2-4340-983d-b76ca8fc36a6)


@kcz358 Hi Kaichen,

Is the following warning affect the overall performance:

```
[h264 @ 0x54f683c0] mmco: unref short failure
[h264 @ 0x54f683c0] mmco: unref short failure
[h264 @ 0x54f683c0] mmco: unref short failure
[h264 @ 0x48052e00] mmco: unref short failure
[h264 @ 0x48052e00] mmco: unref short failure
[h264 @ 0x48052e00] mmco: unref short failure
[h264 @ 0x48052e00] mmco: unref short failure
[h264 @ 0x48052e00] mmco: unref short failure
[h264 @ 0x554cfb80] mmco: unref short failure
[h264 @ 0x58117380] mmco: unref short failure
[h264 @ 0x56fd22c0] mmco: unref short failure
[h264 @ 0x56fd22c0] mmco: unref short failure
```

How can we resolve this warning?

@liyucheng09 Hi, this warning is related to decord and would not affect the performance. When switching to pyav, this warning should disappear but the overall performance should be similar

> Does the evaluation of this activitynetqa require gpt calling? It uses traditional calculation-based metrics in the paper but when running in lmms-eval, it keeps throwing gpt calling errors (yeah I didn't set the openai api key) and ends with no results.此活动网问答评估是否需要 GPT 调用？论文中使用了传统的基于计算的指标，但在 lmms-eval 运行时，不断抛出 GPT 调用错误（是的，我没有设置 OpenAI API 密钥）并且最终没有结果。 ![Screenshot 2024-11-06 at 10 40 46](https://github.com/user-attachments/assets/ab1f5ff9-3ac2-4340-983d-b76ca8fc36a6)

The same problem

> [@liyucheng09](https://github.com/liyucheng09) Hi, this warning is related to decord and would not affect the performance. When switching to pyav, this warning should disappear but the overall performance should be similar

Excuse me, could you please tell me that how can i evaluate activitynetqa with loacl data? It will be very generous if you have time to reply my naive question.