Repository: EvolvingLMMs-Lab/lmms-eval
Issue: TypeError: 'NoneType' object is not subscriptable
Root cause: Validation Gap

lmms_eval/api/task.py", line 993, in doc_to_target
    return doc[doc_to_target]
TypeError: 'NoneType' object is not subscriptable


```bash
#!/bin/bash
export PYTHONPATH=$PYTHONPATH:/mnt2/***/train_code/LLaVA
# export OPENAI_API_URL="https://api.openai.com/v1/chat/completions"
# export OPENAI_API_KEY="YOUR_API_KEY"
# export API_TYPE="openai"
# export AZURE_ENDPOINT="https://api.cognitive.microsoft.com/sts/v1.0/issueToken"
# export AZURE_API_KEY="YOUR_API_KEY"
# export GOOGLE_API_KEY="xxx"
# export ANTHROPIC_API_URL="https://api.anthropic.com/v1/complete"
# export ANTHROPIC_API_KEY="YOUR_API_KEY"
export HF_HOME="~/.cache/huggingface/"

model_path="/mnt2/***/train_code/LLaVA/checkpoint_model/liuhaotian/llava-v1.5-7b"
model_name='llava-v1.5-7b'
conv_template='vicuna_v1'
run_port=12345

export HF_ENDPOINT=https://hf-mirror.com

#huggingface-cli login


if [ ! -d "eval_log" ];then
    mkdir eval_log
else
    echo "文件夹已经存在"
fi

accelerate launch \
	--main_process_port=$run_port \
    --num_processes=2 \
    -m lmms_eval \
    --model llava \
    --model_args pretrained=$model_path,conv_template=$conv_template \
    --tasks mmbench,scienceqa_img \
    --batch_size 1 \
    --log_samples \
    --verbosity INFO \
    --show_task_to_terminal \
    --log_samples_suffix $model_name \
    --output_path ./eval_log/


```

```
        if show_task_to_terminal:
            for inst in task.instances:
                # print the prompt for the first few documents
                if inst.doc_id < 1:
                    eval_logger.info(
                        f"Task: {task_name}; document {inst.doc_id}; context prompt (starting on next line):\
\n{inst.args[0]}\n(end of prompt on previous line)\ntarget string or answer choice index (starting on next line):\n{task.doc_to_target(inst.doc)}\n(end of target on previous line)"
                    )
                    eval_logger.info(f"Request: {str(inst)}")
```

I also encountered this error using the latest version of LMMS-Eval. How can I solve it?

The log is:

```bash
 File "/root/miniconda3/envs/psp-lmms-eval/lib/python3.10/runpy.py", line 196, in _run_module_as_main    [0/7778]
    return _run_code(code, main_globals, None,
  File "/root/miniconda3/envs/psp-lmms-eval/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/root/opensource/lmms-eval/lmms_eval/__main__.py", line 539, in <module>
    cli_evaluate()
  File "/root/opensource/lmms-eval/lmms_eval/__main__.py", line 366, in cli_evaluate
    raise e
  File "/root/opensource/lmms-eval/lmms_eval/__main__.py", line 347, in cli_evaluate
    results, samples = cli_evaluate_single(args)
  File "/root/opensource/lmms-eval/lmms_eval/__main__.py", line 474, in cli_evaluate_
single
    results = evaluator.simple_evaluate(
  File "/root/opensource/lmms-eval/lmms_eval/utils.py", line 536, in _wrapper
    return fn(*args, **kwargs)
  File "/root/opensource/lmms-eval/lmms_eval/evaluator.py", line 268, in simple_evalu
ate
    results = evaluate(
  File "/root/opensource/lmms-eval/lmms_eval/utils.py", line 536, in _wrapper
    return fn(*args, **kwargs)
  File "/root/opensource/lmms-eval/lmms_eval/evaluator.py", line 462, in evaluate
    print_writeout(task)
  File "/root/opensource/lmms-eval/lmms_eval/evaluator_utils.py", line 185, in print_
writeout
    \n{inst.args[0]}\n(end of prompt on previous line)\ntarget string or answer choice index (starting on next line):\n{task.doc_
to_target(inst.doc)}\n(end of target on previous line)"
  File "/root/opensource/lmms-eval/lmms_eval/api/task.py", line 1347, in doc_to_targe
t
    return doc[doc_to_target]
TypeError: 'NoneType' object is not subscriptable
```

The script I run is:

```bash
# 环境变量设置
export HF_HOME="$LMMS_EVAL_DATASET_CACHE"
export VLLM_CACHE_ROOT="$VLLM_CACHE_ROOT"
export VLLM_WORKER_MULTIPROC_METHOD="spawn"
export HF_TOKEN="$HF_TOKEN"
CUDA_VISIBLE_DEVICES=$GPU_LIST python -m lmms_eval \
    --model "$MODEL_CLASS" \
    --model_args "model_version=${MODEL_DIR},tensor_parallel_size=${TENSOR_PARALLEL_SIZE},\
        trust_remote_code=True,max_images=2,prompt=tool_call,enable_tool_call=True,\
        downsample_image=True,max_token=40960" \
    --tasks "${TASKS}" \
    --batch_size 1024 \
    --log_samples \
    --log_samples_suffix "$LOG_SAMPLES_SUFFIX" \
    --use_cache "$CACHE_PATH" \
    --cache_requests "true" \
    --output_path "$OUTPUT_PATH" \
    --verbosity DEBUG \
    --write_out \
    --seed 42
```

It was fine the first time, but I encountered an error afterward.

May I ask which task you are running?

> May I ask which task you are running?

TASKS="mmbench_en_test,mmmu_val,pope,mme,mathvista_testmini,mathverse_testmini_vision_only,mmvet"

I haven't tested which task causes this error yet.

Disable the write out tag should fix the error. Write out is currently not maintained because it is kind of overlapping with the log_samples.

Hi @pspdada,

I've created PR #839 to fix the exact error you're experiencing.

The PR adds a null check in evaluator_utils.py to handle cases where inst.doc is None:

> Before (line 185, causes crash):
task.doc_to_target(inst.doc)

> After (handles None gracefully):
target = "N/A (document is None)" if inst.doc is None else task.doc_to_target(inst.doc)

Could you try to switch to that branch to test and give us a feedback?