Repository: EvolvingLMMs-Lab/lmms-eval
Issue: qwenvl-7b evaluate refcoco|+|g cider and IOU are all None,
Root cause: Validation Gap

accelerate launch --main_process_port=29501 --num_processes=8 -m lmms_eval --model qwen_vl --model_args pretrained=/Qwen-VL/ --tasks refcoco,refcoco+,refcocog,refcoco_bbox_rec,refcoco+_bbox_rec,refcocog_bbox_rec --batch_size 1 --log_samples --log_samples_suffix qwenvl --output_path ./logs/
### qwenvl-7b evaluate refcoco|+|g cider and IOU are all None
![image](https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/43173636/aa1380b8-6b82-493e-992c-4464e89d7eff)

can you help to solving this problem?

Thank you for your contributions!


@Luodian 

please check,thanks~@kcz358 

Can you check the logs you can see output samples and I guess it's because the model response doesn't match the groud-truth?

You could use `--log_samples --log_samples_suffix=debug --output_path=./logs/ --verbosity=DEBUG` to trigger the log mode.

> Can you check the logs you can see output samples and I guess it's because the model response doesn't match the groud-truth?
> 
> You could use `--log_samples --log_samples_suffix=debug --output_path=./logs/ --verbosity=DEBUG` to trigger the log mode.

LLava1.5 model evaluate to get right result, similar to paper, I think it it may not problem of the groud-truth. do you have evaluated the qwenvl-7B model  for refcoco|+|g Rec tasks or cider task? Is it consistent with the Qwenvl paper?
![image](https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/43173636/1157a03a-89e9-4580-bcee-30743764bd12)


We didnt match it with original paper on refcoco task. But we did match it with other task like AI2D, so I guess the model implementation is correct, but the prompting strategy is little different.

> We didnt match it with original paper on refcoco task. But we did match it with other task like AI2D, so I guess the model implementation is correct, but the prompting strategy is little different.

thansk for your reply. 
1）If i want to match qwenvl with original paper on refcoco task, where and what should i change the prompting strategy to achieve results similar Qwenvl paper?
2）Are there other models like fuyu ect (except llava model) that have tested the refcoco task?


1) I am not pretty much sure about the prompting strategy of Qwen-VL on this task.
2) We also did not test `fuyu` since of the prompting issue. Our most tested models are llava, intern-vl, and other commercial mdoels (GPT4V, Gemini, Claude etc) to show their best performance so that our evaluation toolkit can be used as standard to compare open models with commercial models.

> 1. I am not pretty much sure about the prompting strategy of Qwen-VL on this task.
> 2. We also did not test `fuyu` since of the prompting issue. Our most tested models are llava, intern-vl, and other commercial mdoels (GPT4V, Gemini, Claude etc) to show their best performance so that our evaluation toolkit can be used as standard to compare open models with commercial models.

can you offer your llava1.5 rec task results in refcoco|+|g, this is our official llava1.5 evaluated result
![image](https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/43173636/9e1cce6f-ddf5-45b2-9e17-6b6ad6804362)


Hi @AderonHuang, here are the logs we run for refcoco+ using llava 1.5 7b. But we evaluate this quite a while ago.

[refcoco+_bbox_testA.json](https://github.com/user-attachments/files/15738971/refcoco%2B_bbox_testA.json)

You can see the full score here

[results (1).json](https://github.com/user-attachments/files/15738987/results.1.json)



> Hi @AderonHuang, here are the logs we run for refcoco+ using llava 1.5 7b. But we evaluate this quite a while ago.
> 
> [refcoco+_bbox_testA.json](https://github.com/user-attachments/files/15738971/refcoco%2B_bbox_testA.json)
> 
> You can see the full score here
> 
> [results (1).json](https://github.com/user-attachments/files/15738987/results.1.json)

thanks for your brilliant work. I had solved my problem. 

> Hi @AderonHuang, here are the logs we run for refcoco+ using llava 1.5 7b. But we evaluate this quite a while ago.
> 
> [refcoco+_bbox_testA.json](https://github.com/user-attachments/files/15738971/refcoco%2B_bbox_testA.json)
> 
> You can see the full score here
> 
> [results (1).json](https://github.com/user-attachments/files/15738987/results.1.json)

hi,I also hava another question, refcoco|+|g evaluation why cannot use batch size>1 hava inference, like this every time:
accelerate launch --main_process_port=29501 --num_processes=8 -m lmms_eval --model llava --model_args pretrained=/path,use_flash_attention_2=False,device_map="" --tasks refcoco,refcoco_bbox_rec --batch_size 1 --log_samples 
if I want to achieve batch size >1 , what should I change?

Hi @AderonHuang , Batch size > 1 is not supported for original llava. If you want to evaluate llava using batch size > 1, you might want to take a look into [`llava_sglang`](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/models/llava_sglang.py)

> Hi @AderonHuang , Batch size > 1 is not supported for original llava. If you want to evaluate llava using batch size > 1, you might want to take a look into [`llava_sglang`](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/models/llava_sglang.py)

However, llava_sglang seems not surport multi-processes yet, some log like this, is right?
accelerate launch --main_process_port=29501 --num_processes=8 -m lmms_eval --model llava_sglang --model_args pretrained=/path,use_flash_attention_2=False,device_map="" --tasks refcoco,refcoco_bbox_rec --batch_size 1 --log_samples
"assert accelerator.num_processes == 1, "Llava-sglang does not support multi-processes yet"

Yes, multi process is not supported by sglang, but you can use tensor parallel to shard model into multiple gpus. This allow you to run batch sizes larger than gpu numbers and accelerate the evaluation pipeline. You might want to take a lot into this #54 for more details

> Hi @AderonHuang, here are the logs we run for refcoco+ using llava 1.5 7b. But we evaluate this quite a while ago.
> 
> [refcoco+_bbox_testA.json](https://github.com/user-attachments/files/15738971/refcoco%2B_bbox_testA.json)
> 
> You can see the full score here
> 
> [results (1).json](https://github.com/user-attachments/files/15738987/results.1.json)

Thank you very much for your excellent work.
Could you please provide the result json of llava-13b? The refcoco|+|g result provided in the library does not have the form of iou.
Looking forward to your response.

> > 1. I am not pretty much sure about the prompting strategy of Qwen-VL on this task.
> > 2. We also did not test `fuyu` since of the prompting issue. Our most tested models are llava, intern-vl, and other commercial mdoels (GPT4V, Gemini, Claude etc) to show their best performance so that our evaluation toolkit can be used as standard to compare open models with commercial models.
> 
> can you offer your llava1.5 rec task results in refcoco|+|g, this is our official llava1.5 evaluated result ![image](https://private-user-images.githubusercontent.com/43173636/336830443-9e1cce6f-ddf5-45b2-9e17-6b6ad6804362.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM1MDM2MjMsIm5iZiI6MTcyMzUwMzMyMywicGF0aCI6Ii80MzE3MzYzNi8zMzY4MzA0NDMtOWUxY2NlNmYtZGRmNS00NWIyLTllMTctNmI2YWQ2ODA0MzYyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODEyVDIyNTUyM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWYwYjcwNTY3MmJjZTAzZDA2Y2YzNDNiMTM2ZTMxM2VmMDMzY2FiODBhYjg0YjdiZWVhNjk3MWQyNDdjMmIwYzMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.fxo2iv3dH7Rsu9wkz5TyWXWpyauUayAKCK_b0gEBo1Q)

Hi. Did you implement the REC-based RefCOCO|+|g evaluation yourself? I also have this need now. Could you please share how to implement it?
Looking forward to your reply.

The problem is still there.
I tested Qwen/Qwen2-VL-7B-Instruct and OpenGVLab/InternVL2-8B using  refcoco_bbox_rec_test for visual grounding performance. Results are the same, very low.
![image](https://github.com/user-attachments/assets/022fe2c4-00e1-479e-b456-d09ba7142feb)


Qwen2-VL does have a special prompting strategy according to their paper. But the metrics are too low. I'm not sure where the `bugs` come from.

Okay, I think I find the problem. It seems some models fail to follow the instructions to generate bounding box like "[num,num,num,num]", so the util function(utils_rec.parse_float_sequence_within) always return [0,0,0,0]. And also models may fail to output numbers that are scaled to [0,1].
You can check the field "filtered_resps" in the jsonline-logs.
So generally there are at least two problems:
1. parse function fails. bad instruction or model fails to follow..
2. numbers are not scaled. models' bad
 
I modified two these two functions:
```python
def parse_float_sequence_within(input_str):
    """
    Extract the first sequence of four floating-point numbers within square brackets from a string.

    Args:
    input_str (str): A string that may contain a sequence of four floats within square brackets.

    Returns:
    list: A list of four floats if the pattern is found, or a list of four zeros if the pattern is not found.
    """
    # Define the regex pattern to find the first instance of four floats within square brackets
    # TODO: add more patterns to support various formats
    # pattern1 [num, num, num, num]
    pattern = r"\[\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?)\s*\]"

    # Use re.search to find the first match of the pattern in the input string
    match = re.search(pattern, input_str)

    # If a match is found, convert the captured groups into a list of floats
    if match:
        return [float(match.group(i)) for i in range(1, 5)]
    # pattern2 (num, num, num, num)
    pattern = r"\(\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?)\s*\)"
    match = re.search(pattern, input_str)
    if match:
        return [float(match.group(i)) for i in range(1, 5)]
    # pattern3 (num, num), (num, num)
    pattern = r"\(\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?)\),\s*\(\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?)\)"
    match = re.search(pattern, input_str)
    if match:
        return [float(match.group(i)) for i in range(1, 5)]
    # If the input does not contain the pattern, return the null float sequence
    return [0, 0, 0, 0]


def refcoco_bbox_rec_process_result(doc, result):
    """
    Args:
        doc: a instance of the eval dataset
        results: [pred]
    Returns:
        a dictionary with key: metric name, value: metric value
    """
    pred = result[0] if len(result) > 0 else ""
    pred = parse_float_sequence_within(pred)
    # TODO: answer may not be scaled to [0,1]
    # TODO: 输出的 像素位置是否和原始图像大小无关
    if any([x < 0 - 1e-5 or x > 1 + 1e-5 for x in pred]):
        pred = [
            x / doc["image_width"] if i % 2 == 0 else x / doc["image_height"]
            for i, x in enumerate(pred)
        ]
    ann_id = doc["question_id"]
    data_dict = {
        "answer": doc["answer"],
        "pred": pred,
        "ann_id": ann_id,
        "bbox": doc["bbox"],
    }
    return {f"refcoco_{metric}": data_dict for metric in COCO_REC_METRICS}
```

After these modifications, there are a small increase in all metrics:
![image](https://github.com/user-attachments/assets/19024c96-c658-48d5-86ec-083918bc33f5)


I noticed another thing. Qwen's output are scaled to [0,1000], based on which the process_result fuction should treat qwen2 as a special case.
```python
def refcoco_bbox_rec_process_result(doc, result):
    """
    Args:
        doc: a instance of the eval dataset
        results: [pred]
    Returns:
        a dictionary with key: metric name, value: metric value
    """
    pred = result[0] if len(result) > 0 else ""
    is_qwen2vl = False
    if "
" in pred:
        is_qwen2vl = True
    pred = parse_float_sequence_within(pred)
    if is_qwen2vl is True:
        pred = [num / 1000 for num in pred]
    else:
        if any([x < 0 - 1e-5 or x > 1 + 1e-5 for x in pred]):
            pred = [
                x / doc["image_width"]
                if i % 2 == 0
                else x / doc["image_height"]
                for i, x in enumerate(pred)
            ]
    ann_id = doc["question_id"]
    data_dict = {
        "answer": doc["answer"],
        "pred": pred,
        "ann_id": ann_id,
        "bbox": doc["bbox"],
    }
    return {f"refcoco_{metric}": data_dict for metric in COCO_REC_METRICS}
```

To make the above code functional. Also the `skip_special_tokens` in qwen2_vl.py should be changed to False
![image](https://github.com/user-attachments/assets/1adf64f8-abc7-4df1-85d6-598b7e6ac6c3)

And new scores:
![image](https://github.com/user-attachments/assets/552d5209-351a-4ef0-9907-12b3ba048c8c)


> I noticed another thing. Qwen's output are scaled to [0,1000], based on which the process_result fuction should treat qwen2 as a special case.
> 
> ```python
> def refcoco_bbox_rec_process_result(doc, result):
>     """
>     Args:
>         doc: a instance of the eval dataset
>         results: [pred]
>     Returns:
>         a dictionary with key: metric name, value: metric value
>     """
>     pred = result[0] if len(result) > 0 else ""
>     is_qwen2vl = False
>     if "
" in pred:
>         is_qwen2vl = True
>     pred = parse_float_sequence_within(pred)
>     if is_qwen2vl is True:
>         pred = [num / 1000 for num in pred]
>     else:
>         if any([x < 0 - 1e-5 or x > 1 + 1e-5 for x in pred]):
>             pred = [
>                 x / doc["image_width"]
>                 if i % 2 == 0
>                 else x / doc["image_height"]
>                 for i, x in enumerate(pred)
>             ]
>     ann_id = doc["question_id"]
>     data_dict = {
>         "answer": doc["answer"],
>         "pred": pred,
>         "ann_id": ann_id,
>         "bbox": doc["bbox"],
>     }
>     return {f"refcoco_{metric}": data_dict for metric in COCO_REC_METRICS}
> ```
> 
> To make the above code functional. Also the `skip_special_tokens` in qwen2_vl.py should be changed to False ![image](https://private-user-images.githubusercontent.com/55043972/383446070-1adf64f8-abc7-4df1-85d6-598b7e6ac6c3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzE3NzI1MjUsIm5iZiI6MTczMTc3MjIyNSwicGF0aCI6Ii81NTA0Mzk3Mi8zODM0NDYwNzAtMWFkZjY0ZjgtYWJjNy00ZGYxLTg1ZDYtNTk4YjdlNmFjNmMzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDExMTYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMTE2VDE1NTAyNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWFhOWJjZDU0YWUxZThmOTk5OWZjMjQ5MWVlNTc4ZmJjOWVlN2VhY2FhNzg5YjViOWQ2NjE1MTE2Yzc2OGJiZTkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.YqTtMIjjxSaEbgpU3YwpNMtruwYMylZUmgC_rBLT9PQ)
> 
> And new scores: ![image](https://private-user-images.githubusercontent.com/55043972/383446522-552d5209-351a-4ef0-9907-12b3ba048c8c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzE3NzI1MjUsIm5iZiI6MTczMTc3MjIyNSwicGF0aCI6Ii81NTA0Mzk3Mi8zODM0NDY1MjItNTUyZDUyMDktMzUxYS00ZWYwLTk5MDctMTJiM2JhMDQ4YzhjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDExMTYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMTE2VDE1NTAyNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWRkMDU4ZGEwYmQ5MmRlNmRlODFkZjAzOWQ3MzMyODA4OTJmZWE5NjE5OTBjNTE5MTE2YmNhZjA3ZjEzYTk4ZGMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.mhVpcCG6yKuSMbqCse2Ct2Wz3ffPvBwkSW_ok38tdhA)

hello! Iam also reproducing the result for qwen2-vl-7b, but the performance is slightly belowe the results in their paper, which I deem as the prompt issue.
Could you please share with me the prompt that you are using? Or you just follow the original lmms-eval prompt:Bounding box coordinates are specified in the format (top-left x, top-left y, bottom-right x, bottom-right y). All values are floating point numbers bounded between 0 and 1. Please provide the bounding box coordinate of the region this sentence describes.
Here is my result:
![image](https://github.com/user-attachments/assets/a6926f61-4297-4e95-bb0a-c116c61b7b35)
Really looking forward to your EARLY reply!!! Thank you so much~~~


OK, problem has been solved, you just eliminate the special token: Please provide the bounding box coordinate of the region this sentence describes and it will work

> OK, problem has been solved, you just eliminate the special token: Please provide the bounding box coordinate of the region this sentence describes and it will work

Thank you for your sharing! May I ask where to change the text prompt for Qwen2-VL? The default prompt is:

Bounding box coordinates are specified in the format (top-left x, top-left y, bottom-right x, bottom-right y). All values are floating point numbers bounded between 0 and 1. Please provide the bounding box coordinate of the region this sentence describes: 

I try to change  --system_instruction, but this variable is not the correct one.

Here~
lmms-eval-main/lmms_eval/tasks/refcoco/utils_rec.py
line 48
def refcoco_bbox_rec_doc_to_text(doc):
    assert isinstance(doc["answer"], str), "Answer must be a string"
    return (
        "Please locate the region this sentence describes in the image: " + doc["answer"]
    )

> > I noticed another thing. Qwen's output are scaled to [0,1000], based on which the process_result fuction should treat qwen2 as a special case.
> > def refcoco_bbox_rec_process_result(doc, result):
> >     """
> >     Args:
> >         doc: a instance of the eval dataset
> >         results: [pred]
> >     Returns:
> >         a dictionary with key: metric name, value: metric value
> >     """
> >     pred = result[0] if len(result) > 0 else ""
> >     is_qwen2vl = False
> >     if "
" in pred:
> >         is_qwen2vl = True
> >     pred = parse_float_sequence_within(pred)
> >     if is_qwen2vl is True:
> >         pred = [num / 1000 for num in pred]
> >     else:
> >         if any([x < 0 - 1e-5 or x > 1 + 1e-5 for x in pred]):
> >             pred = [
> >                 x / doc["image_width"]
> >                 if i % 2 == 0
> >                 else x / doc["image_height"]
> >                 for i, x in enumerate(pred)
> >             ]
> >     ann_id = doc["question_id"]
> >     data_dict = {
> >         "answer": doc["answer"],
> >         "pred": pred,
> >         "ann_id": ann_id,
> >         "bbox": doc["bbox"],
> >     }
> >     return {f"refcoco_{metric}": data_dict for metric in COCO_REC_METRICS}
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > To make the above code functional. Also the `skip_special_tokens` in qwen2_vl.py should be changed to False ![image](https://private-user-images.githubusercontent.com/55043972/383446070-1adf64f8-abc7-4df1-85d6-598b7e6ac6c3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzE3NzI1MjUsIm5iZiI6MTczMTc3MjIyNSwicGF0aCI6Ii81NTA0Mzk3Mi8zODM0NDYwNzAtMWFkZjY0ZjgtYWJjNy00ZGYxLTg1ZDYtNTk4YjdlNmFjNmMzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDExMTYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMTE2VDE1NTAyNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWFhOWJjZDU0YWUxZThmOTk5OWZjMjQ5MWVlNTc4ZmJjOWVlN2VhY2FhNzg5YjViOWQ2NjE1MTE2Yzc2OGJiZTkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.YqTtMIjjxSaEbgpU3YwpNMtruwYMylZUmgC_rBLT9PQ)
> > And new scores: ![image](https://private-user-images.githubusercontent.com/55043972/383446522-552d5209-351a-4ef0-9907-12b3ba048c8c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzE3NzI1MjUsIm5iZiI6MTczMTc3MjIyNSwicGF0aCI6Ii81NTA0Mzk3Mi8zODM0NDY1MjItNTUyZDUyMDktMzUxYS00ZWYwLTk5MDctMTJiM2JhMDQ4YzhjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDExMTYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMTE2VDE1NTAyNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWRkMDU4ZGEwYmQ5MmRlNmRlODFkZjAzOWQ3MzMyODA4OTJmZWE5NjE5OTBjNTE5MTE2YmNhZjA3ZjEzYTk4ZGMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.mhVpcCG6yKuSMbqCse2Ct2Wz3ffPvBwkSW_ok38tdhA)
> 
> hello! Iam also reproducing the result for qwen2-vl-7b, but the performance is slightly belowe the results in their paper, which I deem as the prompt issue. Could you please share with me the prompt that you are using? Or you just follow the original lmms-eval prompt:Bounding box coordinates are specified in the format (top-left x, top-left y, bottom-right x, bottom-right y). All values are floating point numbers bounded between 0 and 1. Please provide the bounding box coordinate of the region this sentence describes. Here is my result: ![image](https://github.com/user-attachments/assets/a6926f61-4297-4e95-bb0a-c116c61b7b35) Really looking forward to your EARLY reply!!! Thank you so much~~~

Hello, I would like to confirm whether the result of Location related to RefCOCO in the Qwen2.5VL report is AP50 or mAP.

> > > I noticed another thing. Qwen's output are scaled to [0,1000], based on which the process_result fuction should treat qwen2 as a special case.
> > > def refcoco_bbox_rec_process_result(doc, result):
> > > """
> > > Args:
> > > doc: a instance of the eval dataset
> > > results: [pred]
> > > Returns:
> > > a dictionary with key: metric name, value: metric value
> > > """
> > > pred = result[0] if len(result) > 0 else ""
> > > is_qwen2vl = False
> > > if "
" in pred:
> > > is_qwen2vl = True
> > > pred = parse_float_sequence_within(pred)
> > > if is_qwen2vl is True:
> > > pred = [num / 1000 for num in pred]
> > > else:
> > > if any([x < 0 - 1e-5 or x > 1 + 1e-5 for x in pred]):
> > > pred = [
> > > x / doc["image_width"]
> > > if i % 2 == 0
> > > else x / doc["image_height"]
> > > for i, x in enumerate(pred)
> > > ]
> > > ann_id = doc["question_id"]
> > > data_dict = {
> > > "answer": doc["answer"],
> > > "pred": pred,
> > > "ann_id": ann_id,
> > > "bbox": doc["bbox"],
> > > }
> > > return {f"refcoco_{metric}": data_dict for metric in COCO_REC_METRICS}
> > > To make the above code functional. Also the `skip_special_tokens` in qwen2_vl.py should be changed to False ![image](https://private-user-images.githubusercontent.com/55043972/383446070-1adf64f8-abc7-4df1-85d6-598b7e6ac6c3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzE3NzI1MjUsIm5iZiI6MTczMTc3MjIyNSwicGF0aCI6Ii81NTA0Mzk3Mi8zODM0NDYwNzAtMWFkZjY0ZjgtYWJjNy00ZGYxLTg1ZDYtNTk4YjdlNmFjNmMzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDExMTYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMTE2VDE1NTAyNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWFhOWJjZDU0YWUxZThmOTk5OWZjMjQ5MWVlNTc4ZmJjOWVlN2VhY2FhNzg5YjViOWQ2NjE1MTE2Yzc2OGJiZTkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.YqTtMIjjxSaEbgpU3YwpNMtruwYMylZUmgC_rBLT9PQ)
> > > And new scores: ![image](https://private-user-images.githubusercontent.com/55043972/383446522-552d5209-351a-4ef0-9907-12b3ba048c8c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzE3NzI1MjUsIm5iZiI6MTczMTc3MjIyNSwicGF0aCI6Ii81NTA0Mzk3Mi8zODM0NDY1MjItNTUyZDUyMDktMzUxYS00ZWYwLTk5MDctMTJiM2JhMDQ4YzhjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDExMTYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMTE2VDE1NTAyNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWRkMDU4ZGEwYmQ5MmRlNmRlODFkZjAzOWQ3MzMyODA4OTJmZWE5NjE5OTBjNTE5MTE2YmNhZjA3ZjEzYTk4ZGMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.mhVpcCG6yKuSMbqCse2Ct2Wz3ffPvBwkSW_ok38tdhA)
> > 
> > 
> > hello! Iam also reproducing the result for qwen2-vl-7b, but the performance is slightly belowe the results in their paper, which I deem as the prompt issue. Could you please share with me the prompt that you are using? Or you just follow the original lmms-eval prompt:Bounding box coordinates are specified in the format (top-left x, top-left y, bottom-right x, bottom-right y). All values are floating point numbers bounded between 0 and 1. Please provide the bounding box coordinate of the region this sentence describes. Here is my result: ![image](https://github.com/user-attachments/assets/a6926f61-4297-4e95-bb0a-c116c61b7b35) Really looking forward to your EARLY reply!!! Thank you so much~~~
> 
> Hello, I would like to confirm whether the result of Location related to RefCOCO in the Qwen2.5VL report is AP50 or mAP.

Hi, I found the value in Qwen2.5vl paper is ACC@0.5, but I still can not use lmmeval to reproduce the results, e.g. refcoco testB 0.80 vs 0.85 in paper. Can you reproduce the results?

> > > > I noticed another thing. Qwen's output are scaled to [0,1000], based on which the process_result fuction should treat qwen2 as a special case.
> > > > def refcoco_bbox_rec_process_result(doc, result):
> > > > """
> > > > Args:
> > > > doc: a instance of the eval dataset
> > > > results: [pred]
> > > > Returns:
> > > > a dictionary with key: metric name, value: metric value
> > > > """
> > > > pred = result[0] if len(result) > 0 else ""
> > > > is_qwen2vl = False
> > > > if "
" in pred:
> > > > is_qwen2vl = True
> > > > pred = parse_float_sequence_within(pred)
> > > > if is_qwen2vl is True:
> > > > pred = [num / 1000 for num in pred]
> > > > else:
> > > > if any([x < 0 - 1e-5 or x > 1 + 1e-5 for x in pred]):
> > > > pred = [
> > > > x / doc["image_width"]
> > > > if i % 2 == 0
> > > > else x / doc["image_height"]
> > > > for i, x in enumerate(pred)
> > > > ]
> > > > ann_id = doc["question_id"]
> > > > data_dict = {
> > > > "answer": doc["answer"],
> > > > "pred": pred,
> > > > "ann_id": ann_id,
> > > > "bbox": doc["bbox"],
> > > > }
> > > > return {f"refcoco_{metric}": data_dict for metric in COCO_REC_METRICS}
> > > > To make the above code functional. Also the `skip_special_tokens` in qwen2_vl.py should be changed to False ![image](https://private-user-images.githubusercontent.com/55043972/383446070-1adf64f8-abc7-4df1-85d6-598b7e6ac6c3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzE3NzI1MjUsIm5iZiI6MTczMTc3MjIyNSwicGF0aCI6Ii81NTA0Mzk3Mi8zODM0NDYwNzAtMWFkZjY0ZjgtYWJjNy00ZGYxLTg1ZDYtNTk4YjdlNmFjNmMzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDExMTYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMTE2VDE1NTAyNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWFhOWJjZDU0YWUxZThmOTk5OWZjMjQ5MWVlNTc4ZmJjOWVlN2VhY2FhNzg5YjViOWQ2NjE1MTE2Yzc2OGJiZTkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.YqTtMIjjxSaEbgpU3YwpNMtruwYMylZUmgC_rBLT9PQ)
> > > > And new scores: ![image](https://private-user-images.githubusercontent.com/55043972/383446522-552d5209-351a-4ef0-9907-12b3ba048c8c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzE3NzI1MjUsIm5iZiI6MTczMTc3MjIyNSwicGF0aCI6Ii81NTA0Mzk3Mi8zODM0NDY1MjItNTUyZDUyMDktMzUxYS00ZWYwLTk5MDctMTJiM2JhMDQ4YzhjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDExMTYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMTE2VDE1NTAyNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWRkMDU4ZGEwYmQ5MmRlNmRlODFkZjAzOWQ3MzMyODA4OTJmZWE5NjE5OTBjNTE5MTE2YmNhZjA3ZjEzYTk4ZGMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.mhVpcCG6yKuSMbqCse2Ct2Wz3ffPvBwkSW_ok38tdhA)
> > > 
> > > 
> > > hello! Iam also reproducing the result for qwen2-vl-7b, but the performance is slightly belowe the results in their paper, which I deem as the prompt issue. Could you please share with me the prompt that you are using? Or you just follow the original lmms-eval prompt:Bounding box coordinates are specified in the format (top-left x, top-left y, bottom-right x, bottom-right y). All values are floating point numbers bounded between 0 and 1. Please provide the bounding box coordinate of the region this sentence describes. Here is my result: ![image](https://github.com/user-attachments/assets/a6926f61-4297-4e95-bb0a-c116c61b7b35) Really looking forward to your EARLY reply!!! Thank you so much~~~
> > 
> > 
> > Hello, I would like to confirm whether the result of Location related to RefCOCO in the Qwen2.5VL report is AP50 or mAP.
> 
> Hi, I found the value in Qwen2.5vl paper is ACC@0.5, but I still can not use lmmeval to reproduce the results, e.g. refcoco testB 0.80 vs 0.85 in paper. Can you reproduce the results?

It seems that the number of test data in the refcoco dataset provided by lmms is more than that in other versions. I am still checking.