Repository: EvolvingLMMs-Lab/lmms-eval
Issue: I keep facing "404 Client Error" while evaluating with my custom evaluation dataset and task.
Root cause: Validation Gap

Hi. I would like to ask for some advice from people who tried using custom task for lmms-eval, especially in llava-wilder style.

I have been trying to evaluate LLaVA-NeXT finetuned with my custom train dataset and I keep facing 404 Client error and the generated responses always come messed up.

At first, I thought it might have been the problem with the training process because the loss had not decreased. However, I solved this issue by increasing the epoch.

Anyways, I designed my custom data in [llava-wilder style](https://github.com/EvolvingLMMs-Lab/lmms-eval/tree/main/lmms_eval/tasks/llava_wilder) but added captionings because I want to make the evaluation model accept both image and its textual description for better understanding. Plus, LLaVA-v1.5 and v1.6 accepts textual descriptions, not images.'

I created my a directory for my own task like this:
<img width="183" alt="image" src="https://github.com/user-attachments/assets/e35b274b-0b91-4a1b-9e4f-427af497a775">


Let me share with you my link for my custom data on Huggingface and its associated files.
My dataset: [Bleking/floorplan_dataset_wilder](https://huggingface.co/datasets/Bleking/floorplan_dataset_wilder) (Has captions additionally)
The original LLaVA-wilder dataset: [lmms-lab/LLaVA-Bench-Wilder](https://huggingface.co/datasets/lmms-lab/LLaVA-Bench-Wilder)


**utils.py**
```
import base64
import os
import time
from copy import deepcopy
from http import HTTPStatus
from io import BytesIO
from pathlib import Path

import numpy as np
import requests
import yaml

# Set up a logger
from loguru import logger as eval_logger 

NUM_SECONDS_TO_SLEEP = 5
dir_path = os.path.dirname(os.path.realpath(__file__))

judge_rules = "We would like to request your feedback on the performance of two AI assistants in response to the user question displayed above. The user asks the question on observing an image shown to you. \nPlease rate the helpfulness, relevance, accuracy, level of details of their responses. Each assistant receives an overall score on a scale of 1 to 10, where a higher score indicates better overall performance. Assume assistant 1 always receive a score of 10 and is the correct answer.\nPlease first output a single line containing only two values indicating the scores for Assistant 1 and 2, respectively. The two scores are separated by a space.\nIn the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment."

with open(Path(__file__).parent / "_default_template_wilder_yaml", "r") as f:
    raw_data = f.readlines()
    safe_data = []
    for i, line in enumerate(raw_data):
        # remove function definition since yaml load cannot handle it
        if "!function" not in line:
            safe_data.append(line)

    config = yaml.safe_load("".join(safe_data))

GPT_EVAL_MODEL_NAME = config["metadata"]["gpt_eval_model_name"]
API_TYPE = config["metadata"]["api_type"]

if API_TYPE == "openai":
    API_URL = os.getenv("OPENAI_API_URL", "https://api.openai.com/v1/chat/completions")
    API_KEY = os.getenv("OPENAI_API_KEY", "sk-proj-...EXUA")
    # API_KEY = "sk-proj-...EXUA"
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json",
    }

# elif API_TYPE == "azure":
#     API_URL = os.getenv("AZURE_ENDPOINT", "https://api.cognitive.microsoft.com/sts/v1.0/issueToken")
#     API_KEY = os.getenv("AZURE_API_KEY", "YOUR_API_KEY")
#     headers = {
#         "api-key": API_KEY,
#         "Content-Type": "application/json",
#     }


def image_to_base64(base64_image, prompt, max_retries=5, wait_time=10):
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json",
    }

    payload = {
        "model": GPT_EVAL_MODEL_NAME,
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": prompt},
                    {
                        "type": "image_url",
                        "image_url": f"data:image/jpeg;base64,{base64_image}",
                    },
                ],
            }
        ],
        "max_tokens": 1024,
        "temperature": 0.0,
    }
    print(payload)

    for attempt in range(max_retries):
        try:
            response = requests.post(API_URL, headers=headers, json=payload, timeout=60)
            response.raise_for_status()
            response_data = response.json()
            return response_data["choices"][0]["message"]["content"], GPT_EVAL_MODEL_NAME
        except requests.exceptions.RequestException as e:
            eval_logger.warning(f"Request failed on attempt {attempt+1}: {e}")
            time.sleep(wait_time)
            if attempt == max_retries - 1:
                eval_logger.error(f"Failed to get response after {max_retries} attempts")
                return "", GPT_EVAL_MODEL_NAME
        except Exception as e:
            eval_logger.error(f"Error on attempt {attempt+1}: {e}")
            return "", GPT_EVAL_MODEL_NAME


def image_to_base64(pil_image):
    buffered = BytesIO()
    pil_image.save(buffered, format="PNG")
    return base64.b64encode(buffered.getvalue()).decode("utf-8")


def parse_score(review):
    try:
        score_pair = review.split("\n")[0]
        score_pair = score_pair.replace(",", " ")
        sp = score_pair.split(" ")
        if len(sp) == 2:
            return [float(sp[0]), float(sp[1])]
        else:
            eval_logger.debug(f"Can not split: {review}. Returning [-1, -1]")
            return [-1, -1]
    except Exception as e:
        eval_logger.debug(f"Error: {e}. Returning [-1, -1]")
        return [-1, -1]


def llava_process_results(doc, result):
    """
    Args:
        doc: a instance of the eval dataset
        results: [pred]
    Returns:
        a dictionary with key: metric name (in this case coco_bleu), value: metric value
    """
    try:
        context = doc["caption"]  # 정확성 향상을 위해 새롭게 추가.
        question = doc.get("Question", "")
        ans1 = doc.get("Answer", "")  # reference 답변
        ans2 = result[0] if result else ""  # generated 답변
        # content = f"[Question]\n{question}\n\n" + f"[Assistant 1]\n{ans1}\n\n[End of Assistant 1]\n\n" + f"[Assistant 2]\n{ans2}\n\n[End of Assistant 2]\n\n" f"[System]\n{judge_rules}\n\n"
        content = f"[Context]\n{context}\n\n" + f"[Question]\n{question}\n\n" + f"[Assistant 1]\n{ans1}\n\n[End of Assistant 1]\n\n" + f"[Assistant 2]\n{ans2}\n\n[End of Assistant 2]\n\n" f"[System]\n{judge_rules}\n\n"
        visuals = llava_doc_to_visual(doc)
        image_path = doc["image"]
        base64_image = image_to_base64(image_path)
        review, model_name = get_chat_response(base64_image, content)
        scores = parse_score(review)
    except Exception as e:
        eval_logger.error(f"Error for Question ID: {doc.get('question_id', 'Unknown')}: {e}")
        review = "Failed to Get a Proper Review."
        model_name = "Failed Request"
        scores = [-1, -1]

    data_dict = {"question": question, "ans1": ans1, "ans2": ans2, "review": review, "scores": scores, "eval_model": model_name, "content": content}
    # return {"gpt_eval_llava_all": review_dict}
    return {"gpt_eval_llava_all": data_dict}


def llava_doc_to_visual(doc):
    return [doc["image"].convert("RGB")]


def llava_doc_to_text(doc, lmms_eval_specific_kwargs=None):
    if lmms_eval_specific_kwargs is None:
        lmms_eval_specific_kwargs = {}
    pre_prompt = lmms_eval_specific_kwargs.get("pre_prompt", "")
    post_prompt = lmms_eval_specific_kwargs.get("post_prompt", "")
    return f"{pre_prompt}{doc['Question']}{post_prompt}"


def llava_all_aggregation(results):
    return llava_aggregation(results, "all")


def llava_aggregation(results, category):
    try:
        scores = []
        for result in results:
            if -999 in result["scores"]:
                continue
            scores.append(result["scores"])

        stats = np.asarray(scores).mean(0).tolist()
        stats = [round(x, 3) for x in stats]
        # gpt4_score_percentage = stats[0] * 10
        # model_score_percentage = stats[1] * 10
        # eval_logger.info(f"Category: {category}")
        # eval_logger.info(f"GPT4 Score: {gpt4_score_percentage:.1f}%")
        # eval_logger.info(f"Model Score: {model_score_percentage:.1f}%")
        # eval_logger.info("=========================")
        return round(stats[1] / stats[0] * 100, 1)
    except Exception as e:
        eval_logger.info(f"Error in llava_aggregation: {e}, and in category: {category}")
        return None
```

**_default_template_wilder_yaml**
```
output_type: generate_until
doc_to_visual: !function utils.llava_doc_to_visual
doc_to_text: !function utils.llava_doc_to_text
doc_to_target: "gpt4v_answer"
generation_kwargs:
  max_new_tokens: 4096
  temperature: 0
  top_p: 1.0
  num_beams: 1
  do_sample: false
process_results: !function utils.llava_process_results
process_results_use_image: true
metric_list:
  - metric: gpt_eval_llava_all
    aggregation: !function utils.llava_all_aggregation
    higher_is_better: true
metadata:
  version: 0.0
  api_type : openai
  gpt_eval_model_name: "gpt-4-vision-preview"
```

**floorplan_test_llava_bench_wilder.yaml**
```
dataset_path: Bleking/floorplan_dataset_wilder
dataset_kwargs:
  token: True
task: "floorplan_test_wilder"
test_split: test 
lmms_eval_specific_kwargs:
  default:
    pre_prompt: ""
    post_prompt: ""
  xcomposer2_4khd:
    pre_prompt: "[UNUSED_TOKEN_146]user\nQuestion: "
    post_prompt: "[UNUSED_TOKEN_145]\n[UNUSED_TOKEN_146]assistant\n"
include: _default_template_wilder_yaml
```

The difference of my 'utils.py' from the original llava-wilder's [utils.py](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/tasks/llava_wilder/utils.py) is the inclusion of the line `context = doc["caption"]` and the change of the 'content' value inside the "llava_process_results" function.

Regarding that I receive the following messages
- `2024-10-01 14:39:35.656 | WARNING  | utils:get_chat_response:84 - Request failed on attempt 5: 404 Client Error: Not Found for url: https://api.openai.com/v1/chat/completions`
- `2024-10-01 14:39:43.946 | ERROR    | utils:get_chat_response:87 - Failed to get response after 5 attempts`
- `...AABmNDQAAAAAyGhsAAAAAZDQ2AAAAAMj8fwINFZ/HuGkIAAAAAElFTkSuQmCC'}]}], 'max_tokens': 1024, 'temperature': 0.0}`

I will also share with you my command and the image of my finetuned model's directory for your context.
<img width="184" alt="image" src="https://github.com/user-attachments/assets/3033689b-7470-4474-937c-bdcf59852b89">
`
python3 -m accelerate.commands.launch --num_processes=2 -m lmms_eval --model llava --model_args pretrained=/home/work/testdataset1/LLaVA-NeXT/checkpoints/NeXT/Llama-3.1-Minitron-4B-Width-Base/checkpoint-21,conv_template=llava_llama_3,device=cuda --tasks floorplan_test_wilder --batch_size 1 --log_samples --log_samples_suffix llava_next --output_path ./logs/
`

What are the possible causes to contribute to my current issue?
1) Still an incorrect way of using the open ai API despite properly using the project API key
2) Wrong way of loading a newly added column, captioning, in my utils.py code.
3) Issue with using the model, 'gpt-4-vision-preview'
4) Something else

Thank you.


Well, I had figured out that it was due to the repetitive generation of a certain text as the generated response. I am guessing it was due to the poor finetuning of the model.
If anyone has the similar issue, you may take it as that.

In case you would like to check what the exact issue is, you may add `print(f"Error response: {response.text}")` line to the `utils.py` code. Something like this

```
    for attempt in range(max_retries):
        try:
            response = requests.post(API_URL, headers=headers, json=payload, timeout=60)
            response.raise_for_status()
            response_data = response.json()
            return response_data["choices"][0]["message"]["content"], GPT_EVAL_MODEL_NAME
        except requests.exceptions.RequestException as e:
            print(f"Error response: {response.text}")  # Log the actual error
            eval_logger.warning(f"Request failed on attempt {attempt+1}: {e}")
            time.sleep(wait_time)
            if attempt == max_retries - 1:
                eval_logger.error(f"Failed to get response after {max_retries} attempts")
                return "", GPT_EVAL_MODEL_NAME
        except Exception as e:
            eval_logger.error(f"Error on attempt {attempt+1}: {e}")
            return "", GPT_EVAL_MODEL_NAME
```

I do not have a clear solution but I hope it still could help you. Thank you.