Repository: modelscope/evalscope
Issue: Discrepancy in Custom Model Evaluation with Chameleon - 10pp Lower than Official Benchmark
Root cause: Configuration Error

## Self-Check List

Before submitting an issue, please ensure you have completed the following steps:
- [x] I have carefully read the [relevant user documentation](https://evalscope.readthedocs.io/en/latest/get_started/parameters.html)
- [x] I have reviewed the [Frequently Asked Questions](https://evalscope.readthedocs.io/en/latest/get_started/faq.html)
- [x] I have searched and reviewed existing issues to confirm this is not a duplicate problem

## Problem Description

I am experiencing a discrepancy in the evaluation results of our custom model when integrated with Chameleon. The evaluation score is 10 percentage points lower compared to the official benchmark. I would like to know if there are any special configurations or settings required to achieve the expected performance.

<img width="1104" height="287" alt="Image" src="https://github.com/user-attachments/assets/9e7814d7-4615-48fe-b95d-9bdc0f63c800" />

<img width="445" height="729" alt="Image" src="https://github.com/user-attachments/assets/3d002f7a-6e83-4eb6-beed-aee43a605569" />


## EvalScope Version (Required)
v0.xx.x

## Tools Used
- [x] Native / Native framework
- [ ] Opencompass backend
- [ ] VLMEvalKit backend
- [ ] RAGEval backend
- [ ] Perf / Model inference stress testing tool
- [ ] Arena / Arena mode

## Executed Code or Instructions

Please provide the main code or instructions you executed.

## Error Log

Please paste the complete error log or console output.

## Running Environment

- Operating System:
- Python Version:

## Additional Information

If there is any other relevant information, please provide it here.

Task config:
```yaml
analysis_report: false
api_url: null
chat_template: null
dataset_args:
  arc:
    aggregation: mean
    dataset_id: allenai/ai2_arc
    default_subset: default
    description: 'The ARC (AI2 Reasoning Challenge) benchmark is designed to evaluate
      the reasoning capabilities of AI models through multiple-choice questions derived
      from science exams. It includes two subsets: ARC-Easy and ARC-Challenge, which
      vary in difficulty.'
    eval_split: test
    extra_params: {}
    few_shot_num: 0
    few_shot_prompt_template: null
    few_shot_random: false
    filters: null
    force_redownload: false
    metric_list:
    - acc
    name: arc
    output_types:
    - generation
    pretty_name: ARC
    prompt_template: 'Answer the following multiple choice question. The entire content
      of your response should be of the following format: ''ANSWER: [LETTER]'' (without
      quotes) where [LETTER] is one of {letters}.


      {question}


      {choices}'
    query_template: null
    review_timeout: null
    sandbox_config: {}
    shuffle: false
    shuffle_choices: false
    subset_list:
    - ARC-Easy
    - ARC-Challenge
    system_prompt: null
    tags:
    - Reasoning
    - MCQ
    train_split: train
  gsm8k:
    aggregation: mean
    dataset_id: AI-ModelScope/gsm8k
    default_subset: default
    description: GSM8K (Grade School Math 8K) is a dataset of grade school math problems,
      designed to evaluate the mathematical reasoning abilities of AI models.
    eval_split: test
    extra_params: {}
    few_shot_num: 4
    few_shot_prompt_template: 'Here are some examples of how to solve similar problems:


      {fewshot}


      {question}

      Please reason step by step, and put your final answer within \boxed{{}}.'
    few_shot_random: false
    filters: null
    force_redownload: false
    metric_list:
    - acc:
        numeric: true
    name: gsm8k
    output_types:
    - generation
    pretty_name: GSM8K
    prompt_template: '{question}

      Please reason step by step, and put your final answer within \boxed{{}}.'
    query_template: null
    review_timeout: null
    sandbox_config: {}
    shuffle: false
    shuffle_choices: false
    subset_list:
    - main
    system_prompt: null
    tags:
    - Math
    - Reasoning
    train_split: train
  hellaswag:
    aggregation: mean
    dataset_id: evalscope/hellaswag
    default_subset: default
    description: HellaSwag is a benchmark for commonsense reasoning in natural language
      understanding tasks. It consists of multiple-choice questions where the model
      must select the most plausible continuation of a given context.
    eval_split: validation
    extra_params: {}
    few_shot_num: 0
    few_shot_prompt_template: null
    few_shot_random: false
    filters: null
    force_redownload: false
    metric_list:
    - acc
    name: hellaswag
    output_types:
    - generation
    pretty_name: HellaSwag
    prompt_template: 'Answer the following multiple choice question. The entire content
      of your response should be of the following format: ''ANSWER: [LETTER]'' (without
      quotes) where [LETTER] is one of {letters}.


      {question}


      {choices}'
    query_template: null
    review_timeout: null
    sandbox_config: {}
    shuffle: false
    shuffle_choices: false
    subset_list:
    - default
    system_prompt: null
    tags:
    - Commonsense
    - MCQ
    - Knowledge
    train_split: null
  mmlu:
    aggregation: mean
    dataset_id: cais/mmlu
    default_subset: all
    description: The MMLU (Massive Multitask Language Understanding) benchmark is
      a comprehensive evaluation suite designed to assess the performance of language
      models across a wide range of subjects and tasks. It includes multiple-choice
      questions from various domains, such as history, science, mathematics, and more,
      providing a robust measure of a model's understanding and reasoning capabilities.
    eval_split: test
    extra_params: {}
    few_shot_num: 5
    few_shot_prompt_template: null
    few_shot_random: false
    filters: null
    force_redownload: false
    metric_list:
    - acc
    name: mmlu
    output_types:
    - generation
    pretty_name: MMLU
    prompt_template: 'Answer the following multiple choice question. The last line
      of your response should be of the following format: ''ANSWER: [LETTER]'' (without
      quotes) where [LETTER] is one of {letters}. Think step by step before answering.


      {question}


      {choices}'
    query_template: null
    review_timeout: null
    sandbox_config: {}
    shuffle: false
    shuffle_choices: false
    subset_list:
    - abstract_algebra
    - anatomy
    - astronomy
    - business_ethics
    - clinical_knowledge
    - college_biology
    - college_chemistry
    - college_computer_science
    - college_mathematics
    - college_medicine
    - college_physics
    - computer_security
    - conceptual_physics
    - econometrics
    - electrical_engineering
    - elementary_mathematics
    - formal_logic
    - global_facts
    - high_school_biology
    - high_school_chemistry
    - high_school_computer_science
    - high_school_european_history
    - high_school_geography
    - high_school_government_and_politics
    - high_school_macroeconomics
    - high_school_mathematics
    - high_school_microeconomics
    - high_school_physics
    - high_school_psychology
    - high_school_statistics
    - high_school_us_history
    - high_school_world_history
    - human_aging
    - human_sexuality
    - international_law
    - jurisprudence
    - logical_fallacies
    - machine_learning
    - management
    - marketing
    - medical_genetics
    - miscellaneous
    - moral_disputes
    - moral_scenarios
    - nutrition
    - philosophy
    - prehistory
    - professional_accounting
    - professional_law
    - professional_medicine
    - professional_psychology
    - public_relations
    - security_studies
    - sociology
    - us_foreign_policy
    - virology
    - world_religions
    system_prompt: null
    tags:
    - Knowledge
    - MCQ
    train_split: dev
  piqa:
    aggregation: mean
    dataset_id: extraordinarylab/piqa
    default_subset: default
    description: PIQA addresses the challenging task of reasoning about physical commonsense
      in natural language.
    eval_split: validation
    extra_params: {}
    few_shot_num: 0
    few_shot_prompt_template: null
    few_shot_random: false
    filters: null
    force_redownload: false
    metric_list:
    - acc
    name: piqa
    output_types:
    - generation
    pretty_name: PIQA
    prompt_template: 'Answer the following multiple choice question. The entire content
      of your response should be of the following format: ''ANSWER: [LETTER]'' (without
      quotes) where [LETTER] is one of {letters}.


      {question}


      {choices}'
    query_template: null
    review_timeout: null
    sandbox_config: {}
    shuffle: false
    shuffle_choices: false
    subset_list:
    - default
    system_prompt: null
    tags:
    - Reasoning
    - Commonsense
    - MCQ
    train_split: train
  siqa:
    aggregation: mean
    dataset_id: extraordinarylab/siqa
    default_subset: default
    description: Social Interaction QA (SIQA) is a question-answering benchmark for
      testing social commonsense intelligence. Contrary to many prior benchmarks that
      focus on physical or taxonomic knowledge, Social IQa focuses on reasoning about
      people's actions and their social implications.
    eval_split: validation
    extra_params: {}
    few_shot_num: 0
    few_shot_prompt_template: null
    few_shot_random: false
    filters: null
    force_redownload: false
    metric_list:
    - acc
    name: siqa
    output_types:
    - generation
    pretty_name: SIQA
    prompt_template: 'Answer the following multiple choice question. The entire content
      of your response should be of the following format: ''ANSWER: [LETTER]'' (without
      quotes) where [LETTER] is one of {letters}.


      {question}


      {choices}'
    query_template: null
    review_timeout: null
    sandbox_config: {}
    shuffle: false
    shuffle_choices: false
    subset_list:
    - default
    system_prompt: null
    tags:
    - Commonsense
    - Reasoning
    - MCQ
    train_split: null
  winogrande:
    aggregation: mean
    dataset_id: AI-ModelScope/winogrande_val
    default_subset: default
    description: Winogrande is a benchmark for evaluating AI models on commonsense
      reasoning tasks, specifically designed to test the ability to resolve ambiguous
      pronouns in sentences.
    eval_split: validation
    extra_params: {}
    few_shot_num: 0
    few_shot_prompt_template: null
    few_shot_random: false
    filters: null
    force_redownload: false
    metric_list:
    - acc
    name: winogrande
    output_types:
    - generation
    pretty_name: Winogrande
    prompt_template: 'Answer the following multiple choice question. The entire content
      of your response should be of the following format: ''ANSWER: [LETTER]'' (without
      quotes) where [LETTER] is one of {letters}.


      {question}


      {choices}'
    query_template: null
    review_timeout: null
    sandbox_config: {}
    shuffle: false
    shuffle_choices: false
    subset_list:
    - default
    system_prompt: null
    tags:
    - Reasoning
    - MCQ
    train_split: null
dataset_dir: xxx/.cache/modelscope/hub/datasets
dataset_hub: modelscope
datasets:
- gsm8k
- mmlu
- piqa
- siqa
- hellaswag
- winogrande
- arc
debug: false
eval_backend: Native
eval_batch_size: 1
eval_config: null
eval_type: llm_ckpt
evalscope_version: 1.3.0
generation_config:
  batch_size: 1
  do_sample: false
  max_tokens: 2048
  n: 1
  temperature: 1.0
  top_k: 50
  top_p: 1.0
ignore_errors: false
judge_model_args: {}
judge_strategy: auto
judge_worker_num: 1
limit: null
model: ChameleonInterleavedModel
model_args:
  precision: torch.float16
  revision: master
model_id: chameleon-7b
model_task: text_generation
repeats: 1
rerun_review: false
sandbox_manager_config: {}
sandbox_type: docker
seed: 42
stream: null
timeout: null
use_cache: null
use_sandbox: false
work_dir: ./outputs/20251203_081230

```


Eval code:
```python
from typing import List, Optional, Dict, Any
import torch
from PIL import Image
import base64
import io
from transformers import ChameleonProcessor, ChameleonForConditionalGeneration

from evalscope.api.model import ModelAPI, GenerateConfig, ModelOutput
from evalscope.api.messages import ChatMessage
from evalscope.api.tool import ToolChoice, ToolInfo
from evalscope.api.registry import register_model_api


@register_model_api(name='chameleon')
class ChameleonModel(ModelAPI):

    def __init__(
        self,
        model_name: str,
        base_url: Optional[str] = None,
        api_key: Optional[str] = None,
        config: GenerateConfig = GenerateConfig(),
        model_path: str = "xxx/models/chameleon-7b-hf",
        device_map: str = "auto",
        dtype: torch.dtype = torch.bfloat16,
        **model_args: Dict[str, Any],
    ) -> None:
        super().__init__(model_name, base_url, api_key, config)
        self.model_args = model_args
        self.model_path = model_path
        self.device_map = device_map
        self.dtype = dtype

        self._load_model()

    def _load_model(self):
        try:
            print(f"Loading Chameleon model from {self.model_path}")
            self.processor = ChameleonProcessor.from_pretrained(self.model_path)
            self.model = ChameleonForConditionalGeneration.from_pretrained(
                self.model_path,
                dtype=self.dtype,
                device_map=self.device_map
            )
            print("Model loaded successfully")
        except Exception as e:
            print(f"Error loading model: {e}")
            raise

    def generate(
        self,
        input: List[ChatMessage],
        tools: List[ToolInfo] = None,
        tool_choice: ToolChoice = None,
        config: GenerateConfig = None,
    ) -> ModelOutput:
        try:
            gen_config = config if config is not None else self.config

            text_prompt, images = self._process_messages(input)

            if images:
                inputs = self.processor(
                    images=images[0] if len(images) == 1 else images,
                    text=text_prompt,
                    return_tensors="pt"
                ).to(self.model.device, dtype=self.dtype)
            else:
                inputs = self.processor(
                    text=text_prompt,
                    return_tensors="pt"
                ).to(self.model.device, dtype=self.dtype)

            generation_kwargs = {
                'max_new_tokens': getattr(gen_config, 'max_tokens', 200),
                'temperature': getattr(gen_config, 'temperature', 1.0),
                'do_sample': getattr(gen_config, 'temperature', 1.0) > 0,
            }

            if generation_kwargs['temperature'] == 0:
                generation_kwargs['do_sample'] = False
                generation_kwargs.pop('temperature')

            with torch.no_grad():
                output = self.model.generate(**inputs, **generation_kwargs)

            generated_text = self.processor.decode(output[0], skip_special_tokens=True)

            if text_prompt in generated_text:
                response = generated_text[len(text_prompt):].strip()
            else:
                response = generated_text.strip()

            return ModelOutput.from_content(
                model=self.model_name,
                content=response
            )

        except Exception as e:
            print(f"Error during generation: {e}")
            return ModelOutput.from_content(
                model=self.model_name,
                content=f"Error: {str(e)}"
            )

    def _process_messages(self, messages: List[ChatMessage]) -> tuple[str, List[Image.Image]]:
        text_parts = []
        images = []

        for message in messages:
            role = getattr(message, 'role', 'user')
            content = getattr(message, 'content', str(message))

            if isinstance(content, list):
                for item in content:
                    if isinstance(item, dict):
                        if item.get('type') == 'text':
                            text_parts.append(f"{role}: {item.get('text', '')}")
                        elif item.get('type') == 'image_url':
                            image = self._load_image_from_url_or_base64(item.get('image_url', {}).get('url', ''))
                            if image:
                                images.append(image)
                                text_parts.append("<image>")
                    else:
                        text_parts.append(f"{role}: {str(item)}")
            elif isinstance(content, str):
                if '<image>' in content or any(ext in content.lower() for ext in ['.jpg', '.jpeg', '.png', '.gif', '.bmp']):
                    image_path = self._extract_image_path(content)
                    if image_path:
                        image = self._load_image_from_path(image_path)
                        if image:
                            images.append(image)

                text_parts.append(f"{role}: {content}")
            else:
                text_parts.append(f"{role}: {str(content)}")

        text_prompt = '\n'.join(text_parts)

        return text_prompt, images

    def _extract_image_path(self, text: str) -> Optional[str]:
        import re

        pattern = r'([^\s]+\.(?:jpg|jpeg|png|gif|bmp))'
        matches = re.findall(pattern, text, re.IGNORECASE)

        if matches:
            return matches[0]
        return None

    def _load_image_from_path(self, image_path: str) -> Optional[Image.Image]:
        try:
            return Image.open(image_path).convert('RGB')
        except Exception as e:
            print(f"Error loading image from path {image_path}: {e}")
            return None

    def _load_image_from_url_or_base64(self, url_or_data: str) -> Optional[Image.Image]:
        try:
            if url_or_data.startswith('data:image'):
                header, data = url_or_data.split(',', 1)
                image_data = base64.b64decode(data)
                return Image.open(io.BytesIO(image_data)).convert('RGB')
            elif url_or_data.startswith('http'):
                import requests
                response = requests.get(url_or_data, stream=True)
                return Image.open(response.raw).convert('RGB')
            else:
                return Image.open(url_or_data).convert('RGB')
        except Exception as e:
            print(f"Error loading image from {url_or_data}: {e}")
            return None


if __name__ == "__main__":
    from evalscope import run_task, TaskConfig

    chameleon_model = ChameleonModel(
        model_name='chameleon-7b',
        model_path="xxx/models/chameleon-7b-hf"
    )

    task_config = TaskConfig(
        model=chameleon_model,
        datasets=['gsm8k', 'mmlu', 'piqa', 'siqa', 'hellaswag', 'winogrande', 'arc'],
    )

    results = run_task(task_cfg=task_config)
```


Hello @IRONICBo ,

Thank you for reporting this discrepancy and providing detailed configuration information. I've reviewed your evaluation setup and have some suggestions to help identify the root cause of the performance gap.


1. Please verify that your model is generating responses in the exact format expected by each benchmark. For example:
- ARC/HellaSWAG/MMLU: Ensure responses strictly follow "ANSWER: [LETTER]" format
- GSM8K: Verify step-by-step reasoning and final answer in \boxed{} format

2. You are using a greedy strategy, please check the official recommended generation config.


We're here to help you resolve this. Looking forward to your findings!


> Hello [@IRONICBo](https://github.com/IRONICBo) ,
> 
> Thank you for reporting this discrepancy and providing detailed configuration information. I've reviewed your evaluation setup and have some suggestions to help identify the root cause of the performance gap.
> 
> 1. Please verify that your model is generating responses in the exact format expected by each benchmark. For example:
> 
> * ARC/HellaSWAG/MMLU: Ensure responses strictly follow "ANSWER: [LETTER]" format
> * GSM8K: Verify step-by-step reasoning and final answer in \boxed{} format
> 
> 2. You are using a greedy strategy, please check the official recommended generation config.
> 
> We're here to help you resolve this. Looking forward to your findings!

Thank you very much for your suggestions. I will double-check everything.