Repository: modelscope/evalscope
Issue: hellaswag 相关问题
Root cause: Algorithmic Error

## 使用的工具 / Tools Used
- [x] Native / 原生框架
- [ ] Opencompass backend
- [ ] VLMEvalKit backend
- [ ] RAGEval backend
- [ ] Perf / 模型推理压测工具
- [ ] Arena  /竞技场模式

## 问题描述 / Issue Description

最近在基于 api 的评测（`eval_type=EvalType.SERVICE`）中发现模型在 hellaswag 上跑不出结果且耗时非常长，预测的数据基本都答非所问、比较冗长，对比了基于 OpenCompass 后端在 hellaswag 上进行评测的结果，发现 EvalScope 原生支持的 hellaswag 中没有把它当成一个多选任务：
```python
# benchmarks/hellaswag/hellaswag_adapter.py
    model_adapter=OutputType.CONTINUOUS,
    output_types=[OutputType.CONTINUOUS, OutputType.GENERATION],
```
而是跟 truthful_qa 一样作为了基于 logits 的评测，但在文档中（https://evalscope.readthedocs.io/zh-cn/latest/get_started/supported_dataset.html#id2 ）没有提及，文档中只注明了 truthful_qa 这一项。

另外，注册时的 `prompt_template` 也没有用到：
```python
# benchmarks/hellaswag/hellaswag_adapter.py
    prompt_template=
    'Respond with the index of sentence that makes the most sense, chose from 0, 1, 2, 3, derive your final answer as `The answer is ...`.',  # noqa: E501
```

想问一下这是否应该和 OpenCompass 一样将 hellaswag 作为单纯的多选任务进行评测？暂时不太明白为什么要作为基于 logits 的评测实现。

## 补充 / Supplementary

将 `hellaswag_adapter.py` 的评测修改为多选类型后进行评测，结果略低于 OpenCompass 后端测出的。

Qwen2.5-1.5B-Instruct 评测结果：
- 修改后原生后端：56.28 acc
- OpenCompass 后端：61.74 acc

具体修改：
```python
# Copyright (c) Alibaba, Inc. and its affiliates.
import numpy as np
import os
import re

from evalscope.benchmarks import Benchmark, DataAdapter
from evalscope.constants import EvalType, OutputType
from evalscope.metrics import exact_match
from evalscope.utils.io_utils import jsonl_to_list
from evalscope.utils.logger import get_logger
from evalscope.utils.utils import ResponseParser

# flake8: noqa

logger = get_logger()


@Benchmark.register(
    name='hellaswag',
    pretty_name='HellaSwag',
    dataset_id='modelscope/hellaswag',
    model_adapter=OutputType.MULTIPLE_CHOICE,
    output_types=[OutputType.MULTIPLE_CHOICE, OutputType.GENERATION],
    subset_list=['default'],
    metric_list=['AverageAccuracy'],
    few_shot_num=0,
    train_split='train',
    eval_split='validation',
    prompt_template='{query}',  # noqa: E501
)
class HellaSwagAdapter(DataAdapter):

    def __init__(self, **kwargs):

        few_shot_num = kwargs.get('few_shot_num', 0)
        if few_shot_num != 0:
            logger.warning(f'few_shot_num should be 0 for HellaSwag, but got {few_shot_num}. Use 0-shot by default.')
            kwargs['few_shot_num'] = 0

        super().__init__(**kwargs)
        self.choices = ['0', '1', '2', '3']

    def load_from_disk(self, dataset_name_or_path, subset_list, work_dir, **kwargs) -> dict:
        data_dict = {}
        for subset_name in subset_list:
            data_dict[subset_name] = {}
            for split in [self.train_split, self.eval_split]:
                if os.path.exists(dataset_name_or_path):
                    file_path = os.path.join(dataset_name_or_path, f'hellaswag_{split}.jsonl')
                else:
                    file_path = os.path.join(work_dir, dataset_name_or_path, f'hellaswag_{split}.jsonl')
                if os.path.exists(file_path):
                    data_dict[subset_name][split] = jsonl_to_list(file_path)

        return data_dict

    def gen_prompt(self, input_d: dict, few_shot_list: list, **kwargs) -> dict:
        """
        Generate model prompt from raw data, unify the prompt format for HellaSwag benchmark.

        Args:
            input_d (dict): The raw input. A single data format of the HellaSwag:

            {
                'ind': 4,
                'activity_label': 'Removing ice from car',
                'ctx_a': 'Then, the man writes over the snow covering the window of a car, and a woman wearing winter clothes smiles.',
                'ctx_b': 'then',
                'ctx': 'Then, the man writes over the snow covering the window of a car, and a woman wearing winter clothes smiles. then',
                'endings': [', the man adds wax to the windshield and cuts it.', ', a person board a ski lift, while two men supporting the head of the person wearing winter clothes snow as the we girls sled.', ', the man puts on a christmas coat, knitted with netting.', ', the man continues removing the snow on his car.'],
                'source_id': 'activitynet~v_-1IBHYS3L-Y',
                'split': 'train',
                'split_type': 'indomain',
                'label': '3'
            }

        Returns:
            Refer to function: evalscope.benchmarks.data_adapter.DataAdapter.gen_prompt for details.
        """

        endings: list = [self._preprocess(ending) for ending in input_d['endings']]

        few_shot_prompts = [
            self._generate_prompt(input_d=sample, endings=endings, include_answer=True) for sample in few_shot_list
        ]
        context: str = '\n'.join(few_shot_prompts) + '\n'
        query = context.strip() + self._generate_prompt(input_d=input_d, endings=endings, include_answer=False)

        full_prompt = self.prompt_template.format(query=query)
        return self.gen_prompt_data(full_prompt)

    def get_gold_answer(self, input_d: dict) -> str:
        # Get the gold choice
        return input_d['label']

    def parse_pred_result(self, result: list, raw_input_d: dict = None, eval_type: str = EvalType.CHECKPOINT) -> str:
        """
        Parse the model output to get the answer. Could be the best choice index.

        Args:
            result: Predicted answer from the model. Usually a string for chat.
            raw_input_d: The raw input dict.
            eval_type: The evaluation type. e.g. checkpoint, service, custom.

        Returns:
            The parsed answer. Depending on the dataset. Usually a string for chat.
        """
        if self.model_adapter == OutputType.MULTIPLE_CHOICE:
            return result
        else:
            return ResponseParser.parse_first_option_with_choices(result, self.choices)

    def match(self, gold: str, pred: str) -> float:
        return exact_match(gold=str(gold), pred=str(pred))

    @classmethod
    def _preprocess(cls, text):
        text = text.strip()
        text = text.replace(' [title]', '. ')
        text = re.sub('\\[.*?\\]', '', text)
        text = text.replace('  ', ' ')
        return text

    @classmethod
    def _generate_prompt(cls, input_d: dict, endings: list, include_answer=True) -> str:
        """
        Generate prompt for HellaSwag dataset.

        Args:
            input_d: a single data of the hellaswag.
            endings:  preprocessed endings
            include_answer: bool

        Returns:

        """

        ctx = input_d['ctx_a'] + ' ' + input_d['ctx_b'].capitalize()
        # example: str = cls._preprocess(input_d['activity_label'] + ': ' + ctx)
        example: str = cls._preprocess(ctx)

        example += "\nQuestion: Which ending makes the most sense?"
        for i, ending in enumerate(endings):
            example += f'\n{i}. {ending}'
        example += "\nYou may choose from 0, 1, 2, 3.\nAnswer:"

        if include_answer:
            example += '{}\n\n'.format(endings[int(input_d['label'])])

        return example

```

修改后传给接口的输入：
```

system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.


user\
How to strop a straight razor. Acquire a strop. These are relatively simple to get. You can get cheaper models from general stores online.
Question: Which ending makes the most sense?
0. These might cost you anything from $5 to $25. If you've been using a straight razor for a while, you'll want to invest in a strop that will last longer.
1. They are designed to be effective for a range of purposes except shaving. They are ideal for when you want to really imagine shaving off the hair.
2. Here's what one could buy : A strop is simply to simply hold one blade upright in one hand. Flat, sharp edges attach the blade to the body of the razor.
3. Look for one that has crimp at the ends. Crimp is a soft, feminine material.\n
You may choose from 0, 1, 2, 3.
Answer:


assistant
```

感谢你的反馈！我们将修复此问题