Repository: open-compass/opencompass
Issue: [Feature] Save bad cases when evaluating datasets
Root cause: Unimplemented Feature Gap

### Describe the feature

Currently I can only get the scores of datasets. However, only have the score is not clear that in what aspect / cases the model is not good enough.

I think opencompass should have the function that extract those bad cases in order to deeply analysis models.

I am glad to implement this feature, but I think I need some guidance on how to do this, thanks!

### Will you implement it?

- [ ] I would like to implement this feature and create a PR!

Very welcome! You can have a draft and I can provide your help

I think you can try to add this feature in summarizer, for example, use --show-bad-case to control it, and when turn off, the summarizer will get the score generally, when turn on, the summarizer will also write down the bad cases.

> I think you can try to add this feature in summarizer, for example, use --show-bad-case to control it, and when turn off, the summarizer will get the score generally, when turn on, the summarizer will also write down the bad cases.

Thanks, I will research on it.

I use this demo for test:

```
from mmengine.config import read_base

with read_base():
    from .datasets.cmmlu.cmmlu_gen import cmmlu_datasets
    
_meta_template = dict(
    round=[
        dict(role="HUMAN", begin='
user\n', end='
\n'),
        dict(role="BOT", begin="
assistant\n", end='
\n', generate=True),
    ],
    eos_token_id=151645,
)

GPU_NUMS = 4

from opencompass.models import VLLM

models = [
    dict(
        type=VLLM,
        abbr='models--Qwen--Qwen1.5-1.8B-Chat',
        path='/maindata/data/shared/Security-SFT/hf_models/hub/models--Qwen--Qwen1.5-1.8B-Chat/snapshots/3aede71902ad578aac72678f9f8b6199ca6ab53b',
        model_kwargs=dict(tensor_parallel_size=GPU_NUMS),
        meta_template=_meta_template,
        max_out_len=100,
        max_seq_len=2048,
        batch_size=GPU_NUMS * 8,
        generation_kwargs=dict(temperature=0),
        end_str='
',
        run_cfg=dict(num_gpus=GPU_NUMS, num_procs=1),
    ),
]

datasets = [*cmmlu_datasets]
```

the cmmlu eval config:

```
cmmlu_eval_cfg = dict(
    evaluator=dict(type=AccEvaluator),
    pred_postprocessor=dict(type=first_capital_postprocess))
```

this use `AccEvaluator`, and summarizer is `DefaultSummarizer`

in `DefaultSummarizer` class `_pick_up_results` function, you can only get raw results like this:

```
raw_results: 
{'models--Qwen--Qwen1.5-1.8B-Chat': {'cmmlu-agronomy': {'accuracy': 42.01183431952663}, 'cmmlu-anatomy': {'accuracy': 47.2972972972973}, 'cmmlu-ancient_chinese': {'accuracy': 27.439024390243905}, 'cmmlu-arts': {'accuracy': 69.375}, 'cmmlu-astronomy': {'accuracy': 29.09090909090909}, 'cmmlu-business_ethics': {'accuracy': 53.588516746411486}, 'cmmlu-chinese_civil_service_exam': {'accuracy': 45.0}, 'cmmlu-chinese_driving_rule': {'accuracy': 77.86259541984732}, 'cmmlu-chinese_food_culture': {'accuracy': 43.38235294117647}, 'cmmlu-chinese_foreign_policy': {'accuracy': 62.616822429906534}, 'cmmlu-chinese_history': {'accuracy': 71.8266253869969}, 'cmmlu-chinese_literature': {'accuracy': 40.19607843137255}, 'cmmlu-chinese_teacher_qualification': {'accuracy': 70.39106145251397}, 'cmmlu-clinical_knowledge': {'accuracy': 48.52320675105485}, 'cmmlu-college_actuarial_science': {'accuracy': 3.7735849056603774}, 'cmmlu-college_education': {'accuracy': 62.616822429906534},
```

only accuracy.

This is because in `Evaluator` class `scores` function, it seems use huggingface `evaluate` package only calculate scores by `scores = metric.compute(**self._preprocess(predictions, references))`, so if I want to add this save bad cases function, I need to rewrite a new `metric.compute`?

seems that I need to add this in `score` function of all `Evaluator` classes'? I wonder if my thought is right.

> seems that I need to add this in `score` function of all `Evaluator` classes'? I wonder if my thought is right.

well, yes, I think you'd better to support this feature in `Evaluator` better than in `Summarizer`. Your thought is totally right, (for objective evaluation), but for subjective evaluation, we are planning to reconstruct the pipeline so you can ignore for it.
And you can check the code in here https://github.com/open-compass/opencompass/blob/c1724013233bfb980b3cfa2a12f75d3bea38865a/opencompass/cli/main.py#L127
when use `--dump-eval-details`, opencompass will save all detailed results in the process of `Evaluator`, so you can reference it and add a new feature to save just bad case results.

Hi, I have forgot there is a tool named 'case_analyzer' in the early period, so I think if you want to support this feature, you may follow this tool, more information see in here https://github.com/open-compass/opencompass/issues/1086
Thanks again for your contribution.

We may plan to support this feature in the future, but due to a large number of works, we may not be able to provide much assistance in the near term.