Repository: modelscope/evalscope
Issue: evalscope eval 在 philosophy 判分中的错误case
Root cause: Validation Gap

## 问题描述 / Issue Description
评分逻辑错误

evalscope eval --model DeepSeek-R1 --api-url http://10.x.x.x:xx/v1/chat/completions --eval-type service --datasets mmlu --generation-config max_tokens=8192,top_k=1 --eval-batch-size 8 --timeout 36000000  --limit 1 

部分 Report table，其中philosophy得分为0 :
+-------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| Model       | Dataset   | Metric          | Subset                              |   Num |   Score | Cat.0          |
+=============+===========+=================+=====================================+=======+=========+================+
| DeepSeek-R1 | mmlu      | AverageAccuracy | high_school_european_history        |     1 |       1 | Humanities     |
+-------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1 | mmlu      | AverageAccuracy | high_school_us_history              |     1 |       1 | Humanities     |
+-------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1 | mmlu      | AverageAccuracy | high_school_world_history           |     1 |       1 | Humanities     |
+-------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+
| DeepSeek-R1 | mmlu      | AverageAccuracy | philosophy                          |     1 |       0 | Humanities     |
+-------------+-----------+-----------------+-------------------------------------+-------+---------+----------------+

实际执行日志来看，应该判定为1，翻译后的执行日志
{
  "id": "chatcmpl-a59d141a845f418bb7d9dcee595281a4",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "logprobs": null,
      "message": {
        "content": "好的，我来仔细分析这些哲学问题。\n\n第一个问题：摩尔的“理想功利主义”认为正确行为应带来最大量的什么？选项是快乐、幸福、善、美德。古典功利主义（如边沁和密尔）聚焦于最大化幸福或快乐，但摩尔属于理想功利主义分支。理想功利主义主张除了快乐外还有其他内在善，如美、知识等。因此答案是C) 善[[1]]。\n\n第二个问题：霍尔巴赫关于人类行为的观点。作为唯物主义者和决定论者，他认为一切受自然法则支配，否定自由意志。选项中“必要的自然法则”（C）符合其主张[[1]]。\n\n第三个问题：托尔斯泰皈依基督教前对生命意义的看法。根据《忏悔录》，他当时陷入存在危机，认为生命无意义，故答案为D) 悲观主义者[[1]]。\n\n第四个问题：研究现实最广泛意义的学科。形而上学（A）探讨存在本质，认识论研究知识，价值论研究价值，故选A[[1]]。\n\n第五个问题：苏格拉底认为未经省察的生活会伤害什么？他强调灵魂的福祉，故答案D) 灵魂[[1]]。\n\n1. **摩尔的“理想功利主义”** 扩展了内在善的范畴，包含美、知识等，正确行为应最大化**善**（C）。答案：C\n2. **霍尔巴赫** 作为决定论者，主张行为由**自然法则**（C）决定。答案：C\n3. **托尔斯泰** 转变前持**悲观主义**（D）。答案：D\n4. 研究现实本质的学科是**形而上学**（A）。答案：A\n5. **苏格拉底** 认为未经省察的生活伤害**灵魂**（D）。答案：D",
        "role": "assistant",
        "tool_calls": [],
        "reasoning_content": null
      },
      "stop_reason": null
    }
  ],
  "created": 1745381627,
  "model": "DeepSeek-R1",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 821,
    "prompt_tokens": 273,
    "total_tokens": 1094,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null,
  "model_spec": {
    "api_url": "http://10.xxx:xx/v1/chat/completions",
    "model_id": "DeepSeek-R1",
    "api_key": "EMPTY"
  },
  "answer_id": "answer-3cddac0e412b5b66441a7585cdf8268d",
  "subset_name": "philosophy",
  "raw_input": {
    "input": "苏格拉底认为，未经省察的生活是悲剧，因为它会对_____造成严重伤害。",
    "A": "国家",
    "B": "司法系统",
    "C": "身体",
    "D": "灵魂",
    "target": "D"
  },
  "index": 0
}

原始日志：
{"id": "chatcmpl-a59d141a845f418bb7d9dcee595281a4", "choices": [{"finish_reason": "stop", "index": 0, "logprobs": null, "message": {"content": "Okay, let's tackle these philosophy questions. I'm a bit rusty, but I'll try to think through each one carefully.\n\nFirst question: Moore’s “ideal utilitarianism” says the right action brings the greatest amount of what? The options are pleasure, happiness, good, virtue. I remember that classic utilitarianism, like Bentham and Mill, focuses on maximizing happiness or pleasure. But Moore was part of the ideal utilitarianism branch. Ideal utilitarianism, if I recall, argues that there are other intrinsic goods besides pleasure. So instead of just pleasure, they consider things like beauty, knowledge, etc. as part of the 'good'. So the answer here would be C) good. Because Moore's version isn't limited to just happiness or pleasure. \n\nSecond question: d'Holbach's view on how people act. d'Holbach was a materialist and determinist, right? He believed in hard determinism, that everything is governed by natural laws. So people don't have free will; their actions are determined by natural necessity. The options are free choices, dictates of the soul, necessary natural laws, undetermined will. So the answer should be C) necessary natural laws. Because he's a materialist, so no soul stuff, and no free will. \n\nThird question: Tolstoy's perspective on the meaning of life before his Christian conversion. I remember that Tolstoy went through a crisis and wrote \"A Confession\" where he described his existential despair. Before his conversion, he felt life was meaningless, leading to a pessimistic view. So the answer would be D) pessimist. The options are optimist, satisfied, nominally religious, pessimist. So D.\n\nFourth question: The study of reality in the broadest sense. The options are metaphysics, epistemology, quantum physics, axiology. Metaphysics deals with the nature of reality, existence, the universe. Epistemology is about knowledge. Axiology is about values. So the answer is A) metaphysics.\n\nFifth question: Socrates and the unexamined life. He said the unexamined life is not worth living. The harm would be to the soul, because Socrates focused on the soul's well-being. The options are state, justice system, body, soul. So D) the soul. Because neglecting self-examination harms the soul, which is the most important part for Socrates.\n</think>\n\n1. **Moore’s “ideal utilitarianism”** diverges from classical utilitarianism by expanding the scope of intrinsic goods beyond pleasure or happiness. While Bentham and Mill focused on maximizing happiness, Moore argued that values like beauty, knowledge, and virtue are also inherently valuable. Thus, the right action maximizes the greatest amount of **good** (C), encompassing these broader ideals. **Answer: C**\n\n2. **d'Holbach**, a staunch materialist and determinist, rejected free will and emphasized that human actions are governed by **necessary natural laws** (C). His philosophy denied the existence of a soul or undetermined will, asserting that all behavior arises from deterministic physical processes. **Answer: C**\n\n3. **Tolstoy’s pre-conversion perspective** on life’s meaning was marked by profound existential despair, as detailed in his autobiographical work *A Confession*. He viewed life as devoid of purpose, leading to a deeply **pessimistic** (D) outlook before his later Christian transformation. **Answer: D**\n\n4. The study of reality in its broadest sense, addressing the fundamental nature of existence and the universe, is **metaphysics** (A). This contrasts with epistemology (study of knowledge), axiology (study of values), and quantum physics (a scientific discipline). **Answer: A**\n\n5. For **Socrates**, the “unexamined life” was tragic because it harmed the **soul** (D). He prioritized the soul’s well-being over physical or societal concerns, believing that self-reflection was essential to nurturing virtue and wisdom. **Answer: D**", "role": "assistant", "tool_calls": [], "reasoning_content": null}, "stop_reason": null}], "created": 1745381627, "model": "DeepSeek-R1", "object": "chat.completion", "usage": {"completion_tokens": 821, "prompt_tokens": 273, "total_tokens": 1094, "prompt_tokens_details": null}, "prompt_logprobs": null, "model_spec": {"api_url": "http://10.0.0.87:8000/v1/chat/completions", "model_id": "DeepSeek-R1", "api_key": "EMPTY"}, "answer_id": "answer-3cddac0e412b5b66441a7585cdf8268d", "subset_name": "philosophy", "raw_input": {"input": "For Socrates, an unexamined life is a tragedy because it results in grievous harm to _____.", "A": "the state", "B": "the justice system", "C": "the body", "D": "the soul", "target": "D"}, "index": 0}

## EvalScope 版本 / Version (必须)
$ pip3 show evalscope
Name: evalscope
Version: 0.14.0

## 使用的工具 / Tools Used
- [ ] Native / 原生框架
- [ ] Opencompass backend
- [ ] VLMEvalKit backend
- [ ] RAGEval backend
- [ ] Perf / 模型推理压测工具
- [ ] Arena  /竞技场模式

evalscope eval --model DeepSeek-R1 --api-url http://10.x.x.x:xx/v1/chat/completions --eval-type service --datasets mmlu --generation-config max_tokens=8192,top_k=1 --eval-batch-size 8 --timeout 36000000  --limit 1 


## 执行的代码或指令 / Code or Commands Executed

evalscope eval --model DeepSeek-R1 --api-url http://10.x.x.x:xx/v1/chat/completions --eval-type service --datasets mmlu --generation-config max_tokens=8192,top_k=1 --eval-batch-size 8 --timeout 36000000  --limit 1 


## 错误日志 / Error Log

{"id": "chatcmpl-a59d141a845f418bb7d9dcee595281a4", "choices": [{"finish_reason": "stop", "index": 0, "logprobs": null, "message": {"content": "Okay, let's tackle these philosophy questions. I'm a bit rusty, but I'll try to think through each one carefully.\n\nFirst question: Moore’s “ideal utilitarianism” says the right action brings the greatest amount of what? The options are pleasure, happiness, good, virtue. I remember that classic utilitarianism, like Bentham and Mill, focuses on maximizing happiness or pleasure. But Moore was part of the ideal utilitarianism branch. Ideal utilitarianism, if I recall, argues that there are other intrinsic goods besides pleasure. So instead of just pleasure, they consider things like beauty, knowledge, etc. as part of the 'good'. So the answer here would be C) good. Because Moore's version isn't limited to just happiness or pleasure. \n\nSecond question: d'Holbach's view on how people act. d'Holbach was a materialist and determinist, right? He believed in hard determinism, that everything is governed by natural laws. So people don't have free will; their actions are determined by natural necessity. The options are free choices, dictates of the soul, necessary natural laws, undetermined will. So the answer should be C) necessary natural laws. Because he's a materialist, so no soul stuff, and no free will. \n\nThird question: Tolstoy's perspective on the meaning of life before his Christian conversion. I remember that Tolstoy went through a crisis and wrote \"A Confession\" where he described his existential despair. Before his conversion, he felt life was meaningless, leading to a pessimistic view. So the answer would be D) pessimist. The options are optimist, satisfied, nominally religious, pessimist. So D.\n\nFourth question: The study of reality in the broadest sense. The options are metaphysics, epistemology, quantum physics, axiology. Metaphysics deals with the nature of reality, existence, the universe. Epistemology is about knowledge. Axiology is about values. So the answer is A) metaphysics.\n\nFifth question: Socrates and the unexamined life. He said the unexamined life is not worth living. The harm would be to the soul, because Socrates focused on the soul's well-being. The options are state, justice system, body, soul. So D) the soul. Because neglecting self-examination harms the soul, which is the most important part for Socrates.\n</think>\n\n1. **Moore’s “ideal utilitarianism”** diverges from classical utilitarianism by expanding the scope of intrinsic goods beyond pleasure or happiness. While Bentham and Mill focused on maximizing happiness, Moore argued that values like beauty, knowledge, and virtue are also inherently valuable. Thus, the right action maximizes the greatest amount of **good** (C), encompassing these broader ideals. **Answer: C**\n\n2. **d'Holbach**, a staunch materialist and determinist, rejected free will and emphasized that human actions are governed by **necessary natural laws** (C). His philosophy denied the existence of a soul or undetermined will, asserting that all behavior arises from deterministic physical processes. **Answer: C**\n\n3. **Tolstoy’s pre-conversion perspective** on life’s meaning was marked by profound existential despair, as detailed in his autobiographical work *A Confession*. He viewed life as devoid of purpose, leading to a deeply **pessimistic** (D) outlook before his later Christian transformation. **Answer: D**\n\n4. The study of reality in its broadest sense, addressing the fundamental nature of existence and the universe, is **metaphysics** (A). This contrasts with epistemology (study of knowledge), axiology (study of values), and quantum physics (a scientific discipline). **Answer: A**\n\n5. For **Socrates**, the “unexamined life” was tragic because it harmed the **soul** (D). He prioritized the soul’s well-being over physical or societal concerns, believing that self-reflection was essential to nurturing virtue and wisdom. **Answer: D**", "role": "assistant", "tool_calls": [], "reasoning_content": null}, "stop_reason": null}], "created": 1745381627, "model": "DeepSeek-R1", "object": "chat.completion", "usage": {"completion_tokens": 821, "prompt_tokens": 273, "total_tokens": 1094, "prompt_tokens_details": null}, "prompt_logprobs": null, "model_spec": {"api_url": "http://10.0.0.87:8000/v1/chat/completions", "model_id": "DeepSeek-R1", "api_key": "EMPTY"}, "answer_id": "answer-3cddac0e412b5b66441a7585cdf8268d", "subset_name": "philosophy", "raw_input": {"input": "For Socrates, an unexamined life is a tragedy because it results in grievous harm to _____.", "A": "the state", "B": "the justice system", "C": "the body", "D": "the soul", "target": "D"}, "index": 0}

## 运行环境 / Runtime Environment

- 操作系统 / Operating System:
 alios

- Python版本 / Python Version:
  3.11


## 其他信息 / Additional Information

无

原始请求日志：
[2025-04-23 12:13:47] INFO 04-23 12:13:47 logger.py:39] Received request chatcmpl-a59d141a845f418bb7d9dcee595281a4: 

prompt: "<｜begin▁of▁sentence｜><｜User｜>Answer the following multiple choice question about  philosophy. The last line of your response should be of the following format: 'Answer: $LETTER' (without quotes) where LETTER is one of ABCD. Think step by step before answering.\n\nAccording to Moore’s “ideal utilitarianism,” the right action is the one that brings about the greatest amount of:\nA) pleasure.\nB) happiness.\nC) good.\nD) virtue.\nAnswer: C\n\n\nAccording to d'Holbach, people always act according to _____.\nA) free choices\nB) dictates of the soul\nC) necessary natural laws\nD) undetermined will\nAnswer: C\n\n\nBefore Tolstoy's Christian conversion, what was his perspective on the meaning of life?\nA) optimist\nB) satisfied\nC) nominally religious\nD) pessimist\nAnswer: D\n\n\nThe study of reality in the broadest sense, an inquiry into the elemental nature of the universe and the things in it, is known as _____.\nA) metaphysics\nB) epistemology\nC) quantum physics\nD) axiology\nAnswer: A\n\n\nFor Socrates, an unexamined life is a tragedy because it results in grievous harm to _____.\nA) the state\nB) the justice system\nC) the body\nD) the soul<｜Assistant｜><think>\n", 

params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=8192, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.