Repository: modelscope/evalscope
Issue: deepseek-R1系列模型使用EvalType.SERVICE时，提取的答案和回复中的答案不一致导致精度很低
Root cause: Validation Gap

比如这一条：
{"id": "cmpl-1392aa00bfc04c67a4890cbf04bc7f83", "choices": [{"finish_reason": "stop", "index": 0, "message": {"content": "Okay, let's tackle this Voltaire question. The user provided a passage from Voltaire's letter about the 1755 Lisbon earthquake and asks which characteristic of Enlightenment intellectuals it illustrates. The options are A, B, C, D.\n\nFirst, I need to recall what each option is about. Option A mentions Enlightenment intellectuals as accomplished scientists contributing to understanding the universe. Voltaire was a philosopher and writer, not primarily a scientist, though he did engage with scientific ideas. The passage here is more about the implications of natural disasters on human society and religion, not about scientific discovery per se. So maybe A isn't the right answer.\n\nOption B talks about new communication methods like salons and pamphlets. The passage is a letter, which was a common way intellectuals communicated. However, the question is about the ideas expressed, not the medium. The content of the letter is criticizing the Inquisition and the cruelty of nature, which relates more to social and religious issues. So B might not be the best fit here.\n\nOption C states that most rejected religion for atheism. Voltaire was critical of organized religion, especially the Catholic Church and the Inquisition, as seen in the passage. However, he wasn't an atheist; he was a deist, believing in a distant, non-intervening God. The passage mocks the Inquisition but doesn't outright reject all religion. So C is probably incorrect.\n\nOption D suggests that scientific discoveries led to a more tolerant and objective approach. The passage uses the Lisbon earthquake, a natural disaster, to criticize both the cruelty of nature and human institutions like the Inquisition. Voltaire is arguing against persecution and promoting tolerance, using reason and observation (Enlightenment ideals) to challenge traditional authorities. This aligns with D, as the scientific method and rational thought during the Enlightenment encouraged questioning of social and religious norms, leading to calls for tolerance and reform.\n\nSo the correct answer should be D.\n</think>\n\nThe ideas expressed by Voltaire in his letter about the 1755 Lisbon earthquake best illustrate **D. Many believed that the new scientific discoveries justified a more tolerant and objective approach to social and cultural issues**. \n\nVoltaire uses the earthquake as a catalyst to critique both the harshness of nature and human institutions like the Inquisition. By invoking Enlightenment principles of reason and observation, he challenges religious persecution and advocates for tolerance. This reflects how scientific rationality during the period encouraged questioning of traditional authorities and promoted social reform. \n\n**Answer: D**", "role": "assistant"}, "review": {"gold": "D", "pred": "A", "result": 0}}], "created": 88265, "model": "deepseek-ai/DeepSeek-R1", "object": "chat.completion", "usage": {"completion_tokens": 501, "prompt_tokens": 2638, "total_tokens": 3139}, "model_spec": {"api_url": "http://175.0.0.18:36667/v1/chat/completions", "model_id": "deepseek-ai/DeepSeek-R1", "api_key": "EMPTY"},

推理结果中明显是选择了D，但是"review": {"gold": "D", "pred": "A", "result": 0}} review的结果错误了

这测试的是哪个数据集，会优化答案提取方法

mmlu，最好在答案提取方法中多加几个正则表达式，例如：

f'**Answer:**\\s?\\(?[{options_concat}]\\)?[。\\.]*$',

包括下面的字段匹配也会有问题：
        regexes = [re.compile(pattern) for pattern in patterns]
        for regex in regexes:
            match = regex.search(text)
            if match:
                outputs = match.group(0)
                breakpoint()
                **if 'Answer' in outputs:**
                    outputs = outputs.replace("Answer", "answer")
                for i in options:
                    if i in outputs:
                        return i

尤其是这一句**Answer** :C，我添加了正则后，这样会把Answer中的第一个字母A作为模型输出的选项。。。。

这块我们做了些优化，目前匹配应该会更准确

经过我孜孜不倦地努力（问deepseek），将代码修改为了：
patterns = [
    r'(?i)(?:the (?:best|correct) answer|answer:?|so|thus,)[\s:*\n]*\**\s*([A-D])[\.\*:]*',
    r'(?i)(?:making|makes)[\s*]+\*\*([A-D])\*\*.*?correct',
    r'(?i)\b(?:the answer is|answers?|correct answer)[\s()*:\n]+([A-D])[.)\*:]*',
    r'(?i)\*\*Answer\*\*.*?([A-D])\b',
    r'(?i)(?:correct answer (?:should be|must be|is).*?|so,?)[ \n*]*\(?\*?([A-D])[^A-Za-z]*',
    r'(?i)(?:thus,|sp)[\s*]+\*\*([A-D])\*\*.*?(?:best|revolutionary)',
    r'(?i)So.*? ([A-D]) is correct'
]
 这可以匹配绝大部分的deepseek-R1的英文回答，可供参考


> 这块我们做了些优化，目前匹配应该会更准确

@Yunnglin 请问在哪可以看到具体修改

参考：https://github.com/modelscope/evalscope/commit/c91501c6d993f6060ed001768e39375afea949c4#diff-a7703794cbb3c1b5f0514af99ea8d616e35c7a09ada99e83464cabd04766240c

修改了prompt template，匹配结果更准确了

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！