Repository: open-compass/opencompass
Issue: [Bug] If-eval deepseek-r1 score may not corret
Root cause: Documentation Deficiency

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I have modified the code (config is not considered code), or I'm working on my own tasks/models/datasets.

### Environment

{'CUDA available': True,
 'CUDA_HOME': '/usr/local/cuda',
 'GCC': 'gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0',
 'GPU 0,1,2,3,4,5,6,7': 'NVIDIA A100-SXM4-80GB',
 'MMEngine': '0.10.7',
 'MUSA available': False,
 'NVCC': 'Cuda compilation tools, release 12.1, V12.1.105',
 'OpenCV': '4.11.0',
 'PyTorch': '2.5.1+cu124',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 9.3\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2024.2-Product Build 20240605 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v3.5.3 (Git Hash '
                              '66f0cb9eb66affd2da3bf5f8d897376f04aae6af)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: AVX2\n'
                              '  - CUDA Runtime 12.4\n'
                              '  - NVCC architecture flags: '
                              '-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n'
                              '  - CuDNN 90.1\n'
                              '  - Magma 2.6.1\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, CUDA_VERSION=12.4, '
                              'CUDNN_VERSION=9.1.0, '
                              'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -fvisibility-inlines-hidden '
                              '-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO '
                              '-DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON '
                              '-DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK '
                              '-DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
                              '-O2 -fPIC -Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wsuggest-override '
                              '-Wno-psabi -Wno-error=old-style-cast '
                              '-Wno-missing-braces -fdiagnostics-color=always '
                              '-faligned-new -Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'TORCH_VERSION=2.5.1, USE_CUDA=ON, USE_CUDNN=ON, '
                              'USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, '
                              'USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, '
                              'USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, '
                              'USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, '
                              'USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.16 (main, Dec 11 2024, 16:24:50) [GCC 11.2.0]',
 'TorchVision': '0.20.1+cu124',
 'lmdeploy': "not installed:No module named 'lmdeploy'",
 'numpy_random_seed': 2147483648,
 'opencompass': '0.4.1+bc2969d',
 'sys.platform': 'linux',
 'transformers': '4.48.3'}

### Reproduces the problem - code/configuration sample

None

### Reproduces the problem - command or script

```shell
python3 run.py --models vllm_qwen2_5_32b_instruct --datasets IFEval_gen --debug
```

i modified vllm_qwen2_5_32b_instruct.py to load deepseek r1 model and run IFEval benchmark.
the official release IF-Eval Dataset score is  83.3,  but I only got 34.20.

So I check the prompt and generation and  find that

```
prompt:
"<｜User｜>Write a 300+ word summary of the wikipedia page \"https://en.wikipedia.org/wiki/Raymond_III,_Count_of_Tripoli\". Do not use any commas and highlight at least 3 sections that has titles in markdown format, for example *highlighted section part 1*, *highlighted section part 2*, *highlighted section part 3*.\n<｜Assistant｜>"

generation:
"<think>\nOkay, I need to write a 300+ word summary of the Wikipedia page about Raymond III, Count of Tripoli. The user wants me to avoid using any commas and highlight three sections with titles in markdown format. Let me start by understanding the key points from the article. \n\nFirst, I'll look at the structure of the Wikipedia page. The page probably starts with an introduction about Raymond III, his birth, and his role as Count of Tripoli. He was a significant figure during the Crusades, so his involvement in the political and military aspects would be important. \n\nNext, there might be sections on his early life and succession. Raymond III was born in 1140, became Count after his father's death in 1174. He was part of the Crusader states, which were in conflict with Muslim forces. His leadership in Tripoli and his alliances with other Christian nobles would be key points here. \n\nThen, the page might have a section on his role in the Crusades. He participated in campaigns against Saladin, especially the Battle of Hattin in 1187. This was a major event where he was captured, but later released. His strategies and how he contributed to the defense of Christian territories would be important to highlight.\n\nAnother section might discuss his diplomacy and alliances. He tried to strengthen ties with Byzantine Empire and other Crusader states. This shows his strategic thinking to counter Muslim threats. His role in negotiating treaties and alliances would be a key part of his legacy. \n\nThere could be a section on his legacy and death. He died in 1199, leaving a complex legacy. He was seen as both a capable leader and a controversial figure due to his political maneuvering. His efforts to maintain Christian control in the Levant during a turbulent time would be emphasized here.\n\nNow, I need to pick three sections to highlight with markdown. The user examples use *highlighted section part 1*, etc. So I'll choose sections like Early Life and Succession, Role in the Crusades, and Diplomacy and Alliances. \n\nI need to make sure not to use any commas. So I'll have to structure sentences without commas, maybe using conjunctions or splitting sentences. Also, each highlighted section should be in markdown, probably using asterisks and italics, but the example uses just asterisks. \n\nWait, the user said \"highlighted section part 1\" in markdown, like *highlighted section part 1*. So maybe just using asterisks around the section titles. But the user example uses the format: *highlighted section part 1*. So perhaps I need to format the section titles like that. \n\nWait, the user's example is: \"for example *highlighted section part 1*, *highlighted section part 2*, *highlighted section part 3*.\" So I should create three sections with titles like that. So in the summary, I need to have three parts where each section title is in italics, like *Early Life and Succession*, *Role in the Crusades*, and *Diplomacy and Alliances*. \n\nBut the user also said \"highlight at least 3 sections that has titles in markdown format\". So maybe I need to present the sections with titles in markdown, perhaps as headings. But the example given uses asterisks. \n\nWait, maybe the user wants the section titles to be highlighted with markdown, which is typically done using **bold** or *italics*. But in the example, they wrote \"*highlighted section part 1*\", so maybe just italicizing the section titles. \n\nSo, in the summary, I'll have three sections, each with a title in italics, like *Early Life and Succession*, *Role in the Crusades*, and *Diplomacy and Alliances*. Then, in each section, I'll write a couple of sentences summarizing those parts from the Wikipedia page. \n\nI need to make sure the summary is over 300 words. Let me outline the sections and ensure that each highlighted part is clearly marked and that the entire summary flows without commas. \n\nLet me start writing the summary. First, introduce Raymond III, his birth, becoming Count of Tripoli. Then the first highlighted section on Early Life. Then move to his role in the Crusades, the Battle of Hattin. Then discuss his diplomacy efforts, alliances with Byzantines. Finally, his death and legacy. \n\nWait, the user example uses three sections, each part is a separate highlighted section. So perhaps each of these topics is a section with a title in italics. \n\nI need to make sure not to use any commas. So sentences will be structured with periods and maybe semicolons, but since the instruction is to avoid commas, I'll avoid commas entirely. \n\nLet me check the word count. The summary needs to be 300+ words. Each paragraph can be around 100 words, so three sections plus some additional info. \n\nNow, putting it all together, making sure each highlighted section is marked with *...* and that there are no commas used in the entire summary.\n</think>\n\nRaymond III Count of Tripoli was a prominent figure in the Crusader states during the 12th century. Born in 1140 he inherited the title of Count of Tripoli in 1174 following his father’s death. His rule coincided with a turbulent period marked by conflicts between Crusader factions and Muslim forces led by figures like Saladin. Raymond III played a crucial role in both political and military spheres striving to maintain Christian dominance in the Levant.\n\n*highlighted section part 1*  \n**Early Life and Succession**  \nRaymond III was born into the House of Poitou a noble family with deep ties to the Crusader states. His father Raymond II was assassinated in 1174 leading Raymond III to assume control of Tripoli. The young count faced immediate challenges including internal dissent and external threats from Muslim armies. His leadership was tested early as he sought to stabilize Tripoli’s position amid regional rivalries.\n\n*highlighted section part 2*  \n**Role in the Crusades**  \nRaymond III’s military campaigns were pivotal during the Crusades. He allied with Christian nobles to resist Saladin’s expansion notably participating in the Battle of Hattin in 1187. Though captured during the battle he was later released and continued to defend Crusader territories. His strategies emphasized fortification and tactical alliances which prolonged Christian resistance despite gradual territorial losses.\n\n*highlighted section part 3*  \n**Diplomacy and Alliances**  \nRaymond III was renowned for his diplomatic acumen. He forged ties with the Byzantine Empire and other Crusader states to bolster collective security. These alliances were vital in countering Muslim advances but also sparked controversy as some nobles viewed his pragmatism as compromising ideals. His negotiations often balanced immediate survival with long-term Crusader objectives.\n\nRaymond III died in 1199 leaving a legacy of resilience and adaptability. His tenure as Count of Tripoli exemplified the complexities of Crusader politics where survival often necessitated compromise. Though criticized by contemporaries for his political maneuvers his efforts ensured Tripoli’s relevance in an era of decline. His contributions remain a subject of study for historians analyzing the Crusades’ intricate dynamics.

```


OK, see  that `<think>.....</think>` ？ 
this makes some  prompt  instruction not follows, if I strip this from generation, the final score will achive same score as offical release.

I modified `test_instruction_following_strict` method in `datasets/IFEval/evaluation_main.py`

```python
def test_instruction_following_strict(
    inp,
    response,
):
    """Tests response to see if instrutions are followed."""
    instruction_list = inp.instruction_id_list
    is_following_list = []
    if response.startswith('<think>'):
        response = response[response.rfind('</think>') + 8:].lstrip('\n')
    print("prompt:", inp.prompt)
    print("answer:", response)
    print("\n")
    for index, instruction_id in enumerate(instruction_list):
        instruction_cls = instructions_registry.INSTRUCTION_DICT[
            instruction_id]
        instruction = instruction_cls(instruction_id)
        instruction.build_description(**inp.kwargs[index])
        args = instruction.get_instruction_args()
        if args and 'prompt' in args:
            instruction.build_description(prompt=inp.prompt)
        
        if response.strip() and instruction.check_following(response):
            is_following_list.append(True)
        else:
            is_following_list.append(False)

    return OutputExample(
        instruction_id_list=inp.instruction_id_list,
        prompt=inp.prompt,
        response=response,
        follow_all_instructions=all(is_following_list),
        follow_instruction_list=is_following_list,

```


so will add more process to remove thinking?




### Reproduces the problem - error message

no error message but got wrong score

### Other information

_No response_

Hi, Actually we have already supported some process to remove thinking, try add this postprocessor in your model config.
```
from opencompass.utils import think_pred_postprocess
dict(
        abbr="deepseek-chat-r1",
        type=OpenAISDK,
        path="deepseek-r1-250120",
        key='your key',
        openai_api_base="your base",
        meta_template=api_meta_template,
        query_per_second=16,
        batch_size=32,
        temperature=0.001,
        max_out_len=16384,
        max_seq_len=131072,
        pred_postprocessor=dict(type=think_pred_postprocess,
                                re_pattern=r'</think>([\s\S]*)'),
)
```

> Hi, Actually we have already supported some process to remove thinking, try add this postprocessor in your model config.
> 
> ```
> from opencompass.utils import think_pred_postprocess
> dict(
>         abbr="deepseek-chat-r1",
>         type=OpenAISDK,
>         path="deepseek-r1-250120",
>         key='your key',
>         openai_api_base="your base",
>         meta_template=api_meta_template,
>         query_per_second=16,
>         batch_size=32,
>         temperature=0.001,
>         max_out_len=16384,
>         max_seq_len=131072,
>         pred_postprocessor=dict(type=think_pred_postprocess,
>                                 re_pattern=r'</think>([\s\S]*)'),
> )
> ```

thanks, I also checked `opencompass/examples/eval_deepseek_r1.py`.

Yes, I do found the pred_post_processor `extract_non_reasoning_content`
```python
models += [
    # You can comment out the models you don't want to evaluate
    # All models use sampling mode
    dict(
        type=TurboMindModelwithChatTemplate,
        abbr='deepseek-r1-distill-qwen-7b-turbomind',
        path='deepseek-ai/DeepSeek-R1-Distill-Qwen-7B',
        engine_config=dict(session_len=32768, max_batch_size=128, tp=1),
        gen_config=dict(
                        do_sample=True,
                        temperature=0.6,
                        top_p=0.95,
                        max_new_tokens=32768),
        max_seq_len=32768,
        max_out_len=32768,
        batch_size=64,
        run_cfg=dict(num_gpus=1),
        pred_postprocessor=dict(type=extract_non_reasoning_content)
    ),

```

yes same function

Closing this issue. Feel free to re-open if more help needed, thanks.