Repository: modelscope/evalscope
Issue: Question: standard vs CoT prompting in API calls
Root cause: Unimplemented Feature Gap

Hi, is there currently a way to evaluate the same dataset using different prompt templates (e.g., MultipleChoiceTemplate.SINGLE_ANSWER and MultipleChoiceTemplate.SINGLE_ANSWER_COT) via the API or CLI call, without redefining the benchmark adapter?

This would make it much easier to compare LLMs’ performance on standard vs. CoT prompting styles using the same dataset and evaluation logic.

Warm regards, 
Yang


EvalScope already supports setting custom prompt templates for each benchmark at evaluation runtime. You can specify the prompt template through the `--dataset-args` configuration without needing to redefine the benchmark adapter.

Here's a configuration example:

```bash
--datasets sciq \
--dataset-args '{
  "sciq": {
    "few_shot_num": 0,
    "prompt_template": "Answer the following multiple choice question. The entire content of your response should be of the following format: '\''ANSWER: $LETTER'\'' (without quotes) where LETTER is one of {letters}.\n\n{question}\n\n{choices}"
  }
}'
```

This means you can easily compare different prompting strategies (standard vs. CoT) on the same dataset by running separate evaluations with different `prompt_template` values in the dataset arguments. For example:

1. For standard prompting: Use a simple template like the one above
2. For CoT prompting: Modify the template to include "Let's think step by step" or similar CoT instructions

The `{question}` and `{choices}` placeholders will be automatically populated with the actual test data during evaluation.

Would this approach work for your use case? If you need more specific examples or have questions about template formatting, please let me know!

@Yunnglin thanks for the super quick response! I was wondering if the original prompt for that certain benchmark is more complicated, then how would it be possible to switch between standard and CoT prompting? Take HaluEval for example, because the original default prompt template is very long and complex, is there any more elegant way to input the prompt template argument via API call? 

We could address this by implementing a prompt template management module within the framework that would:

1. Allow pre-defining multiple prompt strategies (standard, CoT, etc.) for each benchmark
2. Enable users to specify which template to use during evaluation via a simple parameter
3. Support template inheritance and customization

This would provide a more elegant solution than having users provide the entire prompt template through API calls. However, this enhancement would require significant development effort as we'd need to adapt each benchmark individually to work with this new templating system.

For our roadmap, I suggest we prioritize this for benchmarks with particularly complex prompts first, then gradually extend the functionality to all benchmarks. Would that approach work for your needs?

That makes a lot of sense, I completely agree with your proposed approach. This would indeed be quite a large piece of work, and I'm not sure if it currently aligns with evalscope’s short-term roadmap.

That said, I don't think it needs to be a high-priority task right now. But if you think this is a worthwhile direction to explore, would you mind opening a PR implementing it for one or two benchmarks as a proof of concept? I'd be happy to help extend the implementation to the remaining benchmarks afterward.

Thanks for the feedback! I'll work on a PR to implement this for one or two benchmarks as a PoC, focusing on the prompt management aspect. Please keep this issue open in the meantime, and I'll share the implementation once it's ready.