Repository: modelscope/evalscope
Issue: Upcoming Benchmarks and Roadmap
GitHub labels: enhancement

Hey, we're currently developing additional benchmarks for Evalscope and wanted to check whether you have a roadmap or plan outlining which evaluations are coming next.

You previously mentioned plans to expand the Artificial Analysis benchmarks — is there an approximate timeline for those releases? Additionally, we’re curious if there are any upcoming efforts to introduce more complex coding benchmarks (e.g., SWE-Bench) or agentic benchmarks (e.g., Terminal Bench).

At Cerebras, we are working to unify our benchmarking efforts through Evalscope and want to ensure we’re not duplicating development that may already be on your roadmap and identify areas we could collaborate.

If it’s easier to discuss this over a call, we’d be happy to set up a meeting — we can also communicate in Chinese if that’s preferred. Thanks so much and love the work you and your team are doing!

@sophies-cerebras Thanks for your interests in EvalScope (and your earlier contributions too!).

We will post a draft of our roadmap for comments and suggestions soon, outlining planned benchmarks and upcomig work for EvalScope, please stay tuned.

In the meantime, we would also love your input regarding how EvalScope can be of more help for usage cases at Cerebras. We certainly welcome discussions for cooperation opportunies: once the roadmap is up, we can check how to better align our efforts, and set up meetings if necessary.

cc @Yunnglin @wangxingjun778 

Hi @sophies-cerebras , Our roadmap is now up at [link](https://github.com/modelscope/evalscope/issues/951). It is open to discussions of coz, so do feel free to provide feedback.

As you can see from the roadmap, we do plan to integrate more benchmarks from Artificial Analysis, to eventually provide coverage of the complete suite. We welcome open contributions so please feel free to pick benchmarks you are most interested in (or already have a downstream implementation) for contributions.

I would also like to highlight capabilities of EvalScope that we think may be of interest to Cerebras:

- Performance Benchmarking: for evaluating performance and throughput of model services
- Evaluation of Benchmark-Collection:  for assessing model performance across a custom collection of datasets, a perfect fit for generating performance indexes in different areas.

Again, please review the proposed roadmap, and let us know your thoughts. We are open to meetings to discuss plans for roadmap executions as well.

Hi @Yunnglin @wangxingjun778 @yingdachen,

Thank you so much for the quick response and for sharing the roadmap so promptly. The roadmap aligns really well with the benchmarks we’re currently developing. Two benchmarks we’re in the process of incorporating into EvalScope are **IFBench** and **OpenAI MRCR** [(a long-context reasoning benchmark)](https://huggingface.co/datasets/openai/mrcr).

We’re especially excited to see your team’s plans around SWE-Bench — it’s been a challenging benchmark to support reliably, so we’d love to see it available in EvalScope soon.

Appreciate you highlighting the Performance Benchmarking and Evaluation of Benchmark-Collection features. We haven’t explored Performance Benchmarking in depth yet, but it’s definitely an area we’ll look into since performance is critical for our workloads. At the moment, we’re not focusing on Evaluation of Benchmark-Collection.

A few small pain points we’ve encountered (which we’re also working to address):

- Generation configs, batch size, judge, and model repeats are currently global configs. It would be great if these could be customized per dataset as well — having both global and dataset-level overrides would offer more flexibility.

- Retry mechanisms: while using judge models, we sometimes hit rate-limit errors. We’ve implemented our own retry logic internally and plan to contribute it upstream soon.

Overall, we’re really excited about EvalScope’s direction and look forward to collaborating further as the roadmap evolves.

@sophies-cerebras We now support SWE-bench, including SWE-bench Verified, SWE-bench Lite, and SWE-bench Verified Mini. You can run evaluations using the main branch code by following this documentation: https://evalscope.readthedocs.io/en/latest/third_party/swe_bench.html. Feel free to reach out if you have any questions.