Repository: modelscope/evalscope
Issue: Feature request: add Drivelology benchmark dataset
Root cause: Unimplemented Feature Gap

Hi @Yunnglin I’d like to propose adding a new benchmark dataset called DrivelHub for evaluating LLMs' socialability and Drivelological text understanding. [[Paper]](https://arxiv.org/pdf/2509.03867) [[Dataset]](https://huggingface.co/datasets/extraordinarylab/drivel-hub)

Currently, evalscope includes mainly knowledge-based and reasoning-based benchmarks, but no dedicated benchmark for non-linear social or Drivelological text comprehension. (Full disclosure: I'm the first author of the paper, a bit of a shameless plug xD, but I believe this benchmark could fit in evalscope.)

A bit of the overview. The Drivelology benchmark includes four tasks:
 - Binary Classification: Determine whether a given text is Drivelology or non-Drivelology.
 - Multi-Label Classification: Classify a text into one or more of five Drivelology Taxonomy categories.
 - Narrative Writing: A generative task, produce an underlying Drivelological narrative or interpretation.
 - Multiple-Choice QA: Select the best description (out of five) for a given Drivelological text.

I’d appreciate some guidance on implementing the benchmark within evalscope, particularly:
 - Multi-Label Classification: I didn’t find existing evalscope tasks that handle multi-label classification. Is there a recommended format or evaluation API to support this?
 - Narrative Writing: I’m considering using an LLM-judge or rubric-based approach for evaluation. Have you experimented with [prometheus-eval](https://github.com/prometheus-eval/prometheus-eval) or a similar rubric-style evaluation module within evalscope?

The binary and multiple-choice tasks should be straightforward to implement based on existing templates, but I'd love some input or examples on handling the two more complex task types above. Thanks a lot for maintaining this great framework, I'm happy to contribute and help integrate this benchmark!

Warm regards, 
Yang

Thank you for sharing the details about Drivelology! After reviewing your paper abstract, I find this to be a fascinating and valuable contribution that explores the gap between statistical fluency and pragmatic understanding in LLMs. I'd be delighted to help integrate this benchmark into evalscope.

Regarding your implementation questions, here are my suggestions:

**Multi-Label Classification:**
You can adapt this as a multiple-choice format. I recommend checking out the `MMMU` implementation as a reference, which handles both OPEN and MULTICHOICE question types. This approach should work well for your taxonomy classification task.

**Narrative Writing:**
evalscope does support LLM-as-a-judge evaluation methods. You can refer to the `SimpleQA` implementation and customize the `llm_match_score` function to suit your rubric-based evaluation criteria for Drivelological narrative assessment.

**Additional Recommendation:**
I'd also suggest creating an ExtraOrdinaryLab organization on ModelScope and uploading your dataset there. This would make it much more accessible for the broader research community to download and utilize your work.

Feel free to reach out if you have any questions during the integration process. Looking forward to collaborating on this!


@Yunnglin Thank you so much for your detailed feedback and guidance! I've already applied to create an ExtraOrdinaryLab organisation on ModelScope and am currently waiting for review.

I really appreciate your suggestions on how to integrate the Drivelology benchmark into evalscope, I'll follow your instructions and test across the four tasks as discussed. 

For the narrative writing tasks, since I don't have remaining funding for GPT-4-based LLM-as-a-judge evaluation, I'll replace it with an open-sourced LLM as the judging model for now, hope that's fine.

Once I've completed the implementation and testing, I'll open another PR for review. I'll keep this issue open until I submit the PR (ideally before the end of next week).

I noticed that you've created the extraordinarylab organization on ModelScope. If possible, could you please grant me access to join the organization? I would be happy to help synchronize the datasets from Hugging Face to ModelScope, which would make them more accessible to a broader user base. I've already submitted an application to join the organization: https://modelscope.cn/organization/extraordinarylab 

@Yunnglin Access granted! Just to let you know, PR #927 has been created for this issue, thanks for all the suggestion that helps me a lot!

Since PR #927 has resolved this issue, I'll close it for now. Thanks for all the support!

Hi @Yunnglin I have one quick follow-up question regarding the Drivelology writing task implementation.

In the current setup, the `bert_score` is computed inside `llm_match_score()` for each sample individually. Since it’s not batched, this significantly slows down evaluation when running on larger datasets.

I was wondering if it might be possible to optimise this by pre-initialising a `BERTScorer` instance from the `bert_score` library (check out from [here](https://github.com/Tiiiger/bert_score/blob/master/bert_score/scorer.py)) in `__init__`, rather than calling `compute_bertscore_one_sample()` repeatedly for each sample. Something like this:

```python
from bert_score import BERTScorer

self.bert_scorer = BERTScorer(model_type='roberta-large', lang='en', rescale_with_baseline=False)
```

Then later in `llm_match_score()` we could simply call:

```python
P, R, F1 = self.bert_scorer.score([filtered_prediction_trunc], [reference_trunc])
```

My thought is that this approach could avoid rebuilding the BERT model from scratch for each sample, potentially speeding up evaluation considerably. Do you think initialising the scorer this way would work safely, or would it conflict with how adapters are instantiated during evaluation?

@penguinwang96825 I've submitted a PR to support bert_scorer lazy initialization. Please check if this matches what you had in mind.

That looks perfect, thanks @Yunnglin!

Just one more question, do you think it would be feasible to further optimise this by batchifying the `bert_score` computation across samples (e.g., collecting predictions and references first, then running bertscorer.score() on them in batches)?

I realise this might make the code a bit less readable or complicate the adapter structure, but I’m curious if that could bring any noticeable speedup during large-scale evaluation.

I'll implement a batch processing version and add bert_score as a basic metric.

Smashing! #938 might be relevant?

@penguinwang96825 After a substantial refactoring, batch-based metric computation is now supported. Currently, the supported metric is **BERTScore**. In the adapter, you can enable batch metric calculation by calling the `batch_match_score` method and setting the `use_batch_scoring` flag to `True`. Refer to #935 for more details!