Repository: modelscope/evalscope
Issue: Support for sequence labelling tasks
Root cause: Unimplemented Feature Gap

Hi

First of all, thanks for the great work on evalscope, this is my second day of using evalscope, and it’s been really useful for evaluating QA and MCQ tasks!

I wanted to ask if there are any chances to support sequence labelling tasks (e.g., NER, POS tagging). This would be particularly valuable for evaluating models on token-level or span-level outputs, beyond just QA/MCQ.

I’m thinking maybe we could add support for sequence labelling via in-text annotated outputs. The approach is to represent labels inline using XML-like tags, for example:

```
Input: Yang lives in Manchester  
Output: <response> <person> Yang </person> lives in <location> Manchester </location> </response>
```

The framework then extracts the corresponding BIO or tagging labels from model outputs and evaluates them with standard metrics (e.g., seqeval F1).

Thanks again for maintaining this great library!

Warm regards, 
Yang

Thank you for your kind words about evalscope! We're glad it's been helpful for your evaluation tasks.

Your suggestion for supporting sequence labelling tasks (NER, POS tagging) is excellent. The XML-like annotation approach you proposed is a clean and practical solution for handling token-level evaluations.

We'll definitely consider this feature for our development roadmap. In the meantime, we encourage you to submit a PR if you're interested in implementing this functionality - we'd be happy to collaborate and provide support!

@Yunnglin I’d be happy to implement this. I want to make sure I’m aligned on the design before I start.

High-level plan:

1. Keep the dataset ground-truth in its native BIO format (e.g. [CoNLL-2003](https://huggingface.co/datasets/lhoestq/conll2003)).
2. Create a pipeline that prompts the LLM to produce an XML-like prediction (wrapped in a top-level `<response>...</response>`. Entity spans wrapped like `<person> Yang </person>` or `<location> Manchester </location>`).
3. Implement a function that converts the model’s XML-like prediction back into BIO tokens aligned with the dataset tokens.
4. Compute token/span metrics with [seqeval](https://github.com/chakki-works/seqeval) (e.g., overall F1) comparing ground-truth BIO to the converted BIO predictions.

Design decisions where I’d appreciate guidance:

1. Where to implement this in the codebase? I'm thinking subclassing DefaultDataAdapter just like QA task under folder `evalscope/benchmarks/conll2003`.
2. How to handle malformed model output? LLMs often return imperfect XML (e.g., missing closing tags, tag name mismatches, extra text). Is there any existing evalscope support for structured-output or retry mechanism utilities?

If that all sounds reasonable I’ll prepare to draft a PR. Happy to start as soon as I get your guidance on the above.

Thank you for the detailed plan! Your high-level approach sounds very reasonable. Here's the reference documentation for adding evaluation benchmarks: https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/add_benchmark.html. I'd recommend reading through it first.

Regarding your design decisions:

**Implementation location**: Yes, subclassing `DefaultDataAdapter` and creating a new folder `evalscope/benchmarks/conll2003` is the right approach. You can reference the general_qa implementation as a template.

**Handling malformed model output**: Currently, there's no built-in fallback or retry mechanism for incorrectly formatted responses. However, you can set the `repeats` parameter, which will evaluate each sample n times repeatedly, allowing you to calculate metrics with better statistical support (similar to how HumanEval handles this).

Your overall plan looks solid. Feel free to proceed with drafting the PR, and we can iterate on the implementation details during the review process. Looking forward to seeing your contribution!

@Yunnglin Just wanted to let you know that I’ve completed the CoNLL2003 NER evaluation code using **Llama3.1-8B-Instruct** and **GPT-4o-mini**.

The setup follows the same **in-line** NER format as described in the previous discussion. I also include the results from this [paper](https://arxiv.org/pdf/2406.04528) (this is the only paper I could find that compares llama and gpt), so we can use it as a reference to validate that the implementation is correct.

I’ve attached two comparison tables below (all the metrics are micro-averaged instead of macro-averaged):

| Model                | Setting | Precision | Recall | F1     |
| :-------------------: | :------: | :--------: | :-----: | :-----: |
| gpt-4o-mini (evalscope)          | 5-shot  | 0.6029    | 0.6850 | 0.6413 |
| llama3.1-8b-instruct (evalscope) | 0-shot  | 0.3030    | 0.3171 | 0.2985 |
| llama3.1-8b-instruct (evalscope) | 5-shot  | 0.3131    | 0.4542 | 0.3707 |
| gpt-3.5 (Villena et al., 2024)              | 0-shot  | -         | -      | 0.5980 |
| llama2-70b-instruct (Villena et al., 2024)  | 0-shot  | -         | -      | 0.3920 |

Let me know if you'd like me to include it in a PR.

Great work on the CoNLL2003 NER evaluation! The results look reasonable and the validation against existing benchmarks is appreciated.

Please go ahead and submit a PR following our contribution guidelines: https://github.com/modelscope/evalscope/blob/main/CONTRIBUTING.md

I'll be happy to review it once it's ready. Thanks for the contribution! 🙏


Thanks so much, @Yunnglin! I really appreciate your guidance and feedback throughout this. I’ll submit the PR later today following the contribution guidelines.

Hi @Yunnglin, I just noticed that the CoNLL2003 dataset has now been added (PR merged) into evalscope, that’s great to see!

I had a quick question: there are still quite a few general-purpose NER datasets available that are widely used in other frameworks. Do you think it would be a good idea to include them as well?

Here are some examples:

| paper | venue | link |
| :---: | :---: | :---: |
| [paper](https://pmc.ncbi.nlm.nih.gov/articles/PMC6351503/pdf/bdj-07-e29626.pdf) | BDJ2019 | https://huggingface.co/datasets/extraordinarylab/copious | 
| [paper](https://ojs.aaai.org/index.php/AAAI/article/view/17587) | AAAI2021 | https://huggingface.co/datasets/extraordinarylab/cross-ner | 
| [paper](https://academic.oup.com/bioinformatics/article/19/suppl_1/i180/227927) | Bioinformatics2003 | https://huggingface.co/datasets/extraordinarylab/genia-ner | 
| [paper](https://aclanthology.org/2022.naacl-main.243.pdf) | NAACL2022 | https://huggingface.co/datasets/extraordinarylab/harvey-ner | 
| [paper](https://aclanthology.org/C16-1111.pdf) | COLING2016 | https://huggingface.co/datasets/extraordinarylab/broad-twitter-corpus | 
| [paper](https://aclanthology.org/W17-4418.pdf) | ACL2017 Workshop | https://huggingface.co/datasets/extraordinarylab/wnut2017 | 
| [paper](https://sls.csail.mit.edu/publications/2013/Liu_ICASSP-2013.pdf) | ICASSP2013 | https://huggingface.co/datasets/extraordinarylab/mit-movie-trivia | 
| [paper](https://sls.csail.mit.edu/publications/2013/Liu_ICASSP-2013.pdf) | ICASSP2013 | https://huggingface.co/datasets/extraordinarylab/mit-restaurant | 
| [paper](https://aclanthology.org/N06-2015.pdf) | NAACL2006 | https://huggingface.co/datasets/extraordinarylab/ontonotes5 | 
| [paper](https://aclanthology.org/W03-0419.pdf) | NAACL2003 | https://huggingface.co/datasets/extraordinarylab/conll2003 | 

These datasets are also included in well-known frameworks such as [llmner](https://github.com/plncmm/llmner), [tner](https://github.com/asahi417/tner), [gpt-ner](https://github.com/ShuheWang1998/GPT-NER), [nervaluate](https://github.com/MantisAI/nervaluate), [InstructUIE](https://github.com/BeyonderXX/InstructUIE), [GoLLIE](https://github.com/hitz-zentroa/GoLLIE), [UniversalNER](https://github.com/universal-ner/universal-ner), [ToNER](https://github.com/jiangguochaoGG/ToNER), [GLiNER](https://github.com/urchade/GLiNER).

I've experimented with most of them, but many seem outdated, inactive, or limited to embedding-based models. The recently merged PR (#912) for evalscope, on the other hand, provides a much cleaner and more maintainable workflow.

Would it make sense to include these datasets in evalscope as well, perhaps grouped under a dedicated NER suite or folder for better organisation? If you think this might not be appropriate right now, I'll go ahead and close this issue for the time being. Please let me know what you think.

I'd be delighted to integrate these benchmarks into evalscope. Creating a dedicated `ner` folder under the benchmark directory and implementing a separate adapter file for each benchmark would be a feasible approach. For example:

```text
├── ner/
│   ├── conll2003_adapter.py
│   ├── copious_adapter.py
│   ├── cross_ner_adapter.py
│   ├── genia_ner_adapter.py
│   ├── harvey_ner_adapter.py
│   ├── broad_twitter_corpus_adapter.py
│   ├── wnut2017_adapter.py
│   ├── mit_movie_trivia_adapter.py
│   ├── mit_restaurant_adapter.py
│   ├── ontonotes5_adapter.py
│   └── __init__.py
```

This structure would provide better organization and maintainability for the NER benchmark suite within evalscope. Feel free to reach out if you have any questions or need further clarification!

@Yunnglin Thanks for the clarification!

Just to confirm: we currently have a `conll2003` folder directly under the `benchmarks` directory. Should we remove that and migrate everything into the new `ner` folder, or keep it as-is for now and simply create `benchmarks/ner` with all the NER adapters inside (i.e., including conll2003_adapter.py)?

If the latter sounds good, I'll go ahead and start restructuring the benchmarks accordingly and open a PR once it’s ready (ideally early next week).

@Yunnglin Thanks for the quick help on this issue! Since PR #912 and PR #921 resolve the problem, I’ll close this issue for now. Really appreciate the support and collaboration!