Repository: EleutherAI/lm-evaluation-harness
Issue: KorMedMCQA Implementation does not match the original paper
Root cause: Configuration Error

I tested some model on KorMedMCQA task and found some models can't reproduce the same result in the paper.

|Model|Reported Avg.|Experiment Avg.|
|--|--|--|
|Gemma-2-9B|45.36|0.0|
|Phi-3.5-mini|42.57|39.55|
|Qwen2-1.5B|34.30|27.05|
|Qwen2.5-1.5B|37.92|23.56|
|Meditron-7B|25.62|25.03|
|Gemma-2-2B|29.18|0.0|

So I have been finding the reason, the yaml of KorMedMCQA doesn't contain the post-processing for generated text. In the 'Appendix B' of the [original article](https://doi.org/10.48550/arXiv.2403.01469), the authors provide regex to extract the answers like below:

```python
import re
def extract_choice(output):
    matches = re.findall(r'정답[:\s]*([ABCDE])', output, re.IGNORECASE)
    if matches:
        return matches[-1].upper()
    else:
        matches = re.findall(r'정답은\s*([ABCDE])\s*입니다.', output, re.IGNORECASE)
        if mathces:
            return matches[-1].upper()
        else:
            matches = re.findall(r'\b([ABCDE])\.', output, re.IGNORECASE)
            if matches:
                return matches[-1].upper()
           else:
                matches = re.findall(r'\b([ABCDE])\b', output, re.IGNORECASE)
                if matches:
                    return matches[-1].upper()
                else:
                    return ''

def extract_cot_choice(output):
    last_pos = output.rfind('따라서')
    if last_post == -1:
        return ''
    else:
        following_text = output[last_pos + len('따라서'):]
        sentences = re.split(r'[.!?\n]', following_text)
        sentences = [s.strip() for s in sentences if s.strip()]
        if sentences:
            sentence = sentences[0]
            match = re.search(r'\(([A-Za-z])\)', sentence)
            if match:
                return match.group(1).upper()
            else:
                return ''
        else:
            return ''
```

I'm reluctant to contribute due to not the author of the paper. But I think it should be reviewed to reproduce the result.

cc: @GyoukChu @sean0042 

I modified yaml in KorMedMCQA like below. The `filter_list` in `_template_yaml` is the added part.

**_kormedmcqa.yaml**
```yaml
group: kormedmcqa
task:
  - kormedmcqa_doctor
  - kormedmcqa_nurse
  - kormedmcqa_pharm
  - kormedmcqa_dentist
aggregate_metric_list:
  - metric: exact_match
    filter_list: "extract_answer"
    aggregation: mean
    weight_by_size: true
metadata:
  version: 2.0
```

**_template_yaml**
```yaml
dataset_path : sean0042/KorMedMCQA
test_split : test
fewshot_split : fewshot
fewshot_config:
  sampler: first_n
  doc_to_text: "{{question.strip()}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}\nE. {{E}}\n정답： {{['A', 'B', 'C', 'D', 'E'][answer-1]}}\n\n"
  doc_to_target: ""
output_type: generate_until
doc_to_text: "{{question.strip()}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}\nE. {{E}}\n정답："
doc_to_target: "{{['A', 'B', 'C', 'D', 'E'][answer-1]}}"
filter_list:
  - name: "extract_answer"
    filter:
      - function: "regex"
        regex_pattern: ".*([A-E])"
      - function: "take_first"
metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true
    ignore_case: true
    ignore_punctuation: true
generation_kwargs:
  until:
    - "Q:"
    - "
"
    - "
"
    - "."
    - "\n\n"
  do_sample: false
  temperature: 0.0
  max_gen_toks: 1024
metadata:
  version: 2.0
dataset_kwargs:
  trust_remote_code: true
```

And I've tested again, most models exceeds their experiment average score. Is my regex wrong? or did the paper underestimate?
|Model|Reported Avg.|Experiment Avg.|Added Regex Avg.|
|--|--|--|--|
|Gemma-2-9B|45.36|0.0|62.38|
|Phi-3.5-mini|42.57|39.55|42.07|
|Qwen2-1.5B|34.30|27.05|39.65|
|Qwen2.5-1.5B|37.92|23.56|41.14|
|Meditron-7B|25.62|25.03|24.79|
|Gemma-2-2B|29.18|0.0|39.42|