Repository: EleutherAI/lm-evaluation-harness
Issue: Enhancing Extraction Function can provide up to 5% increase in scores
Root cause: Validation Gap

I've created a script to produce a csv file to parse all json files including cached responses after benchmarking [deepseek-ai/DeepSeek-R1-0528-Qwen3-8B] on MMLU_Pro with 0-shot to manually inspect the below:
1- Truncation of the output due to max_gen_toks limit
2- Instances where lm-eval reported predicted answer as "invalid" to manually inspect the outputs generated by the model for those samples 

Findings:
1- Some tasks perform great with max_gen_toks of 6144 while other tasks require much greater (maybe up to 12288). 
2- The extraction function can be optimized as manually inspecting samples where lm-eval reported invalid predictions, the raw output of the model actually contained an answer (most of which are correct) though the final answer formatting is not very divergent from the one set in the default prompts. I attached a screenshot showing some samples where the output format is formatted as ANSWER: X

<img width="2999" height="1863" alt="Image" src="https://github.com/user-attachments/assets/4befe8f4-621a-429c-b38b-53a83dee3125" />


The fallback extraction methods seem to be scanning for individual letters between [A-J] through all the output, thus detecting multiples matches and reporting invalid where in fact the model outputs the final correct answer in the format of ANSWER:X or ANSWER is X. 

# MMLU Pro Answer Extraction Proposed Fixes

## Problem Summary
The current MMLU Pro implementation in lm-evaluation-harness lacks robust answer extraction methods, causing ~550/12k samples (4.6%) to be incorrectly marked as "invalid" when models actually provide correct answers in slightly different formats.

## Current Issues

### 1. Missing Regex Patterns
The MMLU Pro paper specifies extraction patterns that aren't implemented:
- Primary: `answer is (?([A-J])?)`
- Secondary: `.[aA]nswer:\s([A-J])`
- Fallback: Random selection

### 2. Observed Model Outputs Not Captured
Models frequently output valid answers in these formats:
- `ANSWER: H`
- `ANSWER is D` 
- `The answer is C`
- `Answer: J`

### 3. Current Fallback Too Simplistic
The existing fallback appears to scan for individual letters [A-J], causing issues when:
- Multiple letters appear in reasoning text
- Letters appear in different contexts
- Final answer is clearly indicated but not in expected format

## Proposed Solution

### Enhanced Filter Configuration
Add comprehensive filter pipelines to MMLU Pro tasks following the paper's specification:

```yaml
# Proposed addition to MMLU Pro _default_template_yaml
filter_list:
  - name: "mmlu_pro_primary"
    filter:
      - function: "regex"
        regex_pattern: "answer is \\(?([A-J])\\)?"
        ignore_case: true
      - function: "take_first"
      
  - name: "mmlu_pro_answer_caps"
    filter:
      - function: "regex"
        regex_pattern: "ANSWER:?\\s*\\(?([A-J])\\)?"
        ignore_case: false  # Explicitly looking for caps
      - function: "take_first"
      
  - name: "mmlu_pro_secondary" 
    filter:
      - function: "regex"
        regex_pattern: "[aA]nswer\\s*:?\\s*\\(?([A-J])\\)?"
        ignore_case: true
      - function: "take_first"
        
  - name: "mmlu_pro_flexible"
    filter:
      - function: "regex"
        regex_pattern: "(?:the\\s+)?answer\\s+is\\s+\\(?([A-J])\\)?"
        ignore_case: true
      - function: "take_first"
        
  - name: "mmlu_pro_final_fallback"
    filter:
      - function: "multi_choice_regex"
        group_select: -1  # Take the last occurrence
        regex_pattern: "\\(?([A-J])\\)?"
        ignore_case: true
        ignore_punctuation: true
      - function: "take_first"
```

### Improved Regex Patterns

#### 1. Primary Pattern (Paper Specified)
```regex
answer is \(?([A-J])\)?
```
**Captures:**
- "answer is A"
- "answer is (B)"
- "answer is C)"

#### 2. All-Caps ANSWER Pattern
```regex
ANSWER:?\s*\(?([A-J])\)?
```
**Captures:**
- "ANSWER: H"
- "ANSWER H"  
- "ANSWER (J)"
- "ANSWER:D"

#### 3. Secondary Pattern (Enhanced from Paper)
```regex
[aA]nswer:?\s*\(?([A-J])\)?
```
**Captures:**
- "Answer: H"
- "answer D"  
- "answer: (J)"
- "ANSWER is C" (with ignore_case: true)

#### 4. Flexible "The Answer Is" Pattern
```regex
(?:the\s+)?answer\s+is\s+\(?([A-J])\)?
```
**Captures:**
- "The answer is D"
- "answer is (C)"
- "ANSWER is B" (with ignore_case: true)

#### 5. Final Fallback Pattern
```regex
\(?([A-J])\)?
```
**With `group_select: -1`** to take the **last** occurrence rather than getting confused by multiple matches.

### Implementation Strategy

#### Option 1: Comprehensive Update (Recommended)
Update the MMLU Pro `_default_template_yaml` to include all filter pipelines, allowing lm-eval to try multiple extraction methods in sequence.

#### Option 2: Backwards Compatible
Add the enhanced filters as a new task variant (e.g., `mmlu_pro_robust`) while keeping the original for comparison.

## Expected Impact

### Performance Improvement
- **Immediate**: Recover ~550 correctly answered samples currently marked invalid
- **Accuracy boost**: Estimated 4-5% improvement in reported scores
- **Reliability**: More consistent extraction across different model output styles

### Better Model Evaluation
- Reduced bias against models with different output formatting preferences
- More accurate representation of actual model capabilities
- Alignment with MMLU Pro paper's intended evaluation methodology