Repository: EleutherAI/lm-evaluation-harness
Issue: mbpp extract_code filter failed to correctly filter out the generated code.
Root cause: Validation Gap

I'm trying to apply mbpp benchmark on my locally deployed qwen3-32b model (sglang) with the instruction below
```
lm_eval \
      --batch_size 64 \
      --seed 42 \
      --model local-completions \
      --tasks mbpp_instruct \
      --model_args model=qwen/qwen3-32b,tokenizer=qwen/qwen3-32b,base_url=http://localhost:30000/v1/completions \
      --gen_kwargs '{{\"temperature\":0,\"top_p\":1,\"top_k\":1,\"min_p\":0.0}}' \
      --confirm_run_unsafe_code \
      --apply_chat_template \
      --log_samples \
```
but i get this confusing result
```
|    Tasks    |Version|   Filter   |n-shot| Metric  |   |Value|   |Stderr|
|-------------|------:|------------|-----:|---------|---|----:|---|-----:|
|mbpp_instruct|      1|extract_code|     3|pass_at_1|↑  |    0|±  |     0|
```

```json
{
  "doc_id": 6,
  "doc": {
    "task_id": 17,
    "text": "Write a function to find the perimeter of a square.",
    "code": "def square_perimeter(a):\r\n  perimeter=4*a\r\n  return perimeter",
    "test_list": [
      "assert square_perimeter(10)==40",
      "assert square_perimeter(5)==20",
      "assert square_perimeter(4)==16"
    ],
    "test_setup_code": "",
    "challenge_test_list": []
  },
  "target": "assert square_perimeter(10)==40\nassert square_perimeter(5)==20\nassert square_perimeter(4)==16",
  "arguments": {
    "gen_args_0": {
      "arg_0": "
user\nYou are an expert Python programmer, and here is your task:\nWrite a function to find the similar elements from the given two tuple lists.\nYour code should pass these tests:\nassert similar_elements((3, 4, 5, 6),(5, 7, 4, 10)) == (4, 5)\nassert similar_elements((1, 2, 3, 4),(5, 4, 3, 7)) == (3, 4)\nassert similar_elements((11, 12, 14, 13),(17, 15, 14, 13)) == (13, 14)\n```python\n def similar_elements(test_tup1, test_tup2):\r\n  res = tuple(set(test_tup1) & set(test_tup2))\r\n  return (res) \n```\n\nYou are an expert Python programmer, and here is your task:\nWrite a python function to identify non-prime numbers.\nYour code should pass these tests:\nassert is_not_prime(2) == False\nassert is_not_prime(10) == True\nassert is_not_prime(35) == True\n```python\n import math\r\ndef is_not_prime(n):\r\n    result = False\r\n    for i in range(2,int(math.sqrt(n)) + 1):\r\n        if n % i == 0:\r\n            result = True\r\n    return result\n```\n\nYou are an expert Python programmer, and here is your task:\nWrite a function to find the largest integers from a given list of numbers using heap queue algorithm.\nYour code should pass these tests:\nassert heap_queue_largest( [25, 35, 22, 85, 14, 65, 75, 22, 58],3)==[85, 75, 65] \nassert heap_queue_largest( [25, 35, 22, 85, 14, 65, 75, 22, 58],2)==[85, 75] \nassert heap_queue_largest( [25, 35, 22, 85, 14, 65, 75, 22, 58],5)==[85, 75, 65, 58, 35]\n```python\n import heapq as hq\r\ndef heap_queue_largest(nums,n):\r\n  largest_nums = hq.nlargest(n, nums)\r\n  return largest_nums\n```\n\nYou are an expert Python programmer, and here is your task:\nWrite a function to find the perimeter of a square.\nYour code should pass these tests:\nassert square_perimeter(10)==40\nassert square_perimeter(5)==20\nassert square_perimeter(4)==16
\n
assistant\n<think>\n\n</think>\n\n```python\n",
      "arg_1": {
        "max_gen_toks": 256,
        "until": [],
        "do_sample": false,
        "temperature": 0,
        "top_p": 1,
        "top_k": 1,
        "min_p": 0.0
      }
    }
  },
  "resps": [
    [
      "def square_perimeter(side):\n    return 4 * side\n```"
    ]
  ],
  "filtered_resps": [
    [
      " square_perimeter(side):\n    return 4 * side"
    ]
  ],
  "filter": "extract_code",
  "metrics": [
    "pass_at_1"
  ],
  "doc_hash": "6b2d989c185b127268af6f851011464865a0b1446f46dbcaa32d4626718c94bf",
  "prompt_hash": "da6439c9bb5c664b4b8f5bc11f025f524367e55a2951a7b01e8ff27e7a750e4d",
  "target_hash": "9ebbefdbc542871d7de9505793c9c99c3368d12deeda96473450773b2dfa9d90",
  "
```
it seems that the extract_code filter failed to correctly filter out the generated code.
how to solve it?

Hello, I met the same problem on qwen2.5-72B, have you solved it?

> Hello, I met the same problem on qwen2.5-72B, have you solved it?

Not solved yet