Repository: huggingface/lighteval
Issue: [BUG] GSM8k Zero Accuracy
Root cause: Validation Gap

## Describe the bug

When I run the GSM8K eval, I get 0 accuracy.  If I save details, I see that the `predictions` look like follows:

```
['Please reason step by step, and put your final answer within \\boxed{}.ichtig\n']
```

I'm not sure if this is related to the closed issue #202 , but I've seen others having this issue as well

## To Reproduce

```bash

export VLLM_WORKER_MULTIPROC_METHOD=spawn
model="Qwen/Qwen2.5-Math-1.5B"
seed=0
MODEL_ARGS="pretrained=$model,dtype=bfloat16,max_model_length=4096,gpu_memory_utilization=0.9,generation_parameters={max_new_tokens:4096,temperature:0.6,top_p:0.95,seed:$seed}"

lighteval vllm "$MODEL_ARGS" "leaderboard|gsm8k|0|0" --use-chat-template --output-dir "data" --save-details
```

## Expected behavior

I do not have this issue with other evals.  e.g. gpqa:diamond runs fine. It generates outputs and nonzero accuracies

## Version info

lighteval is version `'0.8.1'`, and vllm is version `'0.8.4'`


Thank you for your help!

yes. the gsm8k parser doesn't work very well. if you want you can use my custom gsm8k_normalizer

```
def gsm8k_normalizer(text: str) -> str:
    # step 1: if the model generated too much, get the first part
    text = text.split("\nQ:")[0]

    # step 2: extract \boxed{ }
    extracted_nums = re.findall(r"\\boxed\{(.*?)\}", repr(text))
    if extracted_nums:
        extracted_nums = extracted_nums[-1]
    else:
        extracted_nums = text
    # step 2: pick the last number
    # try:
    extracted_nums = re.sub(r"[^a-zA-Z0-9\s.,-]", " ", extracted_nums)

    return greedy_extract_xml_answer(extracted_nums)

def greedy_extract_xml_answer(text: str) -> str:
    """
    Greedy extract will try to extract valid answer within \\boxed{..}
    If no valid answer is found, it returns the last digit in the text pharse
    """
    pattern = r"(?<!\S)(?P<prefix>(?:-\$|\$-|\$|-)?)(?P<number>(?:(?:\d{1,3}(?:,\d{3})+)|\d+)(?:\.\d*)?)(?:%)?(?!\S)"
    match = re.findall(pattern, text)
    if match:
        prefix = match[-1][0]  # may include '-' if the number is negative
        number = match[-1][1]  # match.group("number")

        # Remove commas from the number (e.g., "1,765" -> "1765")
        number = number.replace(",", "")

        # Remove a trailing ".00" if present.
        if number.endswith(".00"):
            number = number[:-3]
        # If there's a lone trailing dot (e.g., "32."), remove it.
        elif number.endswith("."):
            number = number[:-1]

        # Determine if a minus sign appears anywhere in the prefix.
        sign = "-" if "-" in prefix else ""
        return sign + number
    try:
        text = text.replace(",", "")
        extracted_nums = [s for s in re.findall(r"-?\d+\.?\d*", text)]
        return str(float(extracted_nums[-1]))
    except Exception:
        return "[invalid]"

```

I found it takes longer to use `lighteval:gsm8k` than to use `leaderboard:gsm8k`

leaderboard:gsm8k
![Image](https://github.com/user-attachments/assets/8b3a0f1d-d16f-4529-9fbd-9578525419f1)

lighteval:gsm8k
![Image](https://github.com/user-attachments/assets/63241f73-df72-4c0c-8255-dd2c303ea082)

why did `lighteval:gsm8k` execute twice? 

> I found it takes longer to use `lighteval:gsm8k` than to use `leaderboard:gsm8k`
> 
> leaderboard:gsm8k ![Image](https://github.com/user-attachments/assets/8b3a0f1d-d16f-4529-9fbd-9578525419f1)
> 
> lighteval:gsm8k ![Image](https://github.com/user-attachments/assets/63241f73-df72-4c0c-8255-dd2c303ea082)
> 
> why did `lighteval:gsm8k` execute twice?

~In this case, lighteval creates two splits. The first one has 1319 questions, which is the test set of GSM8k. The second one has 10552 questions. I'm not sure what these are, but I found [this paper GSM-Plus](https://arxiv.org/abs/2402.19255v1), where they have exactly 10552 questions, so that could be. Although I couldn't find any references to it in the lighteval docs~

UPDATE: nvm, see below 

I encounter the same bug

Hi ! Thanks for the issue

> In this case, lighteval creates two splits. The first one has 1319 questions, which is the test set of GSM8k. The second one has 10552 questions. I'm not sure what these are, but I found [this paper GSM-Plus](https://arxiv.org/abs/2402.19255v1), where they have exactly 10552 questions, so that could be. Although I couldn't find any references to it in the lighteval docs

This is because we use two metrics for gs8mk, one of them requiring to run the same prompt 8 times, so 10552 questions total)

@candemircan you are having weird generations because your model is not instruct and you are using chat templates, I would recommend not using chat templates and using 5 shots for base models.

I have oopened a PR (#688) to change the metric funciton we use to a better one.

```
lighteval vllm "model_name=Qwen/Qwen2.5-Math-1.5B,dtype=bfloat16,max_model_length=4096,gpu_memory_utilization=0.9,generation_parameters={max_new_tokens:4096,temperature:0.6,top_p:0.95}" "lighteval|gsm8k|5|0"  --output-dir "data" --save-details --max-samples 100
```

yields

```
|      Task       |Version|     Metric     |Value|   |Stderr|
|-----------------|------:|----------------|----:|---|-----:|
|all              |       |extractive_match| 0.74|±  |0.0441|
|lighteval:gsm8k:5|      0|extractive_match| 0.74|±  |0.0441|
```

hi @NathanHB, thanks for the quick response and for the clarification! this resolves the issue for me 

PS: `model_name=Qwen/Qwen2.5-Math-1.5B` does not work for me with vllm, but `pretrained=Qwen/Qwen2.5-Math-1.5B` does. Not sure if this is just me, but I saw your README also uses `model_name` whereas examples on the huggingface page use `pretrained`. Sharing in case its unintentional

the documentation for main uses `model_name`, we haven't made the release yet !

I found it's hard to evaluate thinking models, for example, deepseek qwen 7b will output like this:
```
"Alright, let's tackle this problem step by step. So, Tim has a box containing 7 blue shoe boxes and 9 red shoe boxes. That means initially, he has a total of 7 + 9 = 16 shoe boxes.\n\nNow, Tim uses 3 blue shoe boxes for going fishing. So, subtracting those from the blue shoe boxes, we have 7 - 3 = 4 blue shoe boxes remaining.\n\nNext, he uses 1/3 of his red shoe boxes for fishing. He originally has 9 red shoe boxes. Calculating 1/3 of 9 gives us 3 red shoe boxes used. Therefore, the remaining red shoe boxes are 9 - 3 = 6.\n\nFinally, to find the total number of shoe boxes left, we add the remaining blue and red shoe boxes: 4 + 6 = 10.\n\nSo, Tim has 10 shoe boxes left in his box after using some for fishing.\n</think>\n\nTim has a box with **7 blue shoe boxes** and **9 red shoe boxes**, totaling **16 shoe boxes**.\n\nHe uses **3 blue shoe boxes** for fishing, leaving him with:\n\\[ 7 \\text{ blue shoe boxes} - 3 = 4 \\text{ blue shoe boxes} \\]\n\nHe also uses **\\(\\frac{1}{3}\\)** of his **9 red shoe boxes** for fishing:\n\\[ \\frac{1}{3} \\times 9 = 3 \\text{ red shoe boxes} \\]\nSo, the remaining red shoe boxes are:\n\\[ 9 \\text{ red shoe boxes} - 3 = 6 \\text{ red shoe boxes} \\]\n\nAdding the remaining blue and red shoe boxes:\n\\[ 4 \\text{ blue shoe boxes} + 6 \\text{ red shoe boxes} = 10 \\text{ shoe boxes} \\]\n\n**Final Answer:**  \nThere are **10 shoe boxes** left in Tim's box."
```

They don't follow the output format. I can deal with this by remove the --use-chat-template, but if I do that, the model will no thinking. How can I deal with this.

> Hi ! Thanks for the issue
> 
> > In this case, lighteval creates two splits. The first one has 1319 questions, which is the test set of GSM8k. The second one has 10552 questions. I'm not sure what these are, but I found [this paper GSM-Plus](https://arxiv.org/abs/2402.19255v1), where they have exactly 10552 questions, so that could be. Although I couldn't find any references to it in the lighteval docs
> 
> This is because we use two metrics for gs8mk, one of them requiring to run the same prompt 8 times, so 10552 questions total)
> 
> [@candemircan](https://github.com/candemircan) you are having weird generations because your model is not instruct and you are using chat templates, I would recommend not using chat templates and using 5 shots for base models.
> 
> I have oopened a PR ([#688](https://github.com/huggingface/lighteval/pull/688)) to change the metric funciton we use to a better one.
> 
> ```
> lighteval vllm "model_name=Qwen/Qwen2.5-Math-1.5B,dtype=bfloat16,max_model_length=4096,gpu_memory_utilization=0.9,generation_parameters={max_new_tokens:4096,temperature:0.6,top_p:0.95}" "lighteval|gsm8k|5|0"  --output-dir "data" --save-details --max-samples 100
> ```
> 
> yields
> 
> ```
> |      Task       |Version|     Metric     |Value|   |Stderr|
> |-----------------|------:|----------------|----:|---|-----:|
> |all              |       |extractive_match| 0.74|±  |0.0441|
> |lighteval:gsm8k:5|      0|extractive_match| 0.74|±  |0.0441|
> ```

@NathanHB Hi, I ran the same CLI, but I got different results which is shown in the below.

<img width="671" height="102" alt="Image" src="https://github.com/user-attachments/assets/0bed9d78-bce3-44c6-b507-a8f1fab586b5" />

How can I reproduce the performance that you reported.