Repository: EleutherAI/lm-evaluation-harness
Issue: Error when evaluating on math dataset
Root cause: Algorithmic Error

Meet an error when running the following code:
`
python main.py --model hf-causal --model_args prined=/nvme2/llama-7b-hf --tasks math_algebra --device cuda:6 --num_fewshot 1
`
The error is:
Running greedy_until requests
  0%|                                                                             | 0/1187 [00:00<?, ?it/s]
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /nvme2/lm-evaluation-harness/main.py:93 in <module>                                              │
│                                                                                                  │
│   90                                                                                             │
│   91                                                                                             │
│   92 if __name__ == "__main__":                                                                  │
│ ❱ 93 │   main()                                                                                  │
│   94                                                                                             │
│                                                                                                  │
│ /nvme2/lm-evaluation-harness/main.py:59 in main                                                  │
│                                                                                                  │
│   56 │   │   with open(args.description_dict_path, "r") as f:                                    │
│   57 │   │   │   description_dict = json.load(f)                                                 │
│   58 │                                                                                           │
│ ❱ 59 │   results = evaluator.simple_evaluate(                                                    │
│   60 │   │   model=args.model,                                                                   │
│   61 │   │   model_args=args.model_args,                                                         │
│   62 │   │   tasks=task_names,                                                                   │
│                                                                                                  │
│ /nvme2/lm-evaluation-harness/lm_eval/utils.py:243 in _wrapper                                    │
│                                                                                                  │
│   240 │   │   │   │   "deprecated and will be disallowed in a future version of "                │
│   241 │   │   │   │   "lm-evaluation-harness!"                                                   │
│   242 │   │   │   )                                                                              │
│ ❱ 243 │   │   return fn(*args, **kwargs)                                                         │
│   244 │                                                                                          │
│   245 │   return _wrapper                                                                        │
│   246                                                                                            │
│                                                                                                  │
│ /nvme2/lm-evaluation-harness/lm_eval/evaluator.py:94 in simple_evaluate                          │
│                                                                                                  │
│    91 │   if check_integrity:                                                                    │
│    92 │   │   run_task_tests(task_list=tasks)                                                    │
│    93 │                                                                                          │
│ ❱  94 │   results = evaluate(                                                                    │
│    95 │   │   lm=lm,                                                                             │
│    96 │   │   task_dict=task_dict,                                                               │
│    97 │   │   num_fewshot=num_fewshot,                                                           │
│                                                                                                  │
│ /nvme2/lm-evaluation-harness/lm_eval/utils.py:243 in _wrapper                                    │
│                                                                                                  │
│   240 │   │   │   │   "deprecated and will be disallowed in a future version of "                │
│   241 │   │   │   │   "lm-evaluation-harness!"                                                   │
│   242 │   │   │   )                                                                              │
│ ❱ 243 │   │   return fn(*args, **kwargs)                                                         │
│   244 │                                                                                          │
│   245 │   return _wrapper                                                                        │
│   246                                                                                            │
│                                                                                                  │
│ /nvme2/lm-evaluation-harness/lm_eval/evaluator.py:289 in evaluate                                │
│                                                                                                  │
│   286 │   │   #       they should end up next to each other.                                     │
│   287 │   │                                                                                      │
│   288 │   │   print("Running", reqtype, "requests")                                              │
│ ❱ 289 │   │   resps = getattr(lm, reqtype)([req.args for req in reqs])                           │
│   290 │   │   resps = [                                                                          │
│   291 │   │   │   x if req.index is None else x[req.index] for x, req in zip(resps, reqs)        │
│   292 │   │   ]                                                                                  │
│                                                                                                  │
│ /nvme2/lm-evaluation-harness/lm_eval/base.py:893 in fn                                           │
│                                                                                                  │
│   890 │   │   │   │   │   remaining_reqs.append(req)                                             │
│   891 │   │   │                                                                                  │
│   892 │   │   │   # actually run the LM on the requests that do not have cached results          │
│ ❱ 893 │   │   │   rem_res = getattr(self.lm, attr)(remaining_reqs)                               │
│   894 │   │   │                                                                                  │
│   895 │   │   │   # stick the new ones back into the list and also cache any of the new ones     │
│   896 │   │   │   resptr = 0                                                                     │
│                                                                                                  │
│ /nvme2/lm-evaluation-harness/lm_eval/base.py:413 in greedy_until                                 │
│                                                                                                  │
│   410 │   │   │   │   until = [until]                                                            │
│   411 │   │   │                                                                                  │
│   412 │   │   │   if until:                                                                      │
│ ❱ 413 │   │   │   │   (primary_until,) = self.tok_encode(until[0])                               │
│   414 │   │   │   else:                                                                          │
│   415 │   │   │   │   primary_until = None                                                       │
│   416                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: too many values to unpack (expected 1)

Any solutions?

I am also getting the same error. 

It looks like the dataset was changed on us. We’ll need to modify the preprocessing to make sure it loads correctly.

It shouldn’t be too hard to work out manually, if you’re down to submit a PR. 

Closed as we have solved this problem.

Hi Stella! Sorry for commenting on this one as I see you solved it, is the fix in the refactor branch as I can't find it? Or maybe I'm hitting a related issue.

I'm using the main branch and I see that the `doc["solution"]` is permuted with `answer` in the `math_algebra` task [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/hendrycks_math.py#L77)

it looks to me as if this:
```
if self.is_equiv(
            answer, self.remove_boxed(self.last_boxed_only_string(doc["solution"]))
        ):
```

should actually be this:
```
if self.is_equiv(
            doc["answer"], self.remove_boxed(self.last_boxed_only_string(answer))
        ):
```

> Hi Stella! Sorry for commenting on this one as I see you solved it, is the fix in the refactor branch as I can't find it? Or maybe I'm hitting a related issue.
> 
> I'm using the main branch and I see that the `doc["solution"]` is permuted with `answer` in the `math_algebra` task [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/hendrycks_math.py#L77)
> 
> it looks to me as if this:
> 
> ```
> if self.is_equiv(
>             answer, self.remove_boxed(self.last_boxed_only_string(doc["solution"]))
>         ):
> ```
> 
> should actually be this:
> 
> ```
> if self.is_equiv(
>             doc["answer"], self.remove_boxed(self.last_boxed_only_string(answer))
>         ):
> ```

When I throw the following into a Google Colab notebook it runs:
```
!git clone https://github.com/EleutherAI/lm-evaluation-harness
%cd lm-evaluation-harness
!pip install -e .
!python main.py --model hf-causal --model_args pretrained=gpt2 --tasks math_algebra --num_fewshot 1
```