Repository: EleutherAI/lm-evaluation-harness
Issue: Generator Error when evaluating GLUE and superGLUE
Root cause: Algorithmic Error

The error information shows that:
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/34210233/7bee8db6-9a3c-4f2b-90f0-07400db5c64c)

And the corresponding code is:
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/34210233/4bcf89eb-cebc-4329-8a0d-622b29d6a08c)

I guess the error was caused by the parameter <n_reordered_requests>, which is a generator, and that was assigned by:
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/34210233/e27e2089-0751-4ecf-85fe-24f2a317d0c0)

So I think #1197 commitment changes it and results in this error. When I checkout/back to the previous version from two weeks ago, that's normal.

This should have been fixed in #1229. Are you on the latest commit?

> This should have been fixed in #1229. Are you on the latest commit?

> Yes, the version I used was commit #1238

Sorry, it seems to be ok in #1238, and I didn't realize the bug was fixed yesterday. Thank you!

hmm. Can you provide the full command? The previous bug occurred only when using batch "auto".

> hmm. Can you provide the full command? The previous bug occurred only when using batch "auto".

Yeah,

```
lm_eval --model hf \
    --model_args pretrained=gpt2-xl,trust_remote_code=true,dtype=bfloat16 \
    --tasks glue,gsm8k,super-glue-lm-eval-v1 \
    --batch_size auto \
    --output_path ./eval_out/gpt2-xl \
    --device cuda:0
```

btw, the following command triggered another error
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/34210233/bcdddcd2-5457-47bd-a663-3ea9e628f2ef)
```
lm_eval --model hf \
    --model_args pretrained=Qwen/Qwen-14B-Chat,trust_remote_code=true,dtype=bfloat16 \
    --tasks glue,gsm8k,super-glue-lm-eval-v1 \
    --batch_size auto \
    --output_path ./eval_out/qwen-14b \
    --device cuda:0
```


The second one looks like a tokenizer bug. @haileyschoelkopf 

Even worse, the first command cannot run properly
```
lm_eval --model hf \
    --model_args pretrained=gpt2-xl,trust_remote_code=true,dtype=bfloat16 \
    --tasks glue,gsm8k,super-glue-lm-eval-v1 \
    --batch_size auto \
    --output_path ./eval_out/gpt2-xl \
    --device cuda:0
```
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/34210233/7286fc49-61fb-4b28-8386-2281d10c76e3)


> Even worse, the first command cannot run properly更糟糕的是，第一个命令无法正常运行
> 
> ```
> lm_eval --model hf \
>     --model_args pretrained=gpt2-xl,trust_remote_code=true,dtype=bfloat16 \
>     --tasks glue,gsm8k,super-glue-lm-eval-v1 \
>     --batch_size auto \
>     --output_path ./eval_out/gpt2-xl \
>     --device cuda:0
> ```
> 
> ![image](https://private-user-images.githubusercontent.com/34210233/293877839-7286fc49-61fb-4b28-8386-2281d10c76e3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDQzMDAwMTcsIm5iZiI6MTcwNDI5OTcxNywicGF0aCI6Ii8zNDIxMDIzMy8yOTM4Nzc4MzktNzI4NmZjNDktNjFmYi00YjI4LTgzODYtMjI4MWQxMGM3NmUzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAxMDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMTAzVDE2MzUxN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTU4NWNjM2ExMzE3YmQ3MGJkMmU2MWQ1NGU2M2UwMzExMTU0NDhmNTk5Yjk3OGM0ODdkNGZlODA4OTViOWU1YjUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.RkWY1849pMbvpao3AZENDlny4MnSGTu5nYcqaIH0i04)

Furthermore, I located the error caused by super-glue-lm-eval-v1 evaluating.
May there be any idea for solving it？

Taking a look right now!