Repository: EleutherAI/lm-evaluation-harness
Issue: Qwen 7B: RuntimeError: The expanded size of the tensor (32768) must match the existing size (32767) at non-singleton dimension 1.  
Root cause: External Dependency Breakage

When I try evaluating Qwen 7B, I receive an off-by-one shape error:

```
Running Command:  
            lm_eval --model hf                 --model_args pretrained=Qwen/Qwen-7B,trust_remote_code=True                 --tasks bbh_cot_zeroshot_sports_understanding                 --batch_size auto:4                 --device cuda:4                 --output_path eval_results/Qwen_7B/bbh_cot_zeroshot_sports_understanding                 --log_samples
            
RuntimeError: The expanded size of the tensor (32768) must match the existing size (32767) at non-singleton dimension 1.  Target sizes: [16, 32768, 32, 128].  Tensor sizes: [1, 32767, 1, 1]
```

The off-by-one shape error suggests to me that something might have changed with a tokenizer and I need to add/subtract a token. Could someone please advise?

This might be related to a previous issue: https://github.com/EleutherAI/lm-evaluation-harness/issues/1092

Hi! is there a longer traceback? Would be helpful to know where this error is being raised.

Yes, of course :)

```
$ lm_eval --model hf                 --model_args pretrained=Qwen/Qwen-7B,trust_remote_code=True                 --tasks bbh_cot_zeroshot_sports_understanding                 --batch_size auto:4                 --device cuda:4                 --output_path eval_results/Qwen_7B/bbh_cot_zeroshot_sports_understanding                 --log_samples
2024-03-25:08:54:39,922 INFO     [__main__.py:225] Verbosity set to INFO
2024-03-25:08:54:39,922 INFO     [__init__.py:373] lm_eval.tasks.initialize_tasks() is deprecated and no longer necessary. It will be removed in v0.4.2 release. TaskManager will instead be used.
2024-03-25:08:54:43,338 INFO     [__main__.py:311] Selected Tasks: ['bbh_cot_zeroshot_sports_understanding']
2024-03-25:08:54:43,338 INFO     [__main__.py:312] Loading selected tasks...
2024-03-25:08:54:43,340 INFO     [evaluator.py:129] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234
2024-03-25:08:54:44,896 WARNING  [logging.py:61] Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
2024-03-25:08:54:44,897 INFO     [huggingface.py:162] Using device 'cuda:4'
2024-03-25:08:54:45,481 WARNING  [modeling_qwen.py:943] The model is automatically converting to bf16 for faster inference. If you want to disable the automatic precision, please manually add bf16/fp16/fp32=True to "AutoModelForCausalLM.from_pretrained".
2024-03-25:08:54:45,481 WARNING  [modeling_qwen.py:969] Try importing flash-attention for faster inference...
2024-03-25:08:54:45,482 WARNING  [modeling_qwen.py:89] Warning: import flash_attn rotary fail, please install FlashAttention rotary to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/rotary
2024-03-25:08:54:45,482 WARNING  [modeling_qwen.py:98] Warning: import flash_attn rms_norm fail, please install FlashAttention layer_norm to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm
2024-03-25:08:54:45,482 WARNING  [modeling_qwen.py:118] Warning: import flash_attn fail, please install FlashAttention to get higher efficiency https://github.com/Dao-AILab/flash-attention
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████| 8/8 [00:02<00:00,  2.71it/s]
2024-03-25:08:54:49,370 INFO     [evaluator.py:190] get_task_dict has been updated to accept an optional argument, `task_manager`Read more here:https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/interface.md#external-library-usage
2024-03-25:08:54:51,209 WARNING  [task.py:322] [Task: bbh_cot_zeroshot_sports_understanding] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended.
2024-03-25:08:54:51,209 WARNING  [task.py:322] [Task: bbh_cot_zeroshot_sports_understanding] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended.
2024-03-25:08:54:51,215 INFO     [task.py:395] Building contexts for bbh_cot_zeroshot_sports_understanding on rank 0...
100%|███████████████████████████████████████████████████████████████████████████| 250/250 [00:00<00:00, 2105.13it/s]
2024-03-25:08:54:51,338 INFO     [evaluator.py:357] Running generate_until requests
Running generate_until requests:   0%|                                                      | 0/250 [00:00<?, ?it/s]Passed argument batch_size = auto. Detecting largest batch size
Traceback (most recent call last):
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/lfs/skampere1/0/rschaef/KoyejoLab-Predictable-LLM-Evals/submodules/lm-evaluation-harness/lm_eval/__main__.py", line 318, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/lfs/skampere1/0/rschaef/KoyejoLab-Predictable-LLM-Evals/submodules/lm-evaluation-harness/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "/lfs/skampere1/0/rschaef/KoyejoLab-Predictable-LLM-Evals/submodules/lm-evaluation-harness/lm_eval/evaluator.py", line 230, in simple_evaluate
    results = evaluate(
  File "/lfs/skampere1/0/rschaef/KoyejoLab-Predictable-LLM-Evals/submodules/lm-evaluation-harness/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "/lfs/skampere1/0/rschaef/KoyejoLab-Predictable-LLM-Evals/submodules/lm-evaluation-harness/lm_eval/evaluator.py", line 368, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/lfs/skampere1/0/rschaef/KoyejoLab-Predictable-LLM-Evals/submodules/lm-evaluation-harness/lm_eval/models/huggingface.py", line 1111, in generate_until
    batch_size = self._detect_batch_size()
  File "/lfs/skampere1/0/rschaef/KoyejoLab-Predictable-LLM-Evals/submodules/lm-evaluation-harness/lm_eval/models/huggingface.py", line 643, in _detect_batch_size
    batch_size = forward_batch()
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/accelerate/utils/memory.py", line 136, in decorator
    return function(batch_size, *args, **kwargs)
  File "/lfs/skampere1/0/rschaef/KoyejoLab-Predictable-LLM-Evals/submodules/lm-evaluation-harness/lm_eval/models/huggingface.py", line 638, in forward_batch
    out = F.log_softmax(self._model_call(test_batch, **call_kwargs), dim=-1)  # noqa: F841
  File "/lfs/skampere1/0/rschaef/KoyejoLab-Predictable-LLM-Evals/submodules/lm-evaluation-harness/lm_eval/models/huggingface.py", line 744, in _model_call
    return self.model(inps).logits
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/lfs/skampere1/0/rschaef/.cache/huggingface/modules/transformers_modules/Qwen/Qwen-7B/ef3c5c9c57b252f3149c1408daf4d649ec8b6c85/modeling_qwen.py", line 1043, in forward
    transformer_outputs = self.transformer(
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/lfs/skampere1/0/rschaef/.cache/huggingface/modules/transformers_modules/Qwen/Qwen-7B/ef3c5c9c57b252f3149c1408daf4d649ec8b6c85/modeling_qwen.py", line 891, in forward
    outputs = block(
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/lfs/skampere1/0/rschaef/.cache/huggingface/modules/transformers_modules/Qwen/Qwen-7B/ef3c5c9c57b252f3149c1408daf4d649ec8b6c85/modeling_qwen.py", line 610, in forward
    attn_outputs = self.attn(
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/lfs/skampere1/0/rschaef/.cache/huggingface/modules/transformers_modules/Qwen/Qwen-7B/ef3c5c9c57b252f3149c1408daf4d649ec8b6c85/modeling_qwen.py", line 490, in forward
    query = query * logn_tensor.expand_as(query)
RuntimeError: The expanded size of the tensor (32768) must match the existing size (32767) at non-singleton dimension 1.  Target sizes: [16, 32768, 32, 128].  Tensor sizes: [1, 32767, 1, 1]
Running generate_until requests:   0%|                                                      | 0/250 [00:01<?, ?it/s]
```

@RylanSchaeffer I suspect the following line https://huggingface.co/Qwen/Qwen-7B/blob/ef3c5c9c57b252f3149c1408daf4d649ec8b6c85/modeling_qwen.py#L291 should be up to `32768` inclusive. 

patching `32768->32769` seems to run, but then I get a tokenizer error of some kind (Qwen's tokenizer also uses custom code).