Repository: EleutherAI/lm-evaluation-harness
Issue: hf mamba failing with generate tasks 
Root cause: Interface Contract Mismatch

Here's an error i get. I was able to resolve by just removing "attention_mask" from generate_kwargs here: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/huggingface.py#L857
Although I'm not sure how this fix will affect other models. I am using the 0.4.4 version of the eval harness, but it doesn't look like it's fixed in the latest version either?

```
(perplexity-correlations) tthrush@jagupard19:/juice5/scr5/tthrush/perplexity-correlations/examples/get_error_and_bpb$ lm_eval --model hf --model_args pretrained=state-spaces/mamba-130m-hf --tasks coqa --limit 5000 --device cuda:0
2024-11-14:17:21:24,640 INFO     [__main__.py:279] Verbosity set to INFO
2024-11-14:17:21:24,728 INFO     [__init__.py:491] `group` and `group_alias` keys in TaskConfigs are deprecated and will be removed in v0.4.5 of lm_eval. The new `tag` field will be used to allow for a shortcut to a group of tasks one does not wish to aggregate metrics across. `group`s which aggregate across subtasks must be only defined in a separate group config file, which will be the official way to create groups that support cross-task aggregation as in `mmlu`. Please see the v0.4.4 patch notes and our documentation: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#advanced-group-configs for more information.
2024-11-14:17:21:32,920 WARNING  [__main__.py:312]  --limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2024-11-14:17:21:32,921 INFO     [__main__.py:376] Selected Tasks: ['coqa']
2024-11-14:17:21:32,925 INFO     [evaluator.py:161] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234
2024-11-14:17:21:32,925 INFO     [evaluator.py:198] Initializing hf model, with arguments: {'pretrained': 'state-spaces/mamba-130m-hf'}
2024-11-14:17:21:33,010 WARNING  [other.py:349] Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
2024-11-14:17:21:33,010 INFO     [huggingface.py:130] Using device 'cuda:0'
2024-11-14:17:21:33,306 INFO     [huggingface.py:366] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
/nlp/scr/tthrush/miniconda3/envs/perplexity-correlations/lib/python3.12/site-packages/mamba_ssm/ops/selective_scan_interface.py:163: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @custom_fwd
/nlp/scr/tthrush/miniconda3/envs/perplexity-correlations/lib/python3.12/site-packages/mamba_ssm/ops/selective_scan_interface.py:239: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
  @custom_bwd
/nlp/scr/tthrush/miniconda3/envs/perplexity-correlations/lib/python3.12/site-packages/mamba_ssm/ops/triton/layer_norm.py:985: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @custom_fwd
/nlp/scr/tthrush/miniconda3/envs/perplexity-correlations/lib/python3.12/site-packages/mamba_ssm/ops/triton/layer_norm.py:1044: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
  @custom_bwd
/nlp/scr/tthrush/miniconda3/envs/perplexity-correlations/lib/python3.12/site-packages/mamba_ssm/distributed/tensor_parallel.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @custom_fwd
/nlp/scr/tthrush/miniconda3/envs/perplexity-correlations/lib/python3.12/site-packages/mamba_ssm/distributed/tensor_parallel.py:61: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
  @custom_bwd
/nlp/scr/tthrush/miniconda3/envs/perplexity-correlations/lib/python3.12/site-packages/mamba_ssm/ops/triton/ssd_combined.py:757: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @custom_fwd
/nlp/scr/tthrush/miniconda3/envs/perplexity-correlations/lib/python3.12/site-packages/mamba_ssm/ops/triton/ssd_combined.py:835: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
  @custom_bwd
2024-11-14:17:21:38,847 INFO     [evaluator.py:279] Setting fewshot random generator seed to 1234
2024-11-14:17:21:38,847 WARNING  [model.py:422] model.chat_template was called with the chat_template set to False or None. Therefore no chat template will be applied. Make sure this is an intended behavior.
2024-11-14:17:21:38,864 INFO     [task.py:423] Building contexts for coqa on rank 0...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [00:00<00:00, 56175.72it/s]
2024-11-14:17:21:39,110 INFO     [evaluator.py:465] Running generate_until requests
Running generate_until requests:   0%|                                                                                      | 0/500 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/nlp/scr/tthrush/miniconda3/envs/perplexity-correlations/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/nlp/scr/tthrush/miniconda3/envs/perplexity-correlations/lib/python3.12/site-packages/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nlp/scr/tthrush/miniconda3/envs/perplexity-correlations/lib/python3.12/site-packages/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/nlp/scr/tthrush/miniconda3/envs/perplexity-correlations/lib/python3.12/site-packages/lm_eval/evaluator.py", line 301, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/nlp/scr/tthrush/miniconda3/envs/perplexity-correlations/lib/python3.12/site-packages/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/nlp/scr/tthrush/miniconda3/envs/perplexity-correlations/lib/python3.12/site-packages/lm_eval/evaluator.py", line 476, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nlp/scr/tthrush/miniconda3/envs/perplexity-correlations/lib/python3.12/site-packages/lm_eval/models/huggingface.py", line 1279, in generate_until
    cont = self._model_generate(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/nlp/scr/tthrush/miniconda3/envs/perplexity-correlations/lib/python3.12/site-packages/lm_eval/models/huggingface.py", line 842, in _model_generate
    return self.model.generate(
           ^^^^^^^^^^^^^^^^^^^^
  File "/nlp/scr/tthrush/miniconda3/envs/perplexity-correlations/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/nlp/scr/tthrush/miniconda3/envs/perplexity-correlations/lib/python3.12/site-packages/transformers/generation/utils.py", line 1689, in generate
    self._validate_model_kwargs(model_kwargs.copy())
  File "/nlp/scr/tthrush/miniconda3/envs/perplexity-correlations/lib/python3.12/site-packages/transformers/generation/utils.py", line 1243, in _validate_model_kwargs
    raise ValueError(
ValueError: The following `model_kwargs` are not used by the model: ['attention_mask'] (note: typos in the generate arguments will also show up in this list)
```

Hi! Thanks for reporting the issue. `mamba_ssm` model type should be able to handle the HF variants as well now.