Repository: transformers
Issue: Strange behavior with attn_implementation="eager"
Labels: bug,Multimodal

### System Info

- `transformers` version: 4.47.0
- Platform: Linux-5.15.0-120-generic-x86_64-with-glibc2.35
- Python version: 3.10.15
- Huggingface_hub version: 0.26.2
- Safetensors version: 0.4.5
- Accelerate version: 1.1.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA A100-PCIE-40GB

### Who can help?

@zucchini-nlp

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am trying to analyze the attention pattern of the `LLAVA v1.5 7B` model, so I used `attn_implementation="eager"` when initing the model to obtain the attention weights. However, this has led to several issues. Firstly, the output IDs are incorrect, and secondly, errors may occur. I've noticed that this problem only appears with specific images and user prompts, while it does not occur in other cases, which is quite peculiar. Below is my code:
``` python
import numpy as np
import torch
from dotenv import load_dotenv
from PIL import Image
from transformers import (
    LlavaForConditionalGeneration,
    LlavaProcessor,
)
from transformers.generation.utils import GenerateDecoderOnlyOutput

np.set_printoptions(threshold=np.inf)
model_name = "llava-hf/llava-1.5-7b-hf"

model: LlavaForConditionalGeneration = LlavaForConditionalGeneration.from_pretrained(
    model_name,
    cache_dir="/root/llm/utils/models/hub",
    torch_dtype=torch.float16,
    low_cpu_mem_usage=True,
    device_map="cuda:0",
    attn_implementation="eager",
)
processor: LlavaProcessor = LlavaProcessor.from_pretrained(
    model_name,
    cache_dir="/root/llm/utils/models/hub",
    padding_side="left",
    patch_size=model.config.vision_config.patch_size,
    vision_feature_select_strategy=model.config.vision_feature_select_strategy,
)

images = [
    Image.open("/root/llm/utils/eval/Object_HalBench/images/339761.jpg"),
    Image.open("/root/llm/utils/eval/Object_HalBench/images/431256.jpg"),
]
users = [
    "Provide a thorough description of the given image.",
    "What is this photo about? Please answer in great detail.",
]

prompts: list[str] = []
for u in users:
    conversation: list[dict[str]] = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {"type": "text", "text": u},
            ],
        },
    ]
    prompt: str = processor.apply_chat_template(
        conversation,
        tokenize=False,
        add_generation_prompt=True,
    )
    prompts.append(prompt)

with torch.inference_mode():
    encoded_inputs: dict[str, torch.Tensor] = processor(
        images=images,
        text=prompts,
        return_tensors="pt",
        return_token_type_ids=False,
        padding=True,
    ).to("cuda:0", torch.float16)

    output: GenerateDecoderOnlyOutput = model.generate(
        **encoded_inputs,
        max_new_tokens=50,
        num_beams=1,
        do_sample=False,
        temperature=0.7,
        output_attentions=True,
        use_cache=True,
        return_legacy_cache=True,
        return_dict_in_generate=True,
    )
generated_ids: list[torch.LongTensor] = output.sequences  # list of shape (batch_size, sequence_length)
print(generated_ids.cpu().numpy())
generated_ids = [o[len(i) :] for i, o in zip(encoded_inputs.input_ids, generated_ids)]
print()
decoded_outputs: list[str] = processor.batch_decode(
    generated_ids,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=True,
)
print(decoded_outputs)
decoded_outputs = [d.rstrip("\n").strip(" ") for d in decoded_outputs]
print(decoded_outputs)
print(len(output.attentions))
```
Notice: the image I used is from Object_HalBench benchmark

The output is:
Some other warning:
```
/root/anaconda3/envs/LVLM/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:628: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.7` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
  warnings.warn(
Expanding inputs for image tokens in LLaVa should be done in processing. Please add `patch_size` and `vision_feature_select_strategy` to the model's processing config or set directly with `processor.patch_size = {{patch_size}}` and processor.vision_feature_select_strategy = {{vision_feature_select_strategy}}`. Using processors without these attributes in the config is deprecated and will throw an error in v4.50.
From v4.47 onwards, when a model cache is to be returned, `generate` will return a `Cache` instance instead by default (as opposed to the legacy tuple of tuples format). If you want to keep returning the legacy format, please set `return_legacy_cache=True`.
```
The generated_ids: (I remove a large number of `<image>` token for readability)
```
[[32001     1  3148  1001 29901 29871 32000 32000 32000 32000 32000 32000
  32000 32000 32000 32000 32000 32000 29871    13  1184 29894   680   263
  17826  6139   310   278  2183  1967 29889   319  1799  9047 13566 29901
      0     0     0     0     0     0     0     0     0     0     0     0
      0     0     0     0     0     0     0     0     0     0     0     0
      0     0     0     0     0     0     0     0     0     0     0     0
      0     0     0     0     0     0     0     0     0     0     0     0
      0     0]
 [    1  3148  1001 29901 29871 32000 32000 32000 32000 32000 32000 32000
  32000 32000 32000 32000 32000 29871    13  5618   338   445 15373  1048
  29973  3529  1234   297  2107  9493 29889   319  1799  9047 13566 29901
    450  1967  4332  1973   263 15007  3377   261   297  3158 29892 15859
    263  8938   373   263 15007 29899 11911   287   364  1160 29889   450
  15007  3377   261   338   297   278  7256   310   278  9088 29892   411
   1009 15007  3377  7962 19540   963 29889 29871    13    13  8439   526
   3196   916]]
```
Notice that `<image>`: 32000, `<pad>`: 32001

The output after `batch_decode` is:
```
['', 'The image captures a snowboarder in action, performing a trick on a snow-covered ramp. The snowboarder is in the middle of the scene, with their snowboard visible beneath them. \n\nThere are several other']
```
It's strange that there is token id 0 generated. 

Only set `output_attentions=False` and `return_dict_in_generate=False` without removing `attn_implementation="eager",` won't make any change.

Notice that removing `attn_implementation="eager"`, and not returning dict overcome this question, the output then become correct:
```
[[32001     1  3148  1001 29901 29871 32000 32000 32000 32000 32000 32000
  32000 32000 32000 32000 32000 32000 29871    13  1184 29894   680   263
  17826  6139   310   278  2183  1967 29889   319  1799  9047 13566 29901
    450  1967  5680   263 27683  8345   411   263  2919  7933  8024 15678
    701   278 10090 29892  4969   263   301  1878   322   325  4626   424
  25005 29889   450  8024   338 24046  2978   278  1510   261  4038 29892
   4417   263  6023   310  5469   304   278  2913 29889 29871    13    13
    797   278]
 [    1  3148  1001 29901 29871 32000 32000 32000 32000 32000 32000 32000
  32000 32000 32000 32000 32000 29871    13  5618   338   445 15373  1048
  29973  3529  1234   297  2107  9493 29889   319  1799  9047 13566 29901
    450  1967  4332  1973   263 15007  3377   261   297  3158 29892 15859
    263  8938   373   263 15007 29899 11911   287   364  1160 29889   450
  15007  3377   261   338   297   278  7256   310   278  9088 29892   411
   1009 15007  3377  7962 19540   963 29889 29871    13    13  8439   526
   3196   916]]

['The image features a bathroom with a large green plant growing up the wall, creating a lush and vibrant atmosphere. The plant is situated near the shower area, adding a touch of nature to the space. \n\nIn the',
'The image captures a snowboarder in action, performing a trick on a snow-covered ramp. The snowboarder is in the middle of the scene, with their snowboard visible beneath them. \n\nThere are several other']
```
Beside this, some error may occur with `attn_implementation="eager"`, in some other case (different Image input)
```
Loading checkpoint shards: 100%|████| 3/3 [00:03<00:00,  1.12s/it]
../aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion `probability tensor contains either `inf`, `nan` or element < 0` failed.
Traceback (most recent call last):
  File "/root/llm/LVLM/test2.py", line 38, in <module>
    print(generator.gen(images, users,do_sample=True,
  File "/root/llm/LVLM/model/generator/llava.py", line 184, in gen
    out = gen_hf(
  File "/root/llm/LVLM/model/generator/utils.py", line 279, in gen_hf
    output, encoded_inputs = _gen_hf(
  File "/root/llm/LVLM/model/generator/utils.py", line 229, in _gen_hf
    output: GenerateDecoderOnlyOutput = model.generate(
  File "/root/anaconda3/envs/LVLM/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/root/anaconda3/envs/LVLM/lib/python3.10/site-packages/transformers/generation/utils.py", line 2252, in generate
    result = self._sample(
  File "/root/anaconda3/envs/LVLM/lib/python3.10/site-packages/transformers/generation/utils.py", line 3297, in _sample
    next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

```

### Expected behavior

fix it