Repository: transformers
Issue: Problems when using SinkCache for model.generate()
Labels: WIP,Generation,Cache

### System Info

- `transformers` version: 4.41.2
- Platform: Linux-5.4.0-173-generic-x86_64-with-glibc2.31
- Python version: 3.10.0
- Huggingface_hub version: 0.23.3
- Safetensors version: 0.4.3
- Accelerate version: 0.31.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@zucchini-nlp @gante @tomaarse

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am trying to use sinkcache for multi-turn dialog where the cache should contains the previous turn's dialog. Here is my code:

```
import torch

from transformers import SinkCache

from transformers import AutoConfig, LlamaForCausalLM, AutoTokenizer

model_id = "meta-llama/Llama-2-7b-hf"
model = LlamaForCausalLM.from_pretrained(
      model_id, low_cpu_mem_usage=True, device_map='auto',
      torch_dtype=torch.bfloat16,cache_dir="cache")
model = model.eval()
tokenizer = AutoTokenizer.from_pretrained(model_id,cache_dir="cache")
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = 'left'

device = model.device

cache = SinkCache(window_length=1024, num_sink_tokens=4)
prefix_list=["hello,my name is yy","what is my name?"]
for prefix in prefix_list:
    inputs = tokenizer(prefix, return_tensors='pt').to(device)

    
    gen_out = model.generate(**inputs, do_sample=False, max_new_tokens=64,
                            use_cache=True,
                            past_key_values=cache,
                            pad_token_id=tokenizer.pad_token_id)
    
    decoded = tokenizer.batch_decode(gen_out, skip_special_tokens=True)

    print(decoded)
```
I am not sure if  I was doing it right. But I got the following error:

```
Traceback (most recent call last):
  File "/data/yaoy/long_context/repeat_sirllm/main.py", line 25, in <module>
    gen_out = model.generate(**inputs, do_sample=False, max_new_tokens=64,
  File "/data/yaoy/anaconda/envs/long/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data/yaoy/anaconda/envs/long/lib/python3.10/site-packages/transformers/generation/utils.py", line 1758, in generate
    result = self._sample(
  File "/data/yaoy/anaconda/envs/long/lib/python3.10/site-packages/transformers/generation/utils.py", line 2390, in _sample
    model_kwargs = self._get_initial_cache_position(input_ids, model_kwargs)
  File "/data/yaoy/anaconda/envs/long/lib/python3.10/site-packages/transformers/generation/utils.py", line 1326, in _get_initial_cache_position
    model_kwargs["cache_position"] = torch.arange(past_length, cur_len, device=input_ids.device)
RuntimeError: upper bound and larger bound inconsistent with step sign
```

### Expected behavior

 Cache should include information from previous conversations.