Repository: transformers
Issue: Incorrect Whisper long-form decoding timestamps
Labels: Audio

### System Info

- `transformers` version: 4.42.4
- Platform: Linux-6.8.9-arch1-2-x86_64-with-glibc2.39
- Python version: 3.11.9
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.3.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA GeForce RTX 3090


### Who can help?

@Narsil @sanchit-gandhi


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
import numpy as np
import json
from transformers import pipeline, AutoModelForSpeechSeq2Seq, AutoProcessor
import torch
from datasets import load_dataset

device = "cuda"
torch_dtype = torch.bfloat16
# model_id = "openai/whisper-large-v3"
model_id = "distil-whisper/distil-large-v3"

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=False, use_safetensors=True
)
model.to(device)
processor = AutoProcessor.from_pretrained(model_id)
pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    max_new_tokens=None,
    torch_dtype=torch_dtype,
    device=device,
)

dataset = load_dataset(
    "hf-internal-testing/librispeech_asr_dummy", "clean", split="validation"
)
sample = dataset[0]["audio"]
sample = np.concatenate([sample["array"]] * 10)

results = pipe(
    sample,
    return_timestamps=True,
    generate_kwargs={
        "language": "english",
    },
)
print(json.dumps(results, indent=4))
```

### Output
```json
{
    "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome His Gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel.",
    "chunks": [
        {
            "timestamp": [
                0.0,
                6.5
            ],
            "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel."
        },
        {
            "timestamp": [
                6.5,
                12.5
            ],
            "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel."
        },
        {
            "timestamp": [
                12.5,
                18.24
            ],
            "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome His Gospel."
        },
        {
            "timestamp": [
                18.24,
                24.0
            ],
            "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel."
        },
        {
            "timestamp": [
                24.0,
                29.84
            ],
            "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel."
        },
        {
            "timestamp": [
                0.0,
                4.7
            ],
            "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel."
        },
        {
            "timestamp": [
                5.76,
                10.54
            ],
            "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel."
        },
        {
            "timestamp": [
                11.6,
                16.4
            ],
            "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel."
        },
        {
            "timestamp": [
                17.46,
                22.26
            ],
            "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel."
        },
        {
            "timestamp": [
                23.12,
                28.12
            ],
            "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel."
        }
    ]
}

```


### Expected behavior

Currently the timestamp resets to zero after 30s of audio. I expect the timestamps to increase monotonically.