Repository: transformers
Issue: Whisper - get probability of detected language
Labels: Feature request,Audio

### System Info

- `transformers` version: 4.38.0.dev0
- Platform: Linux-4.15.0-142-generic-x86_64-with-glibc2.23
- Python version: 3.10.11
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.0+cu121 (True)
- Tensorflow version (GPU?): 2.12.0 (True)

### Who can help?

@sanchit-gandhi I guess, since he's the one who provided the answer in the previous git issue.

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Following #25138, @sanchit-gandhi provided an answer to retrieve the language using Whisper model and processor (since Whisper conditionnal tokens include the language token). He later provided a little adaptation in order to get the probability of the language. This is a nice possibility. However, using the latest version of transformers it seems that it's not possible anymore (that's why I write it as a bug but could also be a feature request).

Quick example in order to check :

```python
language_identification = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small").to("cuda:0")
lid_processor = WhisperProcessor.from_pretrained("openai/whisper-small")

audio, _ = librosa.load(<my_file>, sr=16000)

lid = lid_processor(audio, sampling_rate=16000, return_tensors="pt", truncation=True)
input_features = lid.input_features.to("cuda:0", torch.float32)

outputs = language_identification.generate(input_features, 
      output_scores=True,  
      return_dict_in_generate=True, 
      max_new_tokens=1)

pred_text = lid_processor.batch_decode(outputs.sequences, skip_special_tokens=False)
pred_text
```
`pred_text` is : 
```
['



 80']
```

Here we see the conditionnal tokens as well as my only transcription token ` 80` (because of `max_new_tokens=1`)
The issue is that `outputs.scores` object (which is used for the probabilities of each token. Size is (N_TOKEN, 1, 51865), 51865 is Whisper vocabulary size) only returns the probabilities for the tokens **after** the conditionnal tokens. I.e, `outputs.scores` has a length of only 1 because I asked only 1 token for generation (if I would have wrote 5, I would have got a length of 5).

This means that using the transitions scores computed as follow :

```python
transition_scores = language_identification.compute_transition_scores(
    outputs.sequences, outputs.scores, normalize_logits=True
)
```
will produce only the scores for the tokens generated **after** the specials tokens SoT, lang, task, notimestamps (if not asking for).

I also tried without asking for timestamps because my guess was that since `notimestamps` token is after lang and task, maybe having the `notimestamps` token injected manually was maybe making the code to fall in a special `if condition` where the scores of the previous tokens (lang and task) would be ignored somehow.


### Expected behavior

I would have expected the `outputs.scores` to have the scores for the language token (if language isn't forced obviously) as it was probably meant to be according to the answer in #25138.

With that, we could easily guess the score for the language, and maybe have a ranking (like EN with score of 0.8, FR with score of 0.1 and so on).