Repository: transformers
Issue: Nougat-small VisionEncoderDecoderModel failed when max_new_tokens > 3584,  Index out of range in self
Labels: Good Difficult Issue

### System Info

# System Info

- transformers == 4.35.0
- torch == 2.0.1+cu117
- Error shows on both cpu and cuda.

# Task
Use nougat to parse text from large documents.
# Issue
NougatTokenizerFast. model_max_length = 3584 . Setting max_new_tokens=3585 (or larger) in model.generate works well on most pages, but fails on pages which output should be '[MISSING_PAGE_POST]'.
Once the nougat fails, parsing all the following pages will fail unless kernel restart.
# Error
IndexError: index out of range in self
# What I Tried 

- Setting `nougat_model.config.decoder.pad_token_id = nougat_image_processor.tokenizer.eos_token_id` doesn't work.
- I suspect one special token is generated outside of vocabulary, which prevents from end-of-senescence detection.
- Nougat_base can generate an empty string with max_new_tokens=3585 on the same page, without errors.


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

**Sample Document**
![failed_page](https://github.com/huggingface/transformers/assets/103061109/05ecf35a-67e5-4720-b123-dece84eec195)


**Code**
```
nougat_image_processor = NougatProcessor.from_pretrained("facebook/nougat-small")
nougat_model = VisionEncoderDecoderModel.from_pretrained("facebook/nougat-small")

pixel_values = nougat_image_processor(image, return_tensors="pt").pixel_values

device = 'cpu'#"cuda" if torch.cuda.is_available() else "cpu"
if device == 'cuda':
    pixel_values = pixel_values.cuda()
    nougat_model = nougat_model.cuda()
    
outputs = nougat_model.generate(
    pixel_values.to(device),
    min_length=1,
    max_new_tokens=3585,
    bad_words_ids=[[nougat_image_processor.tokenizer.unk_token_id]],
)
generated = nougat_image_processor.batch_decode(outputs[0], skip_special_tokens=True)[0]
generated = nougat_image_processor.post_process_generation(generated, fix_markdown=False)
print(generated)
```

### Expected behavior

**Expected Output**

'[MISSING_PAGE_POST]' or '  '.

**Observed Output**
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/tmp/ipykernel_11659/2341160804.py in <module>
      6     nougat_model = nougat_model.cuda()
      7 
----> 8 outputs = nougat_model.generate(
      9     pixel_values.to(device),
     10     min_length=1,

~/.conda/envs/py-chunker-client-ipython/lib/python3.9/site-packages/torch/utils/_contextlib.py in decorate_context(*args, **kwargs)
    113     def decorate_context(*args, **kwargs):
    114         with ctx_factory():
--> 115             return func(*args, **kwargs)
    116 
    117     return decorate_context

~/.conda/envs/py-chunker-client-ipython/lib/python3.9/site-packages/transformers/generation/utils.py in generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, negative_prompt_ids, negative_prompt_attention_mask, **kwargs)
   1671         if generation_mode == GenerationMode.GREEDY_SEARCH:
   1672             # 11. run greedy search
-> 1673             return self.greedy_search(
   1674                 input_ids,
   1675                 logits_processor=logits_processor,

~/.conda/envs/py-chunker-client-ipython/lib/python3.9/site-packages/transformers/generation/utils.py in greedy_search(self, input_ids, logits_processor, stopping_criteria, max_length, pad_token_id, eos_token_id, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, synced_gpus, streamer, **model_kwargs)
   2519 
   2520             # forward pass to get next token
-> 2521             outputs = self(
   2522                 **model_inputs,
   2523                 return_dict=True,

~/.conda/envs/py-chunker-client-ipython/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs)
   1499                 or _global_backward_pre_hooks or _global_backward_hooks
   1500                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501             return forward_call(*args, **kwargs)
   1502         # Do not call functions when jit is used
   1503         full_backward_hooks, non_full_backward_hooks = [], []

~/.conda/envs/py-chunker-client-ipython/lib/python3.9/site-packages/transformers/models/vision_encoder_decoder/modeling_vision_encoder_decoder.py in forward(self, pixel_values, decoder_input_ids, decoder_attention_mask, encoder_outputs, past_key_values, decoder_inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict, **kwargs)
    602 
    603         # Decode
--> 604         decoder_outputs = self.decoder(
    605             input_ids=decoder_input_ids,
    606             attention_mask=decoder_attention_mask,

~/.conda/envs/py-chunker-client-ipython/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs)
   1499                 or _global_backward_pre_hooks or _global_backward_hooks
   1500                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501             return forward_call(*args, **kwargs)
   1502         # Do not call functions when jit is used
   1503         full_backward_hooks, non_full_backward_hooks = [], []

~/.conda/envs/py-chunker-client-ipython/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py in forward(self, input_ids, attention_mask, encoder_hidden_states, encoder_attention_mask, head_mask, cross_attn_head_mask, past_key_values, inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict)
   2046 
   2047         # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
-> 2048         outputs = self.model.decoder(
   2049             input_ids=input_ids,
   2050             attention_mask=attention_mask,

~/.conda/envs/py-chunker-client-ipython/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs)
   1499                 or _global_backward_pre_hooks or _global_backward_hooks
   1500                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501             return forward_call(*args, **kwargs)
   1502         # Do not call functions when jit is used
   1503         full_backward_hooks, non_full_backward_hooks = [], []

~/.conda/envs/py-chunker-client-ipython/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py in forward(self, input_ids, attention_mask, encoder_hidden_states, encoder_attention_mask, head_mask, cross_attn_head_mask, past_key_values, inputs_embeds, use_cache, output_attentions, output_hidden_states, return_dict)
   1236 
   1237         # embed positions
-> 1238         positions = self.embed_positions(input, past_key_values_length)
   1239 
   1240         hidden_states = inputs_embeds + positions.to(inputs_embeds.device)

~/.conda/envs/py-chunker-client-ipython/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs)
   1499                 or _global_backward_pre_hooks or _global_backward_hooks
   1500                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501             return forward_call(*args, **kwargs)
   1502         # Do not call functions when jit is used
   1503         full_backward_hooks, non_full_backward_hooks = [], []

~/.conda/envs/py-chunker-client-ipython/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py in forward(self, input_ids, past_key_values_length)
    120         ).expand(bsz, -1)
    121 
--> 122         return super().forward(positions + self.offset)
    123 
    124 

~/.conda/envs/py-chunker-client-ipython/lib/python3.9/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    160 
    161     def forward(self, input: Tensor) -> Tensor:
--> 162         return F.embedding(
    163             input, self.weight, self.padding_idx, self.max_norm,
    164             self.norm_type, self.scale_grad_by_freq, self.sparse)

~/.conda/envs/py-chunker-client-ipython/lib/python3.9/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   2208         # remove once script supports set_grad_enabled
   2209         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 2210     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   2211 
   2212 

IndexError: index out of range in self