Repository: transformers
Issue: tracker: `generate` composability refactor 
Labels: WIP

## `generate` + composability = more use cases with minimal rewrites

As I write this issue, `generate` is mostly a sequential monolith. Many internal blocks were carved into functions over the last two years, but navigating there as a beginner is still messy. It is also very challenging to adapt `generate` to different tasks and/or modalities, forcing us to overwrite the entire generate function (e.g. [RAG](https://github.com/huggingface/transformers/blob/ccdabc5642bf84849af93f591e207dc625c8e1e1/src/transformers/models/rag/modeling_rag.py#L907), [MusicGen](https://github.com/huggingface/transformers/blob/ccdabc5642bf84849af93f591e207dc625c8e1e1/src/transformers/models/musicgen/modeling_musicgen.py#L1542)). All these aspects make using, documenting, maintaining, and testing `generate` a challenge.

This issue is a tracker for the refactor of `generate`, where we aim to build the structure outlined in [this board](https://miro.com/app/board/uXjVNgMGfaQ=/). Key ideas for this refactor:
👉 All models can use the base `generate` API
👉 Reduce if/else blocks
👉 Reduce the barriers to entry for new decoding methods/modalities/use cases
👉 Reduce per-model overwrites when possible
👉 Add unit tests
👉 Add documentation regarding the structure of `generate`

### Tasks

- [ ] 1. Isolate prefill into a separate function, pulling it from the decoding methods. Note that 
    - a) prefill is done excluding the latest token (`input_ids[:, -1:]`), so we don't compute variables regarding the latest token twice; 
    - b) prefill only runs when `use_cache=True` and cache length < input length - 1; 
    - c) `_expand_inputs_for_generation` needs to be changed (it copied inputs before prefill, we will need to copy prefill outputs)
- [ ] 2. (depends on 1.) Separate generate on the 5 stages described in the diagram, passing around the data structures described therein
- [ ] 3. (depends on 1.) Streaming 2.0
    - a) Add option to `yield`/`yield from` instead of `return` 
    - b) Deprecate the old streamer classes; 
    - c) Add a new class to print the stream into the screen. For beam methods, build a class that prints up to the point where all beams agree with each other.
    - d) thoroughly document and communicate this feature
    - e) enable streaming into the screen with `pipeline`
- [ ] 4. (depends on 2.) Separate stage 1 into a set of functions as described in the diagram. Add unit tests.
- [ ] 5. (depends on 2.) Separate stage 2 into a set of functions as described in the diagram. Add unit tests. Move the preparation of common model inputs here, such as `position_ids`.
- [ ] 6. (depends on 2.) Separate stage 3 into a set of functions as described in the diagram. Add unit tests. Deprecate `LogitsWarper` in this step (it's a copy of `LogitsProcessor`)
- [ ] 7. (depends on 2.) Separate stage 5 into a set of functions as described in the diagram. Add unit tests.
- [ ] 8. Add a new document page walking through the structure of `generate`

[From this point onwards the tasks are only a sketch, need more detailed planning when we get there]
- [ ] 9. Reduce if/elses through templates (e.g. LLMs have a certain default for `prepare_inputs_for_generation`, VLMs also have their special preprocessing steps, ...)
- [ ] 10. Play around with caching of some blocks to determine whether it speeds up generation
- [ ] 11. Rework `prepare_inputs_for_generation` ?
- [ ] 12. Remove `generate` from models that have a custom implementation
- [ ] (other tasks, TBD)