possible (proven) research direction -> SCD - seperable causable diffusion
you were asking for suggestions - so I couldn't resist sharing this -
(my 5090 gpu is in for repairs so this isn't over the line ) - but i point you to some research to reduce compute made possible by this paper
https://arxiv.org/abs/2602.10095
this was my findings with LTX2 - it worked (though it need far more training)
see the proven reconstructions on wandb.
https://github.com/Lightricks/LTX-2/issues/168#event-29619039281
this is minimax work in progress
https://github.com/johndpope/MiniMax-H3/commit/a78474141207998c36da9039412bbd7a22fdb609
read this - even if you throw code or disregard the paper
https://github.com/johndpope/MiniMax-H3/blob/a6fdabb50c3139032d320c5116a8cfc741fbd5e8/docs/MINIMAX_H3_SCD_PORT_DESIGN.md
so there's no change in layers - just when we process through
because the layers are chopped up - the opportunity is far less compute - less ram - infinite generation of video capped by hdd space.