Add Flux and Flux Controlnet Support to Diffusion folder
State: open | Pull request: yes
Labels: name: Multi Modal

name: Run CICD

name: PoR

# What does this PR do ?

Add Flux and Flux Controlnet Support 

**Collection**: [DIFFUSION]


```python
torchrun flux_controlnet_infer.py --num_joint_layers 4 --num_single_layers 0 --flux_ckpt /ckpts/nemo_flux_transformer.safetensors --controlnet_ckpt /ckpts/nemo_flux_controlnet_transformer.safetensors --control_image /ckpts/FLUX.1-controlnet-lineart-promeai/images/example-control.jpg
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.




@ethanhe42 I have moved the dit scripts into scrips folder as a NeMo 2 convention. Please take a look.

<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```
************* Module nemo.collections.diffusion.models.flux_controlnet.model
nemo/collections/diffusion/models/flux_controlnet/model.py:352:0: C0301: Line too long (120/119) (line-too-long)
************* Module nemo.collections.diffusion.encoders.conditioner
nemo/collections/diffusion/encoders/conditioner.py:23:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/diffusion/encoders/conditioner.py:170:0: C0115: Missing class docstring (missing-class-docstring)
************* Module nemo.collections.diffusion.models.dit.dit_attention
nemo/collections/diffusion/models/dit/dit_attention.py:380:0: C0301: Line too long (122/119) (line-too-long)
nemo/collections/diffusion/models/dit/dit_attention.py:382:0: C0301: Line too long (122/119) (line-too-long)
nemo/collections/diffusion/models/dit/dit_attention.py:15:0: C0115: Missing class docstring (missing-class-docstring)
************* Module nemo.collections.diffusion.models.dit.dit_layer_spec
nemo/collections/diffusion/models/dit/dit_layer_spec.py:54:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/diffusion/models/dit/dit_layer_spec.py:60:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/diffusion/models/dit/dit_layer_spec.py:66:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/diffusion/models/dit/dit_layer_spec.py:632:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/diffusion/models/dit/dit_layer_spec.py:21:0: W0611: Unused rearrange imported from einops (unused-import)
nemo/collections/diffusion/models/dit/dit_layer_spec.py:22:0: W0611: Unused jit_fuser imported from megatron.core.jit (unused-import)
************* Module nemo.collections.diffusion.models.flux.layers
nemo/collections/diffusion/models/flux/layers.py:131:0: C0115: Missing class docstring (missing-class-docstring)
************* Module nemo.collections.diffusion.models.flux.model
nemo/collections/diffusion/models/flux/model.py:720:0: C0301: Line too long (122/119) (line-too-long)
nemo/collections/diffusion/models/flux/model.py:74:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/diffusion/models/flux/model.py:107:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/diffusion/models/flux/model.py:113:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/diffusion/models/flux/model.py:120:0: C0115: Missing class docstring (missing-class-docstring)
************* Module nemo.collections.diffusion.models.flux.pipeline
nemo/collections/diffusion/models/flux/pipeline.py:385:0: C0301: Line too long (120/119) (line-too-long)
nemo/collections/diffusion/models/flux/pipeline.py:492:0: C0301: Line too long (120/119) (line-too-long)
nemo/collections/diffusion/models/flux/pipeline.py:494:0: C0301: Line too long (120/119) (line-too-long)
nemo/collections/diffusion/models/flux/pipeline.py:800:0: C0301: Line too long (134/119) (line-too-long)
nemo/collections/diffusion/models/flux/pipeline.py:839:0: C0301: Line too long (125/119) (line-too-long)
nemo/collections/diffusion/models/flux/pipeline.py:650:0: C0115: Missing class docstring (missing-class-docstring)
************* Module nemo.collections.diffusion.utils.flux_pipeline_utils
nemo/collections/diffusion/utils/flux_pipeline_utils.py:15:0: W0611: Unused dataclass imported from dataclasses (unused-import)
nemo/collections/diffusion/utils/flux_pipeline_utils.py:17:0: W0611: Unused import torch (unused-import)
************* Module nemo.collections.diffusion.utils.mcore_parallel_utils
nemo/collections/diffusion/utils/mcore_parallel_utils.py:26:0: C0115: Missing class docstring (missing-class-docstring)
************* Module nemo.collections.diffusion.vae.autoencoder
nemo/collections/diffusion/vae/autoencoder.py:26:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/diffusion/vae/autoencoder.py:48:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/diffusion/vae/autoencoder.py:149:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/diffusion/vae/autoencoder.py:264:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/diffusion/vae/autoencoder.py:279:0: C0115: Missing class docstring (missing-class-docstring)
************* Module nemo.lightning._strategy_lib
nemo/lightning/_strategy_lib.py:586:0: C0301: Line too long (130/119) (line-too-long)
nemo/lightning/_strategy_lib.py:35:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/_strategy_lib.py:36:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/_strategy_lib.py:141:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/_strategy_lib.py:168:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/_strategy_lib.py:204:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/_strategy_lib.py:517:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/_strategy_lib.py:611:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/_strategy_lib.py:624:4: C0115: Missing class docstring (missing-class-docstring)
************* Module nemo.lightning.megatron_parallel
nemo/lightning/megatron_parallel.py:245:0: C0301: Line too long (127/119) (line-too-long)
nemo/lightning/megatron_parallel.py:246:0: C0301: Line too long (140/119) (line-too-long)
nemo/lightning/megatron_parallel.py:247:0: C0301: Line too long (130/119) (line-too-long)
nemo/lightning/megatron_parallel.py:554:0: C0301: Line too long (129/119) (line-too-long)
nemo/lightning/megatron_parallel.py:561:0: C0301: Line too long (135/119) (line-too-long)
nemo/lightning/megatron_parallel.py:849:0: C0301: Line too long (137/119) (line-too-long)
nemo/lightning/megatron_parallel.py:1079:0: C0301: Line too long (136/119) (line-too-long)
nemo/lightning/megatron_parallel.py:1652:0: C0301: Line too long (128/119) (line-too-long)
nemo/lightning/megatron_parallel.py:1691:0: C0301: Line too long (146/119) (line-too-long)
nemo/lightning/megatron_parallel.py:71:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:72:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:74:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:109:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:113:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:313:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:337:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:363:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:389:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:525:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:569:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:573:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:639:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:674:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:680:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:686:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:693:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:700:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:734:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:742:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:758:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:785:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:797:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:819:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1345:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1520:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:1526:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1532:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1536:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1541:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:1546:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:1574:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1620:8: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1642:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:1715:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:1761:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1775:0: C0116: Missing function or method docstring (missing-function-docstring)
************* Module nemo.lightning.pytorch.optim.base
nemo/lightning/pytorch/optim/base.py:166:4: C0116: Missing function or method docstring (missing-function-docstring)

-----------------------------------
Your code has been rated at 9.72/10
```

---

Mitigation guide:

* Add sensible and useful docstrings to functions and methods
* For trivial methods like getter/setters, consider adding `# pylint: disable=C0116` inside the function itself
* To disable multiple functions/methods at once, put a `# pylint: disable=C0116` before the first and a `# pylint: enable=C0116` after the last.

By applying these rules, we reduce the occurance of this message in future.

Thank you for improving NeMo's documentation!