Release 2.0.0rc1
State: closed | Pull request: yes
Labels: name: core

name: TTS

name: ASR

name: NLP

name: CI

name: common

name: Multi Modal

name: audio

🚀 PR to release NeMo `2.0.0rc1`.

📝 Please remember the following to-do's before merge:
- [ ] Fill-in the comment `Highlights`
- [ ] Review the comment `Detailed Changelogs`

🚨 Please also keep in mind to _not_ delete the headings of the task commits. They are required by the post-merge automation.

🙏 Please merge this PR only if the CI workflow completed successfully.

# Highlights
_<here-goes-the-summary...>_

# Detailed Changelogs
## ASR

<details><summary>Changelog</summary>

- [Nemo CICD] ASR_dev_run_Speech_To_Text_HF_Finetuning optional as flaky by @pablo-garay :: PR: #9180
- GPU-based vectorized Specaug Version 2 by @amorari-nvidia :: PR: #9155
- Support dataloader as input to `audio` for transcription by @titu1994 :: PR: #9201
- Clean up dev docs collection section by @yaoyu-33 :: PR: #9205
- Fix Online_Offline_Microphone_VAD_Demo.ipynb by @stevehuang52 :: PR: #9251
- Remove .nemo instead of renaming by @mikolajblaz :: PR: #9281
- Fix GreedyBatchedCTCInfer regression from GreedyCTCInfer. by @galv :: PR: #9347
- Revert "Fix GreedyBatchedCTCInfer regression from GreedyCTCInfer." by @titu1994 :: PR: #9351
- Prompt formatter API and canary transcribe tensor input support by @pzelasko :: PR: #9206
- Fix prompt formatter's defaults=None case in multi-task model by @pzelasko :: PR: #9366
- move AED chunked infer script by @stevehuang52 :: PR: #9367
- Use model-cast-to-bfloat16 rather than AMP-to-bfloat16 for inference. by @galv :: PR: #9198
- ci: Fix `L2_Segmentation_Tool_Parallel_ctc_segmentation_test_L2_Eng_C… by @ko3n1g :: PR: #9399
- Fix logging message for ASR by @titu1994 :: PR: #9469
- Adding 'use_dynamo' option for export to use onnx.dynamo_export() instead of onnx.export() by @borisfom :: PR: #9147
- Add support to change Multi task model prompt by @titu1994 :: PR: #9542
- Enable encoder adapters for Canary and MultiTaskAED models by @titu1994 :: PR: #9409
- Audio model collection by @anteju :: PR: #9263
- TitaNet Batch Verify Speaker by @monica-sekoyan :: PR: #9337
- Fix the arguments  of forward_for_export function in msdd_models by @tango4j :: PR: #9624
- chore: Pin branch in notebooks by @ko3n1g :: PR: #9697
- refactor: notebook branch release by @ko3n1g :: PR: #9711
- Canary Adapters tutorial (#9670) by @nithinraok :: PR: #9777

</details>

## TTS

<details><summary>Changelog</summary>

- move tts fixtures by @blisc :: PR: #9183
- Clean up dev docs collection section by @yaoyu-33 :: PR: #9205
- Add mel codec checkpoints by @anteju :: PR: #9228
- Adding 'use_dynamo' option for export to use onnx.dynamo_export() instead of onnx.export() by @borisfom :: PR: #9147
- GPU unit tests: Mark flaky tests to be fixed by @pablo-garay :: PR: #9559
- chore: Pin branch in notebooks by @ko3n1g :: PR: #9697
- refactor: notebook branch release by @ko3n1g :: PR: #9711

</details>

## NLP / NMT

<details><summary>Changelog</summary>

- Change FIM Dataset Random Seed Init by @suiyoubi :: PR: #9165
- Fix MoE EP rank when TP is set at the same time by @gdengk :: PR: #9161
- Alit/optim 8k by @JRD971000 :: PR: #9166
- enable matryoshka embedding learning by @arendu :: PR: #9130
- Implement async distributed checkpoint save by @mikolajblaz :: PR: #9028
- Fix incorrect checkpoint removal logic by @mikolajblaz :: PR: #9192
- Update to using Model Optimizer (formerly AMMO) in PTQ workflow by @janekl :: PR: #9178
- Update nemo.export module for quantized models by @janekl :: PR: #9218
- Add save option to the TRT-LLM export test script by @oyilmaz-nvidia :: PR: #9221
- Checkpoint resuming compatible for 2403 container by @suiyoubi :: PR: #9199
- support QWen1.5/QWen2 by @Agoniii :: PR: #9055
- Implement export with PyT Distributed checkpoints by @mikolajblaz :: PR: #9058
- Clean up dev docs collection section by @yaoyu-33 :: PR: #9205
- use get with fallback when reading checkpoint_callback_params by @akoumpa :: PR: #9223
- Revert rope fusion defaults by @cuichenx :: PR: #9237
- fix import by @akoumpa :: PR: #9240
- Add TRT-LLM params like max_num_tokens and opt_num_tokens by @oyilmaz-nvidia :: PR: #9210
- sum-reduce grad_norm in DP+CP domain by @erhoo82 :: PR: #9262
- Add llama3 and distributed checkpoint support in NeVA by @yaoyu-33 :: PR: #9101
- RAG Pipeline by @huvunvidia :: PR: #9143
- Mcore dist opt ckpt fix by @akoumpa :: PR: #9156
- Alit/bert convert fix by @JRD971000 :: PR: #9285
- conv1d stable version by @JRD971000 :: PR: #9330
- Fix trainer builder when exp_manager is not in config by @yaoyu-33 :: PR: #9293
- Fix Peft Weights Loading in NeVA by @yaoyu-33 :: PR: #9341
- Skip sequence_parallel allreduce when using Mcore DistOpt by @akoumpa :: PR: #9344
- Fix FSDP gradient calculation with orig params by @janEbert :: PR: #9335
- TRT-LLM Export Code Cleanup by @oyilmaz-nvidia :: PR: #9270
- support null/None truncation field by @arendu :: PR: #9355
- NeVa token fusion by @paul-gibbons :: PR: #9245
- bugfix if using mcore distOpt with sft by @akoumpa :: PR: #9356
- Re-org export code by @oyilmaz-nvidia :: PR: #9353
- Add HF siglip vision encoder by @HuiyingLi :: PR: #9185
- Extend multimodal/speech_llm with lhotse, t5 and bestow supports by @zhehuaichen :: PR: #9169
- QLoRA by @cuichenx :: PR: #9340
- PeFT fix for distOpt by @akoumpa :: PR: #9392
- [NeMo-UX] Integrating mcore's DistributedDataParallel into MegatronStrategy by @marcromeyn :: PR: #9387
- cherry pick of #9266 by @dimapihtar :: PR: #9411
- Enable specifying alpha for PTQ INT8 SmoothQuant method by @janekl :: PR: #9423
- add support for new mcore ds features by @dimapihtar :: PR: #9388
- LoRA for MoE Layer by @cuichenx :: PR: #9396
- Mistral-7B: apply user's precision to output checkpoint by @akoumpa :: PR: #9222
- Add option to merge distributed optimizer buckets by @timmoon10 :: PR: #9414
- TRT-LLM 0.10 Update by @oyilmaz-nvidia :: PR: #9402
- In-framework deployment by @oyilmaz-nvidia :: PR: #9438
- Bugfix missing variables and argument changes to MegatronPretrainingRandomSampler by @jstjohn :: PR: #9458
- Hyena Operator by @guyjacob :: PR: #9264
- Refactor Quantizer for reusing in QAT by @kevalmorabia97 :: PR: #9276
- S3 Dirpath + Async Uploading Support for Default Checkpoints by @alxzhang-amazon :: PR: #9045
- move load state dict after initialize parallel state in nlp_model by @ryxli :: PR: #9382
- Enable user to optionally upgrade Megatron by @jstjohn :: PR: #9478
- Fix unwrap model by @cuichenx :: PR: #9480
- fix operator precedence by @akoumpa :: PR: #9403
- [NeMo-UX] Adding context- & expert-parallelism to MegatronStrategy by @marcromeyn :: PR: #9525
- update mcoreddp call by @akoumpa :: PR: #9345
- mcore distOpt restore fix by @akoumpa :: PR: #9421
- vLLM Export Support by @apanteleev :: PR: #9381
- PL: Delete precision if using plugin. TODO switch to MegatronTrainerB… by @akoumpa :: PR: #9535
- extend get_gpt_layer_modelopt_spec to support MoE by @akoumpa :: PR: #9532
- fix mock data generation for legacy dataset by @dimapihtar :: PR: #9530
- add reset learning rate functionality by @dimapihtar :: PR: #9372
- Adding 'use_dynamo' option for export to use onnx.dynamo_export() instead of onnx.export() by @borisfom :: PR: #9147
- Use closed-formula to round by multiple by @akoumpa :: PR: #9307
- GPU unit tests: Mark flaky tests to be fixed by @pablo-garay :: PR: #9559
- Consolidate gpt continue training script into pretraining script by @yaoyu-33 :: PR: #9413
- Enable encoder adapters for Canary and MultiTaskAED models by @titu1994 :: PR: #9409
- PTQ refinements by @janekl :: PR: #9574
- Add ModelOpt QAT example for Llama2 SFT model by @kevalmorabia97 :: PR: #9326
- Multimodal projection layer adapter fix for PP>1 by @paul-gibbons :: PR: #9445
- Add offline quantization script for QLoRA deployment by @cuichenx :: PR: #9455
- Make QLoRA more model-agnostic by @cuichenx :: PR: #9488
- Set n_gpu to None in nemo export by @oyilmaz-nvidia :: PR: #9593
- [NeMo-UX] Fix Megatron-optimizer by @marcromeyn :: PR: #9599
- Chat template support for megatron_gpt_eval.py by @akoumpa :: PR: #9354
- [NeMo-UX] Add PEFT by @cuichenx :: PR: #9490
- Alit/mamba tmp by @JRD971000 :: PR: #9612
- Enable MCore checkpointing optimizations by @mikolajblaz :: PR: #9505
- Change mixtral moe key name for trt-llm by @oyilmaz-nvidia :: PR: #9620
- fix ckpt load bug by @dimapihtar :: PR: #9621
- Alit/mamba by @JRD971000 :: PR: #9575
- Unwrap ckpt_io for model opt (async save) by @mikolajblaz :: PR: #9622
- MCore T5 support for NeMo - Training by @huvunvidia :: PR: #9432
- [Nemo-UX] Expose transformer_layer_spec inside GPTConfig by @marcromeyn :: PR: #9592
- Update NeMo Clip to Use MCore Modules by @yaoyu-33 :: PR: #9594
- Mistral + Mixtral Support for NeVa by @paul-gibbons :: PR: #9459
- Adding support for mcore generate by @shanmugamr1992 :: PR: #9566
- Improve error messaging during trt-llm export by @oyilmaz-nvidia :: PR: #9638
- [Cherrypick] support lora when kv_channel != hidden_size / num_heads by @cuichenx :: PR: #9644
- Parametrize FPS group by @mikolajblaz :: PR: #9648
- Cherry-pick megatron export fix from main by @borisfom :: PR: #9643
- add documentation for reset_lr feature by @dimapihtar :: PR: #9639
- chore: Pin branch in notebooks by @ko3n1g :: PR: #9697
- Cherry pick: LITA Integration by @Slyne :: PR: #9684
- SDXL improvements (and support for Draft+) by @rohitrango :: PR: #9654
- Gemma 2 by @cuichenx :: PR: #9672
- Allows non-strict load with distributed checkpoints by @mikolajblaz :: PR: #9613
- refactor: notebook branch release by @ko3n1g :: PR: #9711
- [NeMo-UX] Make TE and Apex dependencies optional by @ashors1 :: PR: #9550
- Alit/r2.0.0 by @JRD971000 :: PR: #9718
- Manually cherry-pick from PR 9679 (PR to main - Support SFT/Eval/PEFT for mcore T5) by @huvunvidia :: PR: #9737
- In framework export by @oyilmaz-nvidia :: PR: #9658

</details>

## Export

<details><summary>Changelog</summary>

- TRTLLM new API support by @meatybobby :: PR: #9003
- Update nemo.export module for quantized models by @janekl :: PR: #9218
- Add save option to the TRT-LLM export test script by @oyilmaz-nvidia :: PR: #9221
- Implement export with PyT Distributed checkpoints by @mikolajblaz :: PR: #9058
- Add TRT-LLM params like max_num_tokens and opt_num_tokens by @oyilmaz-nvidia :: PR: #9210
- TRT-LLM Export Code Cleanup by @oyilmaz-nvidia :: PR: #9270
- Re-org export code by @oyilmaz-nvidia :: PR: #9353
- Use TensorRT-LLM native parameter names in nemo.export module by @janekl :: PR: #9424
- TRT-LLM 0.10 Update by @oyilmaz-nvidia :: PR: #9402
- vLLM Export Support by @apanteleev :: PR: #9381
- Add page context fmha option in TensorRTLLM export by @meatybobby :: PR: #9526
- Test C++ runtime on demand in nemo_export.py to avoid possible OOMs by @janekl :: PR: #9544
- Adding 'use_dynamo' option for export to use onnx.dynamo_export() instead of onnx.export() by @borisfom :: PR: #9147
- Fix nemo export test by @oyilmaz-nvidia :: PR: #9547
- Add tps and pps params to the export script by @oyilmaz-nvidia :: PR: #9558
- Add Multimodal Exporter by @meatybobby :: PR: #9256
- Set n_gpu to None in nemo export by @oyilmaz-nvidia :: PR: #9593
- Inflight nemo model export support by @JimmyZhang12 :: PR: #9527
- vLLM Export Improvements by @apanteleev :: PR: #9596
- Akoumparouli/nemo ux mixtral export by @akoumpa :: PR: #9603
- Change mixtral moe key name for trt-llm by @oyilmaz-nvidia :: PR: #9620
- Fix the arguments  of forward_for_export function in msdd_models by @tango4j :: PR: #9624
- Improve error messaging during trt-llm export by @oyilmaz-nvidia :: PR: #9638
- Cherry-pick megatron export fix from main by @borisfom :: PR: #9643
- In framework export by @oyilmaz-nvidia :: PR: #9658

</details>

## Bugfixes

<details><summary>Changelog</summary>

- Fix incorrect checkpoint removal logic by @mikolajblaz :: PR: #9192
- use get with fallback when reading checkpoint_callback_params by @akoumpa :: PR: #9223
- fix import by @akoumpa :: PR: #9240
- Remove .nemo instead of renaming by @mikolajblaz :: PR: #9281
- call set_expert_model_parallel_world_size instead of set_cpu_expert_m… by @akoumpa :: PR: #9275
- Fix typos in Mixtral NeMo->HF and Starcoder2 NeMo->HF conversion scripts by @evellasques :: PR: #9325
- Skip sequence_parallel allreduce when using Mcore DistOpt by @akoumpa :: PR: #9344

</details>

## Uncategorized:
  
  <details><summary>Changelog</summary>
  
  - Update minor version by @ericharper :: PR: #9181
  - [Nemo CICD] Increase time limit for Speech_Checkpoints_tests by @pablo-garay :: PR: #9186
  - Bucketing duration bins: less optimal but instant init when not provided + fixes in estimation script by @pzelasko :: PR: #9157
  - Enable CUDA graphs by default only for transcription by @github-actions[bot] :: PR: #9197
  - Add guards to SD imports by @yaoyu-33 :: PR: #9158
  - [Nemo CICD] run_cicd_for_release_branches_also by @pablo-garay :: PR: #9213
  - Update index.rst by @jgerh :: PR: #9080
  - ci: Speeding NeMo-CI up by using caching by @ko3n1g :: PR: #9174
  - rename paths2audiofiles to audio by @github-actions[bot] :: PR: #9220
  - ci: Multi-tenancy for tests and garbage collection by @ko3n1g :: PR: #9179
  - Lhotse Sharding Fix by @tbartley94 :: PR: #9187
  - Fix ASR_Context_Biasing.ipynb contains FileNotFoundError by @github-actions[bot] :: PR: #9234
  - NeMo Dev Doc Feature Updates 1: Some parallelisms by @yaoyu-33 :: PR: #9184
  - ci: Remove duplicated job by @ko3n1g :: PR: #9258
  - Fix document links by @yaoyu-33 :: PR: #9260
  - Pin transformers by @github-actions[bot] :: PR: #9273
  - Fix loading github raw images on notebook by @github-actions[bot] :: PR: #9283
  - Accept None as an argument to decoder_lengths in GreedyBatchedCTCInfer::forward by @github-actions[bot] :: PR: #9278
  - Refactor Sequence Packing Script by @cuichenx :: PR: #9271
  - [Nemo-UX] Move code to collections + fix some small bugs by @marcromeyn :: PR: #9277
  - Fix typo in HF tutorial by @github-actions[bot] :: PR: #9304
  - Expand documentation for data parallelism and distributed optimizer by @timmoon10 :: PR: #9227
  - Update flash attention section in memory_optimizations.rst by @cyanguwa :: PR: #9188
  - Install alerting by @ko3n1g :: PR: #9311
  - typos by @github-actions[bot] :: PR: #9315
  - FP8 feature documentation by @ksivaman :: PR: #9265
  - [Nemo CICD] Comment out flaky tests by @pablo-garay :: PR: #9333
  - Fixed typos in README.rst by @gdevakumar :: PR: #9322
  - Update README.rst to clarify installation via Conda by @SimonCW :: PR: #9323
  - [Nemo CICD] update flaky test by @pablo-garay :: PR: #9339
  - fix lora and ptuning and isort/black by @github-actions[bot] :: PR: #9295
  - Fix P-tuning for Llama based models by @github-actions[bot] :: PR: #9300
  - add large model stable training fix and contrastive loss update for variable seq by @github-actions[bot] :: PR: #9348
  - Nemo readme revisions by @jgerh :: PR: #9129
  - Guard cuda memory allocator update by @github-actions[bot] :: PR: #9313
  - [Nemo CICD] Remove unnecessary commented out code by @pablo-garay :: PR: #9364
  - Update Gemma conversion script by @yaoyu-33 :: PR: #9365
  - Fix GreedyBatchedCTCInfer regression from GreedyCTCInfer. (#9347) by @github-actions[bot] :: PR: #9371
  - Re-enable cuda graphs in training modes. by @github-actions[bot] :: PR: #9343
  - fix typo infer_seq_lenght -> infer_seq_length by @akoumpa :: PR: #9370
  - Rachitg/ag by @github-actions[bot] :: PR: #9083
  - Make a backward compatibility for old MSDD configs in label models by @github-actions[bot] :: PR: #9378
  - Dgalvez/fix greedy batch strategy name r2.0.0rc0 by @github-actions[bot] :: PR: #9253
  - Update README.rst by @jgerh :: PR: #9393
  - a2a fix removed tp world size and group from init by @github-actions[bot] :: PR: #8952
  - Add config option for FP32 embedding grads by @github-actions[bot] :: PR: #8953
  - Add Python AIStore SDK to container and bump min Lhotse version by @pzelasko :: PR: #9537
  - [NeMo-UX] Fix tokenizer IO by @marcromeyn :: PR: #9555
  - [NeMo UX] Move mistral_7b.py to mistral.py by @akoumpa :: PR: #9545
  - ci: Do not attempt to send slack on fork by @ko3n1g :: PR: #9556
  - Fix SDXL incorrect name in Docs by @suiyoubi :: PR: #9534
  - Bump PTL version by @athitten :: PR: #9557
  - [Resiliency] Straggler detection by @jbieniusiewi :: PR: #9473
  - [NeMo-UX] Switch to torch_dist as default distributed checkpointing backend by @ashors1 :: PR: #9541
  - [NeMo-UX] Checkpointing bug fixes by @ashors1 :: PR: #9562
  - Expose MCore path_to_cache option by @maanug-nv :: PR: #9570
  - [NeMo-UX] Fix Trainer serialization by @marcromeyn :: PR: #9571
  - Update click version requirement by @thomasdhc :: PR: #9580
  - [Fault tolerance] Heartbeat detection by @maanug-nv :: PR: #9352
  - [Nemo-UX] Add fabric-API for manual forward-pass by @marcromeyn :: PR: #9577
  - [Nemo-UX] Add SDK-factories to llm-collection by @marcromeyn :: PR: #9589
  - [NeMo-UX] Some improvements to NeMoLogger by @marcromeyn :: PR: #9591
  - Set no_sync_func & grad_sync_fucn by @akoumpa :: PR: #9601
  - [NeMo-UX] Fix nemo logger when trainer has no loggers by @ashors1 :: PR: #9607
  - Fix the dictionary  format returned by the `scheduler` method by @sararb :: PR: #9609
  - [NeMo-UX] Dataloading enhancements and bug fixes by @ashors1 :: PR: #9595
  - Fix serialization of AutoResume by @sararb :: PR: #9616
  - Jsonl support by @adityavavre :: PR: #9611
  - Akoumparouli/mistral import instruct chat template fix by @akoumpa :: PR: #9567
  - Remove .cuda calls, use device isntead by @akoumpa :: PR: #9602
  - fix converter defautl args by @akoumpa :: PR: #9565
  - fix: remove non_blocking from PTL's .cuda call by @akoumpa :: PR: #9618
  - NeVA Minor Fixes by @yaoyu-33 :: PR: #9608
  - [NeMo-UX] fix pretrianing data sizes and weights by @cuichenx :: PR: #9627
  - [NeMo-UX] async checkpointing support by @ashors1 :: PR: #9466
  - Change default parallel_save to False by @mikolajblaz :: PR: #9632
  - Add REST API to deploy module by @athitten :: PR: #9539
  - ci: Timeout per step, not job by @ko3n1g :: PR: #9635
  - [NeMo-UX] Fix when optimizers are setup for PEFT by @marcromeyn :: PR: #9619
  - [NeMo-UX] Fix pipeline parallel bug by @ashors1 :: PR: #9637
  - Fixing import error fior llama-index (RAG pipeline) by @pablo-garay :: PR: #9662
  - llama CI fix by @rohitrango :: PR: #9663
  - [NeMo-UX] Make 'load_directly_on_device' configurable by @ashors1 :: PR: #9657
  - [Nemo-UX] Including all trainable-params in a PEFT-checkpoint by @marcromeyn :: PR: #9650
  - [NeMo-UX] Fix imports so local configuration of runs works again by @marcromeyn :: PR: #9690
  - Set TE flag in legacy -> mcore conversion script by @terrykong :: PR: #9722
  - Update starthere docs text by @erastorgueva-nv :: PR: #9724
  - TorchAudio installation workaround for incorrect `PYTORCH_VERSION` variable by @artbataev :: PR: #9736
  - [NeMo-UX] Match nemo 1's default behavior for drop_last and pad_samples_to_global_batch_size by @ashors1 :: PR: #9707
  - add a bit more for timeout (#9702) by @pablo-garay :: PR: #9754
  - Fix missing parallelisms by @maanug-nv :: PR: #9725
  - update branch by @nithinraok :: PR: #9764
  - Fix data preprocessing script by @cuichenx :: PR: #9759
  
  </details>

# Highlights
## Training

### Features and Model architectures

* PEFT: QLoRA support, LoRA/QLora for Mixture-of-Experts (MoE) dense layer
* State Space Models & Hybrid Architecture support (Mamba2 and NV-Mamba2-hybrid)
* Support Nemotron, Minitron, Gemma2, Qwen, RAG

### Multimodal

* NeVA: Add SOTA LLM backbone support (Mixtral/LLaMA3) and suite of model parallelism support (PP/EP)
* Support Language Instructed Temporal-Localization Assistant (LITA) on top of video NeVA

## ASR

### SpeechLM and SALM

### Adapters for Canary Customization

### Pytorch allocator in PyTorch 2.2 improves training speed up to 30% for all ASR models

### Cuda Graphs for Transducer Inference

### Replaced webdataset with Lhotse - gives up to 2x speedup

### Transcription Improvements - Speedup and QoL Changes

### ASR Prompt Formatter for multimodal Canary