Extend multimodal/speech_llm with lhotse, t5 and bestow supports
State: closed | Pull request: yes
Labels: name: NLP

name: Multi Modal

name: Run CICD

# What does this PR do ?

In multimodal/speech_llm, add lhotse dataloader support and two models, SALM-T5 and Bestow-GPT. Include example configs.


###  Main features under speech_llm
- Lhotse dataloader support for speech SFT in speech_llm
- SALM-style architecture with T5 LLM backbone
- Bestow-style architecture (cross-attention based) with GPT LLM backbone
### Minor edit in nlp collection: 
- megatron_base_model.py: handle the case tokenizer.type is not set
- megatron_lm_encoder_decoder_model.py: hanlde the case encoder_input is used 
- megatron_base_prompt_learning_model.py: group the llm init code under init_model function (follow the pattern from megatron_gpt_prompt_learning_model.py) so that it can be overwritten by subclass when needed
- megatron/utils.py: in gradient accumulation, handle the case where the batch size from dynamic bucketing is not divisible. This happens when using lhotse dataloader with batch_duration

### Lhotse dataloader usage
This is done by either reading the specified "context_key" field from the manifest or use the "default_context_key" field to extract from input_cfg in lhotse. An example input_cfg is as below so as to reuse existing canary manifests when the user wants
```
- input_cfg:
  - manifest_filepath: manifest1.json
    type: nemo
    weight: 2.0
    tags:
      default_context: "pls transcribe the audio"
  - manifest_filepath: manifest2.json
    type: nemo
    weight: 1.0
    tags:
      default_context: "pls translate English audio to German"
  type: group
  weight: 0.4
```

**Collection**: [common,nlp,multimodal]


**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation


> Thanks for the great work, just a few comments. Also please remember to address the CodeQL issues. My main concerns are:
> 
> 1. We need to remove random context and canary-specific dataloading, since they were only used in our internal experiments but not useful for external users. We added them for fast prototyping but they are not good designs.
> 2. We should avoid using `salm` and `bestow` in the dataloader function names, same reason why we don't name the model class as "SALM" or "BESTOW"
> 3. Please ask someone from NLP to see if they're okay with removing the assertions
> 4. Like @titu1994 mentioned in the previous [PR](https://github.com/NVIDIA/NeMo/pull/8741), we want to have flexible fields to use as "context", but it seems that the dataloader is only using `question` as they key for input text. It's better to keep it aligned with the non-lhotse dataset config, since it will be strange to suddenly find that the `input_key` in dataset config doesn't work as expected.
> 5. And some small comments.

1. remove canary-specific parts and use lhotse dataloader support in getting "context" field. This is done by either reading the specified "context_key" field from the manifest or use the "default_context_key" field to extract from input_cfg in lhotse. An example input_cfg is as below so as to reuse existing canary manifests when the user wants
```
- input_cfg:
  - manifest_filepath: manifest1.json
    type: nemo
    weight: 2.0
    tags:
      default_context: "pls transcribe the audio"
  - manifest_filepath: manifest2.json
    type: nemo
    weight: 1.0
    tags:
      default_context: "pls translate English audio to German"
  type: group
  weight: 0.4
``` 

Remove random context.

2. Done
3. Will do after getting approval from speech_llm side
4. Done
5. Fixed