Fix passing options to TransformerEngine layers and MCore
State: closed | Pull request: yes
Labels: name: stale

name: NLP

This affects the NeMo TransformerEngine layers (`AutocastTransformerLayer`) and MCore `TransformerConfig`, which previously did not correctly receive the settings for
- `activation: str`: custom activation, including the GLU property,
- `normalization: str`: custom normalization layers,
- `bias: bool`: optional bias,
- (MCore only) `bias_dropout_add_fusion: bool`: whether to fuse **bias add+dropout+residual add** via JIT-compilation (we could even enable this for TransformerEngine by setting `os.environ["NVTE_BIAS_DROPOUT_FUSION"] = "1"`. Note that Megatron-LM also does not do this), and
- (MCore only) `bias_activation_fusion: bool`: whether to fuse **bias add+activation** via JIT-compilation.

# What does this PR do ?

Fix passing normalization and bias options to TransformerEngine layers. Previously, with `mcore=False, transformer_engine=True`, the mentioned model settings would not be reproduced correctly.

**Collection**: NLP

# Changelog 
- We refactor the functions to convert the NeMo normalization and activation config option to its TransformerEngine/Megatron-LM core equivalent, saving on duplicate code and improving maintainability.
- We refactor the query for whether an activation function uses a GLU layer.
- These refactored functions are now used for creating the TransformerEngine and MCore configs.
- We pass two additional values to the `AutocastTransformerLayer` and its superclass constructor, namely `activation`, `normalization`, and `bias`. 
- We pass five additional values to the MCore `TransformerConfig`, namely `activation_func`, `gated_linear_unit`, `add_bias_linear`, `bias_activation_fusion`, and `bias_dropout_fusion`.

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [X] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests? – There were no existing tests for `AutocastTransformerLayer`, so I did not bother.
- [ ] Did you add or update any necessary documentation? – I do not think this is necessary, we just fix the incorrect behavior.
- [X] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc) – Yes, it implicitly affects behavior in TransformerEngine and Megatron-LM.
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [X] Bugfix
- [ ] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

PTAL.

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

Bla

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.