Workload Inspector (WiT) performance anomaly detection Tool across GPUs 
State: closed | Pull request: yes
Labels: name: core

name: stale

name: NLP

name: Multi Modal

# What does this PR do ?

Workload analysis tool for anomaly detection and perf analysis. Tool uses Nsys trace captures. 
PR takes Nsys profiling changes as a reference.
 
Provides diagnostic information at per GPU level by analysis of Nsys traces.
  - Compute vs. exposed communication. 
  - Communication overlap at each collective and stream level. 
  - Slow kernels causing training bottlenecks. 

```
Kernel time per iteration: 401.182 (ms)
Kernel instances per iteration: 1006
 Kernel Statistic:
 Class            TimePerIter(ms)  InstancePerIter  TimePerIter(%)   TotalTime(ms)    TotalInstance  TotalTime(%)
 others                    25.68             5513            2.90         256.79            55254          2.90
 bf16_gemm                231.65             21236           53.99        2316.48           212400         53.99
 layernorm                  4.81             2808            0.80          48.12            28080          0.80
 reduce                     2.95             1112            0.62          29.53            11148          0.62
 userbuffer                 39.42            31332           23.96         394.18           313344         23.96
 elementwise_kernel         88.76            29088            6.66         887.55           291128          6.66
 attention                 7.83             1841             10.92          78.32            18432         10.92

NCCL time per iteration: 285.951 (ms)
NCCL overlapped comm time per iteration: 175.113 (ms)
NCCL Overlap Statistics:
Class                  TimePerIter(ms)    OverlapPerIter(ms) Overlap(ms)        OverlapPerIter(%)
AllGather                 109.85              71.50             714.98            65.09
ReduceScatter             134.08             103.61            1036.15            77.28

Per NCCL stream Overlap Statistics:
Class                               Stream ID        Grid Size        TimePerIter(ms)    OverlapPerIter(ms) Overlap(ms)        OverlapPerIter(%)    TotalTime(%)
ReduceScatter                       16               6                          20.37              0.00              0.00                0.00          3.53
ReduceScatter                       16               3                          10.48              0.00              0.00                0.00          0.02
AllGather                                10               4                          81.74             32.62            50.48            8.24           19.74
AllGather                                46               2                        35.45             26.84            67.37            14.97           25.95
AllGather                               22               1                           0.05              0.00              0.00                 0.00          0.00

GPU bubbles per iteration: 0.04%
Multiple NCCL streams overlap per iteration: 0.00%

```

# Changelog 

- Added a Workload Inspector Lightening callback [nemo/lightning/pytorch/callbacks/workload_inspector.py](https://github.com/NVIDIA/NeMo/commit/df4887aead1210b94c2edb254e4c05c996f6749e#diff-dd4d5723656416f43d36613d894dcb66a1547cd6de072137d39f118b038838b2)

- WorkloadInspectorPlugin added to enable the tool [nemo/lightning/run/plugins.py](https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-2b4c9a34429f8078889e60cccd2e2b6759100659c2afaa4ebbb493eb97704615)

- Added WiT callback to [nemo/lightning/pytorch/callbacks/__init__.py](https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-c9e02dfa089e3aacf2f24c451f7af596b1573f22d0e4f218511aca881f6dd5e4)

- WiT enabling changes in Interface for Pytorch-lightning based NeMo models [‎nemo/core/classes/modelPT.py](https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-4b936f7283de92c1563db15e6e966eea3994139ff3ff18823b498592e060a75c) 

- Invoke WiT plugin in [tests/lightning/test_nemo_run.py](https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-65cf660991d9d1586bea1caa565d4f8abdf348a4e59432e66d064d79858b38b9)

- WiT enabling steps in various Megatron based models:
[‎nemo/collections/multimodal/models/vision_language_foundation/clip/megatron_clip_models.py](https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-db5f2e6c1e3156d33d2e5afa097b7d425e20c07e021f79e43d9916b993027c48)

[‎nemo/collections/nlp/models/language_modeling/megatron_bert_model.py](https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-d496e911eb06fca3128061957d326a83ba0a603ca8482ad292aee3851f49bf61)

[nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py]
(https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-d84a79499285d34da7d19099472502cb4e7c1b1672832488268066c9c7df852d)

[nemo/collections/vision/models/megatron_vit_classification_models.py](https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-791dbc321fe2f4f37e07d17dd694d7a31b2b849510841958226c6d045b529035)

- Test cases for WiT 
[‎tests/lightning/pytorch/callbacks/test_workload_inspector.py](https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-4b1ecd4d173033a01acb4e7389e7a6f043bb95916f063720ce9dc9ec3fe34319)

[‎tests/collections/nlp/test_falcon_model.py](https://github.com/NVIDIA/NeMo/commit/76586ffd541c605facf7d146dcd58bad42fbf87a#diff-ca09720d0bccee0bd8699c8342d3ba78b427c2b70f8a9387dbf5e8b21d435901)

# Usage

Workload Inspector (WiT) uses NVIDIA Nsight profiling and CUDA Profiler Start/Stop APIs to detect anomalies in training applications. 

```
 # Workload Inspector profiling options in yaml file
  wit_profile:
    enabled: False
    start_step: 10  # Global batch to start profiling
    end_step: 10 # Global batch to end profiling
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

This PR was closed because it has been inactive for 7 days since being marked as stale.