Adds a Knob for OnlineSampling by introducing 'global_sample_mapping' in the SFT config.yaml
State: closed | Pull request: yes
Labels: name: NLP

name: TN/ITN

name: Run CICD


# What does this PR do ?

Adds a Knob for OnlineSampling by introducing a new parameter 'global_sample_mapping' in the SFT config YAML(default value is false).
The feature was proposed in this [Nvbug](https://nvbugspro.nvidia.com/bug/4682975?commentNumber=4). This will allow users to choose the data sample method that best suits their needs.

**Collection**:  NeMo's NLP collection

# Changelog 
- SFT config.yaml
- Passing parameters
- Control whether to use OnlineSampleMapping

# Usage
change 'global_sample_mapping' in megatron_gpt_finetuning_config.yaml
False (default): Use OnlineSampleMapping. Shuffle the dataset within each epoch. 
True: Shuffle the replicated data all together.

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?
@ericharper
Anyone in the community is free to review the PR once the checks have passed. 

# Additional Information
* Related to [nvbug](https://nvbugspro.nvidia.com/bug/4682975?commentNumber=4)
The second suggested Improvement has not been fixed yet

<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```
************* Module nemo.collections.llm.gpt.data.core
nemo/collections/llm/gpt/data/core.py:1:0: C0114: Missing module docstring (missing-module-docstring)
nemo/collections/llm/gpt/data/core.py:11:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/llm/gpt/data/core.py:18:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/llm/gpt/data/core.py:18:0: R0913: Too many arguments (17/5) (too-many-arguments)
nemo/collections/llm/gpt/data/core.py:18:0: R0917: Too many positional arguments (17/5) (too-many-positional-arguments)
nemo/collections/llm/gpt/data/core.py:18:0: R0914: Too many local variables (19/15) (too-many-locals)
nemo/collections/llm/gpt/data/core.py:38:4: C0415: Import outside toplevel (nemo.collections.nlp.data.language_modeling.megatron.gpt_sft_dataset.GPTSFTDataset) (import-outside-toplevel)
************* Module nemo.collections.nlp.data.language_modeling.megatron.gpt_sft_dataset
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:29:0: C0301: Line too long (115/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:53:0: C0301: Line too long (117/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:62:0: C0301: Line too long (111/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:69:0: C0301: Line too long (353/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:70:0: C0301: Line too long (114/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:71:0: C0301: Line too long (173/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:72:0: C0301: Line too long (156/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:75:0: C0301: Line too long (111/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:78:0: C0301: Line too long (157/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:81:0: C0301: Line too long (147/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:82:0: C0301: Line too long (178/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:83:0: C0301: Line too long (138/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:84:0: C0301: Line too long (121/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:85:0: C0301: Line too long (116/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:86:0: C0301: Line too long (144/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:87:0: C0301: Line too long (109/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:89:0: C0301: Line too long (247/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:91:0: C0301: Line too long (116/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:162:0: C0301: Line too long (125/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:171:0: C0301: Line too long (121/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:183:0: C0301: Line too long (109/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:237:0: C0301: Line too long (102/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:240:0: C0301: Line too long (137/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:243:0: C0301: Line too long (133/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:256:0: C0301: Line too long (110/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:267:0: C0301: Line too long (105/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:268:0: C0301: Line too long (146/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:270:0: C0301: Line too long (108/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:273:0: C0301: Line too long (153/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:274:0: C0301: Line too long (155/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:277:0: C0301: Line too long (108/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:296:0: C0301: Line too long (127/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:323:0: C0301: Line too long (114/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:343:0: C0301: Line too long (108/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:385:0: C0301: Line too long (120/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:395:0: C0301: Line too long (107/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:413:0: C0301: Line too long (101/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:483:0: C0301: Line too long (117/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:488:0: C0301: Line too long (113/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:499:0: C0301: Line too long (114/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:501:0: C0301: Line too long (118/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:502:0: C0301: Line too long (116/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:523:0: C0301: Line too long (106/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:528:0: C0301: Line too long (103/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:549:0: C0301: Line too long (108/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:556:0: C0301: Line too long (117/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:575:0: C0301: Line too long (107/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:611:0: C0301: Line too long (113/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:631:0: C0301: Line too long (102/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:645:0: C0301: Line too long (114/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:647:0: C0301: Line too long (109/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:658:0: C0301: Line too long (107/100) (line-too-long)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:1:0: C0114: Missing module docstring (missing-module-docstring)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:19:0: E0401: Unable to import 'datasets' (import-error)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:20:0: E0401: Unable to import 'numpy' (import-error)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:21:0: E0401: Unable to import 'torch' (import-error)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:25:0: E0401: Unable to import 'datasets' (import-error)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:25:0: C0413: Import "from datasets import load_dataset" should be placed at the top of the module (wrong-import-position)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:27:0: C0413: Import "from nemo.collections.common.tokenizers.tokenizer_spec import TokenizerSpec" should be placed at the top of the module (wrong-import-position)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:28:0: C0413: Import "from nemo.collections.nlp.data.language_modeling.megatron.dataset_utils import get_samples_mapping" should be placed at the top of the module (wrong-import-position)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:29:0: C0413: Import "from nemo.collections.nlp.data.language_modeling.text_memmap_dataset import JSONLMemMapDataset, OnlineSampleMapping" should be placed at the top of the module (wrong-import-position)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:30:0: C0413: Import "from nemo.core.classes import Dataset" should be placed at the top of the module (wrong-import-position)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:31:0: C0413: Import "from nemo.utils import logging" should be placed at the top of the module (wrong-import-position)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:36:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:36:0: R0902: Too many instance attributes (31/7) (too-many-instance-attributes)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:37:4: R0913: Too many arguments (29/5) (too-many-arguments)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:37:4: R0917: Too many positional arguments (29/5) (too-many-positional-arguments)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:37:4: R0914: Too many local variables (29/15) (too-many-locals)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:204:8: R1705: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it (no-else-return)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:261:18: R1721: Unnecessary use of a comprehension, use dict(zip(placeholders, prompt_template_values)) instead. (unnecessary-comprehension)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:263:18: R1721: Unnecessary use of a comprehension, use dict(zip(placeholders, self.prompt_template_keys)) instead. (unnecessary-comprehension)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:350:20: R1723: Unnecessary "else" after "break", remove the "else" and de-indent the code inside it (no-else-break)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:364:8: R1705: Unnecessary "elif" after "return", remove the leading "el" from "elif" (no-else-return)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:437:8: R1705: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it (no-else-return)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:483:21: W3301: Do not use nested call of 'max'; it's possible to do 'max(*[len(x) for x in input_ids])' instead (nested-min-max)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:483:25: R1728: Consider using a generator instead 'max(len(x) for x in input_ids)' (consider-using-generator)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:483:59: R1728: Consider using a generator instead 'max(len(x) for x in contexts)' (consider-using-generator)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:522:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:546:15: W0718: Catching too general exception Exception (broad-exception-caught)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:551:12: R1722: Consider using 'sys.exit' instead (consider-using-sys-exit)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:561:12: W0106: Expression "[np.random.shuffle(x) for x in indices]" is assigned to nothing (expression-not-assigned)
nemo/collections/nlp/data/language_modeling/megatron/gpt_sft_dataset.py:580:4: R0914: Too many local variables (16/15) (too-many-locals)
************* Module nemo.collections.nlp.models.language_modeling.megatron_gpt_sft_model
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:29:0: C0301: Line too long (103/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:30:0: C0301: Line too long (115/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:36:0: C0301: Line too long (104/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:44:0: C0301: Line too long (109/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:67:0: C0301: Line too long (107/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:70:0: C0301: Line too long (105/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:74:0: C0301: Line too long (110/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:102:0: C0301: Line too long (125/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:107:0: C0301: Line too long (135/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:115:0: C0301: Line too long (106/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:116:0: C0301: Line too long (124/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:122:0: C0301: Line too long (111/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:123:0: C0301: Line too long (140/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:127:0: C0301: Line too long (159/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:136:0: C0301: Line too long (104/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:143:0: C0301: Line too long (111/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:154:0: C0301: Line too long (107/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:159:0: C0301: Line too long (145/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:172:0: C0301: Line too long (105/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:179:0: C0301: Line too long (110/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:196:0: C0301: Line too long (116/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:206:0: C0301: Line too long (122/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:215:0: C0301: Line too long (134/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:226:0: C0301: Line too long (106/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:229:0: C0301: Line too long (118/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:240:0: C0301: Line too long (152/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:248:0: C0301: Line too long (113/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:258:0: C0301: Line too long (109/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:259:0: C0301: Line too long (108/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:264:0: C0301: Line too long (114/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:286:0: C0301: Line too long (101/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:291:0: C0301: Line too long (125/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:297:0: C0301: Line too long (146/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:303:0: C0301: Line too long (245/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:312:0: C0301: Line too long (117/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:319:0: C0301: Line too long (121/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:333:0: C0301: Line too long (108/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:334:0: C0301: Line too long (132/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:347:0: C0301: Line too long (101/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:373:0: C0301: Line too long (111/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:394:0: C0301: Line too long (108/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:439:0: C0301: Line too long (145/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:442:0: C0301: Line too long (145/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:445:0: C0301: Line too long (102/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:453:0: C0301: Line too long (128/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:487:0: C0301: Line too long (108/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:488:0: C0301: Line too long (142/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:493:0: C0301: Line too long (110/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:519:0: C0301: Line too long (119/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:523:0: C0301: Line too long (108/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:526:0: C0301: Line too long (117/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:553:0: C0301: Line too long (112/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:557:0: C0301: Line too long (108/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:559:0: C0301: Line too long (131/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:569:0: C0301: Line too long (104/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:581:0: C0301: Line too long (130/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:589:0: C0301: Line too long (101/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:601:0: C0301: Line too long (124/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:605:0: C0301: Line too long (108/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:612:0: C0301: Line too long (108/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:615:0: C0301: Line too long (106/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:630:0: C0301: Line too long (109/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:643:0: C0301: Line too long (114/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:676:0: C0301: Line too long (104/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:683:0: C0301: Line too long (104/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:694:0: C0301: Line too long (117/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:696:0: C0301: Line too long (111/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:705:0: C0301: Line too long (101/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:751:0: C0301: Line too long (112/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:777:0: C0301: Line too long (112/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:793:0: C0301: Line too long (107/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:795:0: C0301: Line too long (102/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:803:0: C0301: Line too long (102/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:898:0: C0301: Line too long (109/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:903:0: C0301: Line too long (142/100) (line-too-long)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:419:31: W0511: TODO: need a nicer way to do this via inheritance (@adithyare) (fixme)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:569:9: W0511: TODO: this method should be modularized. It is too long and does too many things. (@adithyare) (fixme)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:1:0: C0114: Missing module docstring (missing-module-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:19:0: E0401: Unable to import 'torch' (import-error)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:20:0: E0401: Unable to import 'omegaconf' (import-error)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:21:0: E0401: Unable to import 'pytorch_lightning.loops.fetchers' (import-error)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:22:0: E0401: Unable to import 'pytorch_lightning.trainer.trainer' (import-error)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:57:0: R0901: Too many ancestors (13/7) (too-many-ancestors)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:57:0: R0902: Too many instance attributes (22/7) (too-many-instance-attributes)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:91:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:91:4: R0912: Too many branches (16/12) (too-many-branches)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:153:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:153:4: R1711: Useless return at end of function or method (useless-return)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:190:4: R0914: Too many local variables (18/15) (too-many-locals)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:196:29: W1309: Using an f-string that does not have any interpolated variables (f-string-without-interpolation)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:214:24: W1309: Using an f-string that does not have any interpolated variables (f-string-without-interpolation)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:230:44: R1728: Consider using a generator instead 'sum(x[0] for x in num_train_samples_per_dataset)' (consider-using-generator)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:308:8: R1705: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it (no-else-return)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:190:4: R0912: Too many branches (16/12) (too-many-branches)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:322:8: R1705: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it (no-else-return)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:332:4: R0914: Too many local variables (25/15) (too-many-locals)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:419:8: R1705: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it (no-else-return)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:332:4: R0912: Too many branches (16/12) (too-many-branches)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:424:4: W0221: Number of parameters was 3 in 'MegatronGPTModel.validation_step' and is now 2 in overriding 'MegatronGPTSFTModel.validation_step' method (arguments-differ)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:430:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:438:15: C0123: Use isinstance() rather than type() for a typecheck. (unidiomatic-typecheck)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:445:15: C0123: Use isinstance() rather than type() for a typecheck. (unidiomatic-typecheck)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:451:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:487:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:487:4: R0913: Too many arguments (6/5) (too-many-arguments)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:487:4: R0917: Too many positional arguments (6/5) (too-many-positional-arguments)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:487:4: R0914: Too many local variables (24/15) (too-many-locals)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:509:33: W0622: Redefining built-in 'input' (redefined-builtin)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:559:20: W1309: Using an f-string that does not have any interpolated variables (f-string-without-interpolation)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:487:4: R0912: Too many branches (15/12) (too-many-branches)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:568:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:568:4: R0912: Too many branches (14/12) (too-many-branches)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:568:4: R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:690:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:692:13: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:705:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:705:4: R0913: Too many arguments (6/5) (too-many-arguments)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:705:4: R0917: Too many positional arguments (6/5) (too-many-positional-arguments)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:722:16: W0707: Consider explicitly re-raising using 'except ValueError as exc' and 'raise ValueError(f'Could not convert {label} to float.') from exc' (raise-missing-from)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:738:16: W0707: Consider explicitly re-raising using 'except ValueError as exc' and 'raise ValueError(f'Could not convert {label} to int.') from exc' (raise-missing-from)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:792:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:792:4: R1711: Useless return at end of function or method (useless-return)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:800:4: W0221: Number of parameters was 1 in 'MegatronGPTModel.build_train_valid_test_datasets' and is now 2 in overriding 'MegatronGPTSFTModel.build_train_valid_test_datasets' method (arguments-differ)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:811:11: R1714: Consider merging these comparisons with 'in' by using 'stage in ('validate', 'test')'. Use a set instead if elements are hashable. (consider-using-in)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:842:31: R1719: The if expression can be replaced with 'test' (simplifiable-if-expression)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:845:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:854:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:865:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:877:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:889:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:902:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:167:8: W0201: Attribute 'init_consumed_samples' defined outside __init__ (attribute-defined-outside-init)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:796:12: W0201: Attribute '_test_ds' defined outside __init__ (attribute-defined-outside-init)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:804:12: W0201: Attribute '_validation_ds' defined outside __init__ (attribute-defined-outside-init)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:814:8: W0201: Attribute '_train_ds' defined outside __init__ (attribute-defined-outside-init)
nemo/collections/nlp/models/language_modeling/megatron_gpt_sft_model.py:57:0: R0904: Too many public methods (24/20) (too-many-public-methods)
************* Module scripts.nlp_language_modeling.prepare_packed_ft_dataset
scripts/nlp_language_modeling/prepare_packed_ft_dataset.py:25:0: C0301: Line too long (105/100) (line-too-long)
scripts/nlp_language_modeling/prepare_packed_ft_dataset.py:30:3: C0303: Trailing whitespace (trailing-whitespace)
scripts/nlp_language_modeling/prepare_packed_ft_dataset.py:73:0: C0301: Line too long (117/100) (line-too-long)
scripts/nlp_language_modeling/prepare_packed_ft_dataset.py:83:0: C0301: Line too long (109/100) (line-too-long)
scripts/nlp_language_modeling/prepare_packed_ft_dataset.py:135:0: C0301: Line too long (112/100) (line-too-long)
scripts/nlp_language_modeling/prepare_packed_ft_dataset.py:159:0: C0301: Line too long (118/100) (line-too-long)
scripts/nlp_language_modeling/prepare_packed_ft_dataset.py:162:0: C0301: Line too long (108/100) (line-too-long)
scripts/nlp_language_modeling/prepare_packed_ft_dataset.py:163:0: C0301: Line too long (120/100) (line-too-long)
scripts/nlp_language_modeling/prepare_packed_ft_dataset.py:165:0: C0301: Line too long (105/100) (line-too-long)
scripts/nlp_language_modeling/prepare_packed_ft_dataset.py:1:0: C0114: Missing module docstring (missing-module-docstring)
scripts/nlp_language_modeling/prepare_packed_ft_dataset.py:19:0: E0401: Unable to import 'numpy' (import-error)
scripts/nlp_language_modeling/prepare_packed_ft_dataset.py:30:0: W0105: String statement has no effect (pointless-string-statement)
scripts/nlp_language_modeling/prepare_packed_ft_dataset.py:118:0: C0115: Missing class docstring (missing-class-docstring)
scripts/nlp_language_modeling/prepare_packed_ft_dataset.py:124:4: C0116: Missing function or method docstring (missing-function-docstring)
scripts/nlp_language_modeling/prepare_packed_ft_dataset.py:137:0: C0116: Missing function or method docstring (missing-function-docstring)
scripts/nlp_language_modeling/prepare_packed_ft_dataset.py:175:4: E1120: No value for argument 'cfg' in function call (no-value-for-parameter)

-----------------------------------
Your code has been rated at 6.59/10
```

---

Mitigation guide:

* Add sensible and useful docstrings to functions and methods
* For trivial methods like getter/setters, consider adding `# pylint: disable=C0116` inside the function itself
* To disable multiple functions/methods at once, put a `# pylint: disable=C0116` before the first and a `# pylint: enable=C0116` after the last.

By applying these rules, we reduce the occurance of this message in future.

Thank you for improving NeMo's documentation!