renwei2024/nemo-github-issues huggingface dataset:
Update tlt integration branch
State: closed | Pull request: yes

merge main into tlt-integration

This pull request **introduces 1 alert** when merging 356ec3a4b14e2c9877fc393d78b1887de38e7056 into 730da9886249c8f4acc766b0bde516a14681ac00 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-2e81cf2c4aafdecb672b604ea76fcae4bf8848ba)

**new alerts:**

* 1 for Variable defined multiple times
Problem faced during Training on Different Language.
State: closed | Pull request: no
Labels: name: bug

I am using pre-trained quartznet 15x5, for transfer learning for the Hindi language, with a different set of vocab (Devanagari characters.) 

While training I am facing mainly 2 issues:
1.  The reference string that appears during training is blank.
2. While saving the model using `quartznet.save_to('path/to/save')` , I am getting `'ascii' codec can't encode character '\u091b' in position 4407: ordinal not in range(128)`

Please provide a solution to overcome the abovementioned issues.


**Environment overview**

 - Environment location: PyTorch GPU docker
 - Method of NeMo install: Installed from source

**Additional context**
GPU model - 4 x RTX 2080Ti , 12 GB vRAM.

I am facing similar issues using unicode, can you share some example lines from your manifest? One possibility is that you need to use `ensure_ascii=False` as an option to json.dump when generating the manifests: `json.dump(metadata, f, ensure_ascii=False)`

@gopesh97 have you got any success on Hindi. I'm working on urdu , just curious to know whether this Nemo model works for right to left language?

Ensure that you have the correct language settings in your environment (native or in container). Most ascii encoding issues can be solved by setting the `LANG` and `LC_ALL` env variables to a UTF-8 default
[Question] How can I get character timestamps / word timestamps in Nemo ASR
State: closed | Pull request: no

How can I get relative character level timestamps or word level timestamps in nemo asr. 

Thanks,
S


I think you need to wait for this PR to merge first https://github.com/NVIDIA/NeMo/pull/1319

Based on what I saw in this notebook https://github.com/NVIDIA/NeMo/blob/73875d874b006b672a8e407687ef414ef7a7b282/tutorials/asr/Offline_ASR.ipynb
QA squad update tutorial and example script with usage examples
State: closed | Pull request: yes

QA squad update tutorial and example script with usage examples after model changes

This pull request **introduces 3 alerts** when merging 916b8c79da313e93bf92ab7fed0d2f0e5436b2d9 into 730da9886249c8f4acc766b0bde516a14681ac00 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-53d748f2d12e33432e771ec04560e9d2cba50fb0)

**new alerts:**

* 2 for Unused import
* 1 for Unused local variable
Update exp_manager for Lightning 1.0
State: closed | Pull request: yes


@okuchaiev that won't work sadly with multiple data loaders. Each will be tagged with a specific name prepended so exp manager cannot infer it. At least not unless we also attach a "default" validation wer to our logs 

We actually do attach a default "val_loss", with the default as the first dataloader passed to the validation filepath (can be changed with a flag). So we can add some logic to our wer logging to add a default wer to the logs too. 

So can this: "modify ASR configs (for speech recognition models only) to monitor: "validation_wer" instead of "val_loss" by default" be done for a case of single data val loader at least (which is what our configs are)?
Megatron-LM checkpoints don't work properly
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Megatron-LM checkpoints appear to not work properly with the recent changes in NLP module.

**Steps/Code to reproduce bug**

1. implement minimal change to the [Token Classification yaml file](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/conf/token_classification_config.yaml) to use Meagtron-LM checkpoint.
  1.1. Change [`bert-base-uncased`](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/conf/token_classification_config.yaml#L77) to `megatron-bert-345m-cased`
  1.2. Download and unpack the `gmb_v-2.2.0_clean` dataset for NER as instructed in the [tutorial](https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Token_Classification_Named_Entity_Recognition.ipynb), specify the [location of the dataset in the yaml file](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/conf/token_classification_config.yaml#L44).
2. Run token classification example `python token_classification.py`


**Expected behavior**

Instead of training, error occurs and the program exits:
`TypeError: bert_extended_attention_mask() missing 1 required positional argument: 'dtype`

**Environment overview (please complete the following information)**

 - Environment location: Based on [NeMo NGC container](nvcr.io/nvidia/nemo:v1.0.0b1), upgrade NeMo with pip install: `python -m pip install --upgrade git+https://github.com/NVIDIA/NeMo.git@main`
 Because without the additional pip install, different error occurs (another bug - mismatch between NGC container and pip package).
 - Method of NeMo install: pip install (`python -m pip install --upgrade git+https://github.com/NVIDIA/NeMo.git@main`)
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used: `docker pull nvcr.io/nvidia/nemo:v1.0.0b1`

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: `nvcr.io/nvidia/nemo:v1.0.0b1` docker container
- PyTorch version: `nvcr.io/nvidia/nemo:v1.0.0b1` docker container
- Python version: `nvcr.io/nvidia/nemo:v1.0.0b1` docker container



I don't think this command
```
python -m pip install --upgrade git+https://github.com/NVIDIA/NeMo.git@main
``` 
is installing NeMo properly. The error you are getting is from using an old version of NeMo with a new version of Megatron-LM. When we upgrade Megatron-LM, we fixed that error. This has all happened post 1.0.0b1 release.

Try:
```
git clone https://github.com/NVIDIA/NeMo.git
cd NeMo
sh reinstall.sh
```

Please let us know if the above suggestion was helpful, thanks
CTC decoders
State: closed | Pull request: yes

- Added CTC beam search decoder (module, package installation)
- Added Offline ASR notebook for greedy decoding, words splitting, decoding with N-gram re-scoring

Reviewers, please replace `main` branch with `ctc-decoders` in the notebook for testing in Colab.

This pull request **introduces 2 alerts** when merging fd2333bc64a3bb281a02e93fc5bd4d0319972e8a into 87f3ef66fdf01eb61ea603024f5d9140fa9055de - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-906e550c4d046ad76f8fb55bfbd498604ef6350d)

**new alerts:**

* 2 for Unused import

@vsl9 Quick question guys - does this PR mean that beam-searching using a Language Model (kenLM) is now supported at least partially? Just trying to keep track of the state of things :) We're all eagerly keeping track of the progress you guys are making.

And apologies if this isn't the appropriate place for such questions (let me know) but it seems like the delay for the timestamps builds up more after each word, ending up with the timestamps not aligning with the right words at all. This could be because I'm running it on much longer (20-30minute) audio files instead of ~25s, but I figured that wouldn't be an issue since there's no RNN.
[Question] ASR_with_NeMo: Missing keys in state_dict
State: closed | Pull request: no

**Describe your question**
I am trying to run the first tutorial for NeMo, from https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/asr/01_ASR_with_NeMo.ipynb

The intent is to use the transfer learning portion of this tutorial and see how it works for Urdu. I want to use the "Out-of-the-box model" and then fine tune in Urdu (local language). 

I am getting an error on the line:
`quartznet = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En")`

```
RuntimeError: Error(s) in loading state_dict for EncDecCTCModel:
	Missing key(s) in state_dict: "_wer.scores", "_wer.words".
```

**Environment overview (please complete the following information)**

 - Environment location: Colab
 - Method of NeMo install: !python -m pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[all]
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Colab
- PyTorch version: Default version on the notebook
- Python version: Default version on the notebook

**Additional context**
Nothing to add here. It is a basic tutorial that isn't working, maybe the tutorial is using the wrong NeMo branch but I would not know enough to troubleshoot. 

Can you try
```
# This line will download pre-trained QuartzNet15x5 model from NVIDIA's NGC cloud and instantiate it for you
quartznet = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En", strict=False)
```
It's related to this issue: https://github.com/NVIDIA/NeMo/issues/1297


This should be fixed now with: https://github.com/NVIDIA/NeMo/pull/1322
We've pinned Colab Notebooks to the latest stable release of NeMo.

Closing since https://github.com/NVIDIA/NeMo/pull/1322 is merged, please re-open if issue persists.
Punctuation model with TLT
State: closed | Pull request: yes


This pull request **introduces 1 alert** when merging c31c5e1e995742c1b70b20089e0b9085fa8b6d6b into fcb5fcec71d7b633fde6cfaaba88857ae79b656e - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-8d5493e35b1d8b30a706ef41354a307fc8c234b9)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 7c58a10d273abc2ac243af90b354bbe87fb6d5a4 into 53b31f7b902f96e3b964214373c328b3fa2d3606 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-2f284336427ca0d23461bf3cb5f36e05a6a3e6e2)

**new alerts:**

* 1 for Unused import
cant load pretrained model
State: closed | Pull request: no

when i try to upload a  pretrained model
quartznet = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En") 
it is giving error 
AttributeError: module 'nemo.collections.asr' has no attribute 'models'
[Question]

Could you try to load the state_dict using nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En", strict=False)? 
The current work-around is to pass strict=False to either #1297

Can you try `from nemo.collections.asr.models import EncDecCTCModel`?

please try suggestion from @blisc and re-open if you still see the issue

There is still this error:
 
AttributeError: type object 'EncDecCTCModel' has no attribute 'restore_from' or 'from_pretrained' 

has anyone try out the latest release of nemo 1.3.0?



> There is still this error:
> 
> AttributeError: type object 'EncDecCTCModel' has no attribute 'restore_from' or 'from_pretrained'
> 
> has anyone try out the latest release of nemo 1.3.0?

i tried , exactly the same issus here  "type object 'EncDecCTCModel' has no attribute 'restore_from'"
even from_pretrained() can not use too , please check and fix it

Please include a colab notebook in which this issue is reproduced and open another issue.
when i try to load a pretrained model like thisquartznet = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En") [Question]
State: closed | Pull request: no

**Describe your question**

A clear and concise description of your question.
Describe what you want to achieve. And/or what NeMo APIs are unclear/confusing.


**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**

Add any other context about the problem here.
Example: GPU model
Disable T5 Test
State: closed | Pull request: yes


Or, a better solution would be to disable T5 as a choice from https://github.com/NVIDIA/NeMo/blob/693c556f7bcffaf78e698f69b36331d872abf533/nemo/collections/nlp/modules/common/lm_utils.py#L44

And in this CI test config use "bert-base-uncased"? @ekmb  what do you think?

> Why would we disable the test? Let’s just put an upper bound on HF transformers lib version and add properly fixing this model to our to-do list

T5 is experimental and no support is guaranteed.
It doesn't look like anyone is planning on fixing this model/test.
Pinning python library versions is not a fix.

I am ok with pinning libraries if someone plans on fixing the test, but it looks like no one is available. Since it won't be fixed, the easiest path to unblock existing PRs is just to remove the test.

> Or, a better solution would be to disable T5 as a choice from
> 
> https://github.com/NVIDIA/NeMo/blob/693c556f7bcffaf78e698f69b36331d872abf533/nemo/collections/nlp/modules/common/lm_utils.py#L44
> 
> And in this CI test config use "bert-base-uncased"? @ekmb what do you think?

Does this test improve our CI coverage? Our CI is reaching >20 mins now which is frustratingly slow.

> > Why would we disable the test? Let’s just put an upper bound on HF transformers lib version and add properly fixing this model to our to-do list
> 
> T5 is experimental and no support is guaranteed.
> It doesn't look like anyone is planning on fixing this model/test.
> Pinning python library versions is not a fix.
> 
> I am ok with pinning libraries if someone plans on fixing the test, but it looks like no one is available. Since it won't be fixed, the easiest path to unblock existing PRs is just to remove the test.

Yeah, let's remove T5 as a choice from ```get_lm_model``` function (e.g. raise exception if T5 is requested) and not use that model (and use bert-base) for this CI test. Then we won't need to pin library and won't have broken example.

> > > Why would we disable the test? Let’s just put an upper bound on HF transformers lib version and add properly fixing this model to our to-do list
> > 
> > 
> > T5 is experimental and no support is guaranteed.
> > It doesn't look like anyone is planning on fixing this model/test.
> > Pinning python library versions is not a fix.
> > I am ok with pinning libraries if someone plans on fixing the test, but it looks like no one is available. Since it won't be fixed, the easiest path to unblock existing PRs is just to remove the test.
> 
> Yeah, let's remove T5 as a choice from `get_lm_model` function (e.g. raise exception if T5 is requested) and not use that model (and use bert-base) for this CI test. Then we won't need to pin library and won't have broken example.

Let's remove this test, bert-base is tested in other places. I'll add refactoring of 'get_lm_model' on my to-do.
TLT integration
State: closed | Pull request: yes

The branch contains up to date changes required for models ported to TLT to work.

- changes to Token Classification Model (NLP)
- changes related to Question Answering (NLP)
- changes related to Punctuation Model (NLP)

This pull request **introduces 3 alerts** when merging f88d17c8c9e18e5ab54c92daa1ad4507bc2b9db6 into cd55594e0f60a4bcbaf6d0e4196c1404591e0076 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-cd7113f9f6c4a6d00463f645cd84fd574adb297f)

**new alerts:**

* 2 for Unused import
* 1 for Unused local variable

This pull request **introduces 3 alerts** when merging 82039f3fb78ac30be9199056082e0c7bfe5d5704 into 16a94758d2dafe7af8aa5bef504cf0584ed7d905 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-78e1e8cd47bfa077824ae972cd863a6354f37cb1)

**new alerts:**

* 2 for Unused import
* 1 for Unused local variable

This pull request **introduces 3 alerts** when merging 0d97efdfe7acdf66b07b22b9ef6998ccb8288d33 into 89579d87649c894fd5c74892308d4ac0b07dec4d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-38f6f286471ed1a78730362ef5ae968f61fbe124)

**new alerts:**

* 2 for Unused import
* 1 for Unused local variable

This pull request **introduces 3 alerts** when merging 00e01e7951e3a7079a88f4df6c5917834cecc4b5 into 2da227a757857fcbd578bad3ba6292de2230ea3d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-b560759e6b5db0a69d3f9c2b307b7333e9877468)

**new alerts:**

* 2 for Unused import
* 1 for Unused local variable

This pull request **introduces 3 alerts** when merging 762b26a9f5ff4012229c0aee7cb4a22dc857d1c8 into b2916578ff7616dcf7d99e3ec5ce98294fa45e91 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-1b08fca26be2960f5071a99e6644c983e65a1442)

**new alerts:**

* 2 for Unused import
* 1 for Unused local variable
TTS Collection Cleanup
State: closed | Pull request: yes

Updates all TTS to new `_target_` hydra config.
Improve tutorial.ipynb to use parent classes
Make a better OperationMode setter for tts models
Update TTS docs

This pull request **introduces 10 alerts** when merging 14140663268bf215e0b77adf3219ae2e9aefa45f into 4561fbfc91c2ffe80be55fdbe9edc9611bd0b2c0 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-ef9bb12c9428d747aca58bd86b42637053f34682)

**new alerts:**

* 10 for Unused import

This pull request **introduces 5 alerts** when merging 2c1f807edd54d2da6f9b1953bf9ef1b338f935a1 into 4561fbfc91c2ffe80be55fdbe9edc9611bd0b2c0 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-e1e17319697e68bb4a9fd6847c52daa12889337a)

**new alerts:**

* 5 for Unused import
Should I modify the Quartznet 5x15 architecture for longer sentences?
State: closed | Pull request: no

I have training data where some clips are up to 33 seconds long. I've tried splitting them on silence and allocating the text proportional to size but that is very ad hoc.  So I prefer to keep them.

I've noticed during training that the QuartzNet 15x5 tends to more errors at the end of a phrase than at the beginning.  I feel like it may be losing long-term dependencies or that those are harder to learn.

On the other hand, what I hear you saying about the parameter space is that it is independent of the `max_duration` parameter of the config.  So same number of parameters to parse 1 second phrases as 33 second phrases.

Is that really true?  Or can I get better long-term dependency by adding layers?  If so, can you recommend a config which will work better for training against longer phrases?

If that would improve matters, I assume that means I have to train from scratch?  I.e. if I add layers, I can't use the pretrained weights for the base Q15x5?  Or if I can, can you give me a snippet of code which would show me how to add the extra layers to the pretrained model?


Yes, increasing depth will increase your effective receptive field. 
But, yes, then you'll have to train from scratch.


So 5x30 or 10x15?  Can you say intuitively what the 5 and 15 do in 5x15?  If it was 5x30 could I start with weights for 5x15 and randomize the extra 5x15 to begin with?

NOTE: I think what I'm really going to do is just manually inspect and split the very long training set annotated phrases into shorter ones, because I can chunk the test set any size I want.  Also to improve performance I will manually inspect and discard bad training data, for example where a 19 word transcription is assocated to a single short "Hunh" sound.  In practice I've observed that 6 seconds is the most anybody speaks without pausing at all.
[Question] Are the examples under the same Apache License 2.0
State: closed | Pull request: no

Are the examples under the same Apache License 2.0 as the rest of the source?

I would like to know if I need to document the changes if I take [this example](https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/02_Online_ASR_Microphone_Demo.ipynb) as the starting point.

Thanks

yes, if not noted otherwise, all files in the repo are under Apache 2.0
How to apply inference with trained model?
State: closed | Pull request: no

I have trained a model using speech_to_text.py and got some files in my checkpoint directory. The files are: 

`QuartzNet15x5--end.ckptQuartzNet15x5--last.ckpt`
`QuartzNet15x5--val_loss=0.00-epoch=77.ckpt`
`QuartzNet15x5--val_loss=0.00-epoch=85.ckpt`

However, there is no .nemo file and I can't apply these file to  speech_to_text_infer.py to perform inference with the command  "--asr_model= QuartzNet15x5--end.ckpt"
So how should I apply inference with the files in the checkpoint directory? The version I use is 1.0.0b1.

The .ckpt files are saved by Pytorch Lightning, you can load them like so:
```
model = model.load_from_checkpoint('foo.ckpt')
```
Every NeMo model is a LightningModule which is a torch.nn.Module. Thus for inference, you can use it either with PTL or "pure" Pytorch.

@okuchaiev  what about when you have 3 .pt files  i.e JasperDecoderForCTC-STEP-82000.pt  JasperEncoder-STEP-82000.pt    trainer-STEP-82000.pt
Loading NLP and ASR models might result in `Missing key(s) in state_dict` error
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

After #1278 is merged, any model that uses Metrics (WERBPE, WER, TopKClassificationAccuracy, ClassificationReport, Perplexity) might throw an error like:
```
RuntimeError: Error(s) in loading state_dict for PunctuationCapitalizationModel:

	Missing key(s) in state_dict: "punct_class_report.tp", "punct_class_report.fn", "punct_class_report.fp", "capit_class_report.tp", "capit_class_report.fn", "capit_class_report.fp". 
```

**Steps/Code to reproduce bug**

N/A

**Expected behavior**

N/A

**Environment overview (please complete the following information)**

N/A

**Environment details**

N/A

**Additional context**

The current work-around is to pass strict=False to either
NeMo's Model.from_pretrained(..., strict=False)
NeMo's ModelPT.load_from_checkpoint(..., strict=False)
Lightnings's LightningModule.load_from_checkpoint(..., strict=False)
Torch's Module.load_state_dict(..., strict=False)
[Question] what is benefit of lazy execution and also other nemo datatype such NMtensor
State: closed | Pull request: no

Hi, thanks for such an awesome project, but I am confusing **why nemo make the eager execution of pytorch to be lazy execution** and also some other data structure such as **NMtensor**. I have search all the documentation but did not find some more explicit explanation. Really to hope to get some answers!


hi, we recently completed a major re-design on NeMo and this (lazy execution and NMtensor) is no longer the applicable.

Starting from 1.0.0b1 NeMo is built on top of Pytorch Lightning. It still has Neural Types though in slightly better form. I would recommend you refer to this blogpost for high-level overview and pointers to more details https://medium.com/pytorch/nvidia-nemo-neural-modules-and-models-for-conversational-ai-d660480d9696
How do I set num_workers in DataLoader for an ASR model?
State: closed | Pull request: no

**Describe your question**

I am starting up training for an ASR model like this:

```
from ruamel.yaml import YAML
from omegaconf import DictConfig
config_path = 'amharic_16000.yaml'
yaml = YAML(typ='safe')
with open(config_path) as f:
    params = yaml.load(f)
train_manifest=f'{C.build_dir}/train_manifest.json'
test_manifest=f'{C.build_dir}/test_manifest.json'
params['model']['train_ds']['manifest_filepath'] = train_manifest
params['model']['validation_ds']['manifest_filepath'] = test_manifest
model.set_trainer(trainer)
model.setup_training_data(train_data_config=params['model']['train_ds'])
model.setup_validation_data(val_data_config=params['model']['validation_ds'])
model.setup_optimization(optim_config=DictConfig(params['model']['optim']))
trainer.fit(model)
```
Then `trainer.fit` gives me the message

```
[NeMo W 2020-10-15 21:33:23 nemo_logging:349] /home/catskills/anaconda3/envs/nemo/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:37:
 UserWarning: The dataloader, train dataloader, does not have many workers which may be a bottleneck. 
Consider increasing the value of the `num_workers` argument` 
(try 16 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
```
Where/how do I set `num_workers`?

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment overview (please complete the following information)**

 - Environment location: Ubuntu 20.04, Anaconda Python 3.7
 - Method of NeMo install: pip install

**Environment details**

- OS version: Ubuntu 20.04LTS
- PyTorch version: 1.6.0
- Python version: 3.7.4
- GPU model: NVidia GeForce RTX 2080TI
- CUDA: 10.1.243
- cuDNN: 7.6.5
Add any other context about the problem here.
Example: GPU model


Actually I think I figured it out, it goes in the YAML under `train_ds` and `validation_ds` per `NeMo/examples/tts/conf/glow_tts.yaml`.
Strange # of hours
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I put in 10 hours of clips and I get this:

```
[NeMo I 2020-10-15 20:26:54 collections:173] Dataset loaded with 9341 files totalling 259472.22 hours
[NeMo I 2020-10-15 20:26:54 collections:174] 0 files were filtered totalling 0.00 hours

```
**Steps/Code to reproduce bug**

```
import nemo.collections.asr as nemo_asr
from glob import glob
model = nemo_asr.models.EncDecCTCModel.load_from_checkpoint(model_fn)
files=list(sorted(glob(f'audio/*.wav')))
translations=model.transcribe(paths2audio_files=files, batch_size=16)
```

**Expected behavior**

When I run this:

```
model.setup_training_data(train_data_config=params['model']['train_ds'])
model.setup_validation_data(val_data_config=params['model']['validation_ds'])
```
I get this, which is a more realistic count of # hours:

```
[NeMo I 2020-10-15 20:19:29 collections:173] Dataset loaded with 8367 files totalling 8.43 hours
[NeMo I 2020-10-15 20:19:29 collections:174] 30 files were filtered totalling 0.17 hours
```
**Environment overview (please complete the following information)**

 - Environment location: Ubuntu 20.04, Anaconda Python 3.7
 - Method of NeMo install: pip install

**Environment details**

- OS version: Ubuntu 20.04LTS
- PyTorch version: 1.6.0
- Python version: 3.7.4
- GPU model: NVidia GeForce RTX 2080TI
- CUDA: 10.1.243
- cuDNN: 7.6.5


@okuchaiev FYI

@catskillsresearch not sure what is going on here. Could you please help by answering these questions:
1) Is the number of files correct (e.g. 9341)
2) In .json manifest files, is field "duration" populated correctly?
3) which code produced 259472.22 hours? Did you use setup_training/validation_data method?


@okuchaiev I think it's the duration of 100000 in this line https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/models/ctc_models.py#L143 multiplied by 9341 files. 

9341 * 100000 seconds ~ 259 472.222 hours

@catskillsresearch can you please confirm that this happens with transcribe method call?

Yes, I put a quit() after that line, and it stops there.

I can see it doesn't matter, it's just putting in dummy values for transcription.

However it is a source of confusion to print out that # of hours.

I guess it would cost extra to load each file just for that count to find the actual  duration.

You could have the loader take in manifest records with the duration populated instead of just a list of files, as an alternate method.

Can I know why the wav file format has nohash in each file. As well as the wav file format I am unable to understand.
Perplexity is computed incorrectly
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Perplexity is computed incorrectly [here](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/metrics/perplexity.py) and in [TransformerLMModel](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/models/language_modeling/transformer_lm_model.py). Perplexity is an exponent of entropy whereas in these scripts it is computed as an exponent of cross-entropy.
Saved model named QuartNet15x5--val_loss-0.00-epoch-xxx.ckpt but val_loss on tensorboard is non-zero
State: closed | Pull request: no

Hi,

I used NeMo-v1.0.0b1 and trained the ASR model by modifying examples/asr/speech_to_text.py.
After training I got 3 models named QuartzNet15x5--val_loss-0.00-epoch-xxx.ckpt.
The val_loss in model name is 0.00 but is non-zero on tensorboard.

<img width="353" alt="picture" src="https://user-images.githubusercontent.com/19429441/95961171-48111000-0e37-11eb-9f8c-c356b9e07a52.png">

My solution:
I modified line 550 in nemo/utils/exp_manager.py :
- original
```
checkpoint_callback = NeMoModelCheckpoint(
    filepath=Path(log_dir / 'checkpoints' / '{val_loss:.2f}-{epoch}'), ...
)
```
- my solution
```
checkpoint_callback = NeMoModelCheckpoint(
    filepath=Path(log_dir / 'checkpoints' / '{validation_loss:.2f}-{epoch}'), ...
)
```
The val_loss in model name is correct after modifying the code.
Is this a bug?

Thanks for your reply.

Yes, this is a misconfiguration bug that will raise from failing silently to an application crash after #1278.
We will work to fix this on all models soon. For now, your solution is a good work around if you are sticking to the NeMo ASR models.
Fix Typo
State: closed | Pull request: yes


This pull request **introduces 4 alerts** and **fixes 2** when merging c3eaff2836c80fa495410d35975128e81a5783f1 into 5dc7942ddaa7390c10888dcc5b5b24625d4447eb - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-ff9b11205d6444349936f5cd9ea2c5a196ef4695)

**new alerts:**

* 2 for Unused local variable
* 1 for Signature mismatch in overriding method
* 1 for Unused import

**fixed alerts:**

* 2 for Syntax error
Update Lightning
State: closed | Pull request: yes

Fixes:

- row_log_interval -> log_every_n_steps
- log_save_interval -> flush_logs_every_n_steps
- LearningRateLogger -> LearningRateMonitor

This pull request **introduces 2 alerts** and **fixes 2** when merging 7c84696134f25a2a0db4deeb0956490025789cb0 into b147370de5a05ef29e96901c383a0ac81d9e352e - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-549dc7ace7021ee28a131c1ef4e5f669078256bb)

**new alerts:**

* 2 for Syntax error

**fixed alerts:**

* 1 for Signature mismatch in overriding method
* 1 for Unused import
[Question] Why using Learning Rate = 0.001 instead of 0.01 during fine-tuning ASR NeMo on non-English?
State: closed | Pull request: no

**Question**
According to #1107  which @okuchaiev suggest "**Try 0.001 (or even smaller). your learning rate is certainly too high. **" 
I want to know the reason why we should do that? 
I would appreciate if anyone can help me with that.

**Environment overview:**
-Environment location: Google Colab
-Method of NeMo install: !python -m pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[all]

**Environment details**
-OS version : Ubuntu 18.04
-PyTorch version : 1.6.0+cu101
-Python version : Python 3.6.9
-GPU model: Tesla T4

When we train base model (e.g. from which you fine-tune), we typically decay learning rate using cosine policy. The full lr plot thus looks something like this:
![image](https://user-images.githubusercontent.com/22035961/95894741-5e877080-0d3e-11eb-8c41-1304bb84a3a1.png)
Notice that at the end of the training it is 0 (or very near to zero).

When fine-tuning, you typically re-use the encoder weights (which is most of the network's weights) and therefore significantly increasing learning rate can cause divergence. The decoder, is randomly initialized for fine-tuning and would tolerate a bigger learning rate, but you have to balance that with encoder's needs - hence smaller lr is needed.

Using 10x smaller rate is the advice we found to work well empirically for several transfer learning applications. Please see this paper for experiments: https://arxiv.org/abs/2005.04290 


@okuchaiev what is the default min-lr for the cosine scheduling?

> @okuchaiev what is the default min-lr for the cosine scheduling?

By default it decays to 0 at the end of last epoch. You can change that using ``sched`` configuration section

Thanks you @okuchaiev . You were right.
I changed the Learning Rate from 0.01 to 0.001 while using transfer learning and now WER had been decreased about 30%.
Add Frequency re-sampling
State: closed | Pull request: yes

Add code to change the frequency to the desired (16 kHz) from any pre-specified microphone sample rate.
Solves #1279 

**Approach:**
Calculating the LCM (Least Common Multiple) of the Microphone sample rate and the expected sample rate(16 kHz)
**Upscaling** the Microphone chunk array by duplicating adjacent values.
**Downscaling** the upscaled array to the desired frequency by calculating mean to approximate closely without loss of generality

**Example**
Consider a very common microphone frequency 44.1 kHz or 44100
The LCM of 44100 and 16000 is 7056000

In simple terms:
44100 * 160 = 7056000: Every element in the input array is repeated 160 times in the upscaled array,
16000 * 441 = 7056000: The mean of every 441 elements comprise of a single element in the downscaled array
This ensures that regardless of the input sampling rate, the re-sampled array always has a length of 16000

Thank you very much for this PR! Could you please sign your committed commits. DCO bot's output tells you how to do that https://github.com/NVIDIA/NeMo/pull/1281/checks?check_run_id=1244379368 

Thanks for the PR! It is a nice example of very fast signal resampling.

But I wouldn't recommend to use it due to potential degradation in audio quality and transcription accuracy:
Upsampling by duplicating samples and downsampling by averaging samples will result in severe aliasing in spectrum. 
It might be ok for neural model if it was trained with such distortions. But we employed high-quality sinc-based resampling for all signal preprocessing before training. So while ASR model might produce correct transcriptions on audio with aliasing, it is definitely not optimal.

Thanks, for your changes. However as per @vsl9's advice, I will close this PR.
Add numba_utils to beta
State: closed | Pull request: yes

# Changelog
- Add back the numba_utils for time stretch augmentation on CPU at sample level perturbation

Signed-off-by: smajumdar <titu1994@gmail.com>

where this will be used?

It's checked and imported by perterb.py, but I guess git didn't add this file during the updates to perturb.py. 

This pull request **introduces 6 alerts** and **fixes 9** when merging 47dac024901a77d0cea2aab0e7367c872d26b519 into 661d4321bc5f3db601b6fa9faaf0a3d5dacaaeee - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-993d40422985b33263a94538a908b98585aeb88f)

**new alerts:**

* 4 for Unused import
* 1 for Mismatch in multiple assignment
* 1 for Variable defined multiple times

**fixed alerts:**

* 6 for Unused import
* 1 for Unused local variable
* 1 for Unused argument in a formatting call
* 1 for Comparison of constants
Incompatible SAMPLE_RATE
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Arises when a microphone of a different sample frequency is used to run `/tutorials/02_Online_ASR_Microphone_Demo.ipynb`
Working on a fix as well!
**Steps/Code to reproduce bug**

run the ipython notebook found at `/tutorials/02_Online_ASR_Microphone_Demo.ipynb`
at the last cell. a SAMPLE_RATE of 16000 or 16khz is expected. This cannot be the case always. A simple approach might be to transform the input frequency to the expected frequency using LCM -> Upscaling -> Downscaling. Working on a PR for the same 


this notebook was written for models which we distributed and those are in 16Khz.
You are right that different sampling rate won't work there and re-sampling would be needed. But this looks more like a feature request to add to the notebook rather than bug to me. Feel free to send a PR with it. 
Closing this issue as this is not a bug
Add initial integration with NVIDIA DALI
State: closed | Pull request: yes

Enables the possibility to use NVIDIA DALI for data loading and preprocessing, instead of Pytorch's data loader and torchaudio, respectively.

General idea:
A user can easily switch to a DALI based preprocessing pipeline by simply switching a flag ``use_dali`` in the configuration file.

A new new module `AudioToCharDALIDataset` is introduced, which is able to also handle preprocessing (`AudioToMelSpectrogramPreprocessor` and `AudioToMFCCPreprocessor`) as requested. Because the DALI implementation does things a little bit differently (bypass Pytorch dataloader, combine loading and processing into a single module, etc) it requires special treatment in ctc_models.py.

I was able to obtain significant speedups. I am observing ~3x speed improvement when training Quartznet with a batch size of 128.

Signed-off-by: Joaquin Anton <janton@nvidia.com>

This pull request **introduces 6 alerts** when merging ea3f41cc628b363163f58e9130577d655c2bed2c into ccab141b54fab6fd7f6370e1bd3e18ce0c529364 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-42dbc1969064c1fd79da7e6da14818775c8b4584)

**new alerts:**

* 4 for Unused import
* 2 for Asserting a tuple

Finally, before merge, there needs to be at least one Jenkins task to run the Dali pipeline with fast dev run to ensure the pipeline runs. 

there could be an issue with non-En languages - I'll reach out via email with detailed instructions

> 
> 
> Looks good to me. Thank you!
> Are we ready to merge or this is still a WIP?

I think it is ready to merge.

This pull request **introduces 1 alert** when merging 5cbe85b34d448eb15bb2b97c238a524df1d99494 into cd55594e0f60a4bcbaf6d0e4196c1404591e0076 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-48d04130cd1ee07dfba9a8ce701cf557657e18db)

**new alerts:**

* 1 for First parameter of a method is not named &#39;self&#39;

This pull request **introduces 1 alert** when merging 75d069031ea36b2a1cff00f390280ba937aba42b into cd55594e0f60a4bcbaf6d0e4196c1404591e0076 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-159ef2925cbe9a8ead7e7c20e6882f60eb7cd1f3)

**new alerts:**

* 1 for Syntax error

@okuchaiev @jantonguirao, seems like our CI currently crashed due to this merge. Can we either disable the DALI tests or revert?

@blisc - build and test before the merge:)
I think you can disable until DALI 0.28 is public.
This is rather expected as this PR utilizes functionalities that are not available in the DALI public release, only in the nightly build.
Download Permission Denied
State: closed | Pull request: no
Labels: name: bug

I am getting "permission denied" when trying to pull v1.0.0.b1 using the tag created.

`docker pull nvcr.io/nvidia/nemo:v1.0.0b1`

`Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/images/create?fromImage=nvcr.io%2Fnvidia%2Fnemo&tag=v1.0.0b1: dial unix /var/run/docker.sock: connect: permission denied`

Solved, I had to use Sudo

You can also add your user to the Docker group via these instructions: https://docs.docker.com/engine/install/linux-postinstall/
Then you don't have to use sudo when using Docker commands.
LinearAdapter: propagate args to _init_adapter
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Changes:

1. pass params from `LinearAdapter`'s constructor to `LinearAdapter._init_adapter` -> previously was using `_init_adapter`'s default values
2. switch from dropout 0 to 0.1
**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```
************* Module nemo.collections.llm.peft.lora
nemo/collections/llm/peft/lora.py:158:4: C0116: Missing function or method docstring (missing-function-docstring)

-----------------------------------
Your code has been rated at 9.90/10
```

---

Mitigation guide:

* Add sensible and useful docstrings to functions and methods
* For trivial methods like getter/setters, consider adding `# pylint: disable=C0116` inside the function itself
* To disable multiple functions/methods at once, put a `# pylint: disable=C0116` before the first and a `# pylint: enable=C0116` after the last.

By applying these rules, we reduce the occurance of this message in future.

Thank you for improving NeMo's documentation!

[🤖]: Hi @akoumpa 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12874116340) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
max_steps and time calculation are not working as expected.
State: open | Pull request: no

I'm encountering this issue where it shows `?` instead of the max steps and the time for the training process is not being calculated, this is my configuration:

```
trainer.max_steps=-1 \
trainer.max_epochs=1 \

```

`Epoch 0: :   0%|          | 614/? [23:53<00:00, v_num=, reduced_train_loss=, global_step=, consumed_samples=, train_step_timing in s=, val_loss=]`
ci: Bump Mcore in weekly PR
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
chore(beep boop 🤖): Bump `MCORE_TAG=7ba0d6d...` (2025-01-19)
State: closed | Pull request: yes
Labels: name: Run CICD

🚀 PR to bump `NVIDIA/Megatron-LM` in `Dockerfile.ci` to `MCORE_TAG=7ba0d6ddaab1e4d1c587af8fea8a0dfe13b20f13`.  

📝 Please remember the following to-do's before merge: 
- [ ] Verify the presubmit CI  

🙏 Please merge this PR only if the CI workflow completed successfully.

[🤖]: Hi @ko3n1g 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12851214361) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
chore: Update mcore-tag-bump-bot.yml
State: closed | Pull request: yes
Labels: name: CI

# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)