Wav2Vec 2.0 Integration
State: closed | Pull request: yes

Closes #1342. I think the code has been ported over, but I'm sure it can be cleaned up and contains a few bugs.

Haven't tested the wav2vec_asr.py model at all and it's probably broken, but I managed to get pre-training working on AN4, but still need to investigate further!

Any assistance will be appreciated :) cc @VahidooX 

This pull request **introduces 5 alerts** when merging 8c0e7e3d211117ebadddc45c3491357adb7fc085 into 94b64d32c0c54fd680b34c3b7dfb664b1035481b - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-590b9ded82eb997004669a408fbab57b8f746582)

**new alerts:**

* 3 for Unused import
* 1 for Inconsistent method resolution order
* 1 for Unused local variable

I have started the review. It is 2k+ added lines of code and may take some time. In the meanwhile:
+ Please sign off all your commits. Your previous commits should also get signed. The DCO check gives you the commands how to do that.
+Run "python setup.py style --fix" in the main folder to autoformat the style of the code.
+ There are some LGTM warnings to get fixed.
+ You may pull from main to make it up to date as some important changes have happened recently

It's definitely far from done, and I think I've done a lot of changes here that are not required per say for wav2vec! Hoping to get back to working on this soon

> 
> 
> It's definitely far from done, and I think I've done a lot of changes here that are not required per say for wav2vec! Hoping to get back to working on this soon

It is a large PR with more than 2500 lines. It may take some time, and may need more than one round of review. I did a quick review and put some comments. You may take a look at them whenever you can. We can do another round of review when it is closer to be working.

This pull request **introduces 5 alerts** when merging 5adcca5f702ff612251577ae937da07945af4a17 into 75774cf24755bddb393e3b1a4cbcc1921daf8898 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-d48d57930ccb41cdffb6ffbd9927de928ef56599)

**new alerts:**

* 3 for Unused import
* 1 for Inconsistent method resolution order
* 1 for Unused local variable

This pull request **introduces 3 alerts** when merging 859fb90e01c55d15d1b8fba31ef213df95a4fca4 into 75774cf24755bddb393e3b1a4cbcc1921daf8898 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-9218028773ddc1cb058ee252fc4846b02855376e)

**new alerts:**

* 1 for Inconsistent method resolution order
* 1 for Unused local variable
* 1 for Unused import

This pull request **introduces 3 alerts** when merging 6ea68ca7fae1bfd212f2286d9f590b06f913c6b8 into ece60b02f10494d73b3ca109d2c4fa6ef556ec47 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-2fe1f316ff31b2a3d8065a960a1068e0cea14c89)

**new alerts:**

* 1 for Inconsistent method resolution order
* 1 for Unused local variable
* 1 for Unused import

This pull request **introduces 3 alerts** when merging 9281c2753e744be773c3d15ffc60b267a3e30b03 into f0eeaed82d975b7ba1833185eadda541d004de51 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-ef8a924d9ae72d209dc564e4fb6f89409c1071e4)

**new alerts:**

* 1 for Inconsistent method resolution order
* 1 for Unused local variable
* 1 for Unused import

This pull request **introduces 1 alert** when merging ad6f9606cd4cc3386b00b4b83cfe88ecd7c9c17b into 99ed9c2ae5c840a8723f236530ccb1abe8c75377 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-885ae99a66fdfdadbb3c1c2e7f4201bb133fc412)

**new alerts:**

* 1 for Unused local variable

This pull request **introduces 1 alert** when merging e0609f220b7892c48e57d7c15381e4268bb8be8e into 99ed9c2ae5c840a8723f236530ccb1abe8c75377 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-79e6baa6a67092c16f519462fc92217a9be1eb04)

**new alerts:**

* 1 for Unused local variable

I'm still plugging away at this! Currently working on fixing the _asr part of the model.

This pull request **introduces 1 alert** when merging 6b222801860b53af3f53b56602de97b7f1de251d into 5b6dd3850129898be47cf0d65587897ec45a5b59 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-2e7b9ab2f75863578795c7e1095dd415e1c864d8)

**new alerts:**

* 1 for Unused local variable

This pull request **introduces 1 alert** when merging 931f52012a65adc1110b309493fc3bfd8467d2e7 into 5b6dd3850129898be47cf0d65587897ec45a5b59 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-d5d0615c7db662d2f81b690aa617611d5cbed223)

**new alerts:**

* 1 for Unused local variable

This pull request **introduces 2 alerts** when merging c796af5463e19dc909943dd2e76425271088fd25 into 5b6dd3850129898be47cf0d65587897ec45a5b59 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-0a26c3076a69e3d2bc054f0989cf5b1b190ddf9f)

**new alerts:**

* 1 for Unused local variable
* 1 for Unused import

This pull request **introduces 1 alert** when merging e208c1a3be7b85bf82c6fc7abe1bc4c955e8e360 into 5b6dd3850129898be47cf0d65587897ec45a5b59 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-c2732d6b47b826a8cf7ff9e7dbd5ab0b68f6ce93)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 9e4ba49392bc80ed5d807c453e776da5d38b0c87 into 5b6dd3850129898be47cf0d65587897ec45a5b59 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-f28d04442be47dc9858ee62568f2c877d0e1dce2)

**new alerts:**

* 1 for Unused import

This pull request **introduces 2 alerts** when merging 07eb404e25b6f05dfcd15e251164c10f10b221df into 30b2ed2a84b5e568d15b97079b15a9559450f480 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-7873b032551df5e45b13b07c8a75437785b5bf93)

**new alerts:**

* 2 for Unused import

This pull request **introduces 2 alerts** when merging f95b247d75694874b75562f815436de179792c54 into 4e804fd88bc197d031e97972957d8ad21e0c002f - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-90f03bcc7d66c9fd1c19abd5688ee1cab11f8fbf)

**new alerts:**

* 2 for Unused import

Thanks for all the suggestion @VahidooX @okuchaiev, I addressed most of them but need to follow up on a few, let me give you guys a status update:

Whats been done:

- The code works! I've successfully pre-trained a model on AN4/fine-tuned on AN4, and am working on testing on mozilla common voice right now. Novograd seems to work well, so will be using that instead of adam. I don't have much GPU resources (using AWS atm) but will leave it over the weekend and see how pre-training goes.
- I've cleaned up the code extensively. A lot of the fairseq code seemed very weirdly interconnected so I spent a lot of time ensuring the code flow makes sense, and adding a bit of explanation where necessary. I've removed a lot of modules which were not necessary with latest AMP changes and such, and rely on torch modules where I can.
- Code sits on all existing NeMo code now with no code changes required :)

What needs to be addressed:

- (EDIT: done)! Padding mask is being generated in the forward pass of the wav2vec model. I tested the runtime here and it's very minimal overhead but not vectorized. I'm going to try vectorize it.
- I created wav2vec base to wrap basic dataloader functions that I copied from the ctc_model.py. I feel like there is a refactor that can happen to make this an inheritance. I assume NeMo never ran into this because the ctc_model class worked well for all models supported
- As suggested, pull out the fine-tuning portion for a separate PR. But I'll keep it here for the time-being for model training sake as we ensure the codebase works. 

My current steps:

- Train mozilla pre-trained model, ensure convergence. Ensure fine-tuning works, and compare against a weaker baseline to ensure the model has learnt something from pre-training. EDIT: I'm running into OOM issues, if the model can be trained on a 32gb or higher VRAM card this will be beneficial. Currently I'm pruning the manifest to 7 second files to keep the memory low on a 16gb VRAM card.
- Pull out fine-tuning for separate PR

Eventually doc changes etc but lets get the code working first.

If @VahidooX @okuchaiev can do code reviews again please do!! I'll update with mozilla results as soon as I can.

Great review @VahidooX super thorough which is what I needed. Will take time to address everything but will get there!

This pull request **introduces 1 alert** when merging 13f689cf01bf7881f661762cac4798bc16ef59bd into 4efd886121e5dd5fe25792116fd0c6afca036b98 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-2fa6f899df9c5b3aba9f8c04d2888ac844cc6e06)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 1f941e7a14b0f5d1898927dfd37a4546f08ca5a4 into cd47137817e6d686f7551db18fd35610fd442343 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-1f480ecb9864b59c383b6e2bacf8608125b36b58)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 02b499bb39793d59712f57bdd9c2c745481fb6da into 1c24a5d35a9680b7436f505a8e69dc8f8373f3c2 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-dcd8e800d90d66d0fb9649e3f868cfc198f23642)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 4e97e126b423bcb888469b43397ed7c4dd0e9f19 into 51092ef7a9810ab096148cdf2d44ddfe6c72d028 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-3e11f6b66b14676dc90d9791f2eb2d861dbe1e71)

**new alerts:**

* 1 for Unused import

Let's try get this over the finish line! @VahidooX I tried my best to answer as much as I can from the review but a lot became outdated. Would you be able to carry out another review on top?

I also tried my best to clear out some of the hydra config stuff and expose them all in the yaml file but you may have a better idea on how to do so, please lemme know!

Finally I'd like to get rid of the `wav2vec_base` class, which is only needed because there is no base class currently that only contains the data loading aspect of `ctc_models.py`. If we make `ctc_models.py` inherit from a similar class, then I think I'll be able to use that base class. Any suggestions?

This pull request **introduces 1 alert** when merging 569a5c94c05314f21b98a441e25ef41c95db18ef into 4f46bd84d162da80a310c69f72980de935c1e99a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-6522d0e1c0b32401060e1adfea8004a59f00c03e)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 9f760fb9f347b158889b6e5ad080a7ae3e8f5017 into 4f46bd84d162da80a310c69f72980de935c1e99a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-bb7528edcc01028ef93ae413a89a5d77d2532c9e)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging cfeec2b466b2e4219777792c3063f53a8cb1026d into 4f46bd84d162da80a310c69f72980de935c1e99a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-166680e05e4b47e397e78d39247381b264885d15)

**new alerts:**

* 1 for Unused import