Integrate Wav2Vec 2.0
State: closed | Pull request: no
Labels: name: feature

Hey guys! Well done on the repo and congrats on the PL 1.0 integration :)

Something I've been working on for a bit is migrating [fairseq's Wav2Vec 2.0](https://github.com/pytorch/fairseq/tree/master/examples/wav2vec) implementation to NeMo, primarily because FairSeq is a pain to install, has far too much boilerplate for my liking and doesn't offer the flexibility/integration like NeMo (backed by PL).

Based on results from the [paper](https://arxiv.org/pdf/2006.11477.pdf) this model has extreme promise. If it's made more accessible to users and easier to scale, we could continue to see impressive results!

You can see my progress [here](https://github.com/SeanNaren/NeMo/tree/feature/wav2vec/nemo/collections/asr/models/wav2vec) which has stagnated a bit (planning to get back to it soon).

Questions I have to the team are:

- Is there the all clear to integrate Wav2Vec into NeMo via porting code from FairSeq into the repo? Currently it seems most support ASR architectures are NVIDIA based research, but I'm hoping for flexibility here :) The alternative would be to live on it's own fork or branch but I'd rather see it make its way into NeMo
- Is there enough popularity for the model for any assistance in the integration portion? I think I can manage on my own, but would be nice to gain additional eyes


Thanks for starting this! Yes, wave2vec 2.0 looks very intriguing and is something we wanted to try but did not have a chance yet.

To answer your questions:

1. Yes! We will be happy to see your wave2vec 2.0 PR. Please have a look at https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md and reach out to us if you have any questions. NeMo was never intended to only support NVIDIA-based research, so external models are very welcome too.

2. I pretty sure we'll be able to help with code and pre-training some models, so that others can built upon them.

Thanks @okuchaiev! Another question I had is the collection this belongs to. Technically the pre-training only requires unlabelled audio, and then the subsequent model can be fine-tuned for ASR, but I think all the code should live in the ASR collection for the time being. Was curious if there was any disagreement here

ASR collection sounds like the right fit to me.

Hi Sean! Please open a WIP PR with what you have. I would help you with any questions or problems.

Will do :)

Exciting! Looking forward to this and NVIDIA's pre-training + noise augmentation. 

@SeanNaren will the pretrained weights shared by fairseq be compatible with your implementation on NeMo? @okuchaiev  if not will NVidia provide one in NGC?
P.S. It might be worth looking into https://arxiv.org/abs/2010.10504 (using conformer layers in a wav2vec2 training) as well.

I just wanted to say that this will be really great for the community -- very much looking forward to seeing this in NeMo!!

> @SeanNaren will the pretrained weights shared by fairseq be compatible with your implementation on NeMo? @okuchaiev if not will NVidia provide one in NGC?
> P.S. It might be worth looking into https://arxiv.org/abs/2010.10504 (using conformer layers in a wav2vec2 training) as well.

Eventually I hope we create our own pre-trained models so we're not tied to the fairseq implementation. I feel like there are lots of cleanliness issues with the codebase, once I get ASR fine-tuning working, I plan to make a few clean-ups to the code for readability.

I can't promise, but I think it is very likely we'll provide pre-trained checkpoints.
Also, @VahidooX is currently adding Conformer-like ASR model

This might need to be re-opened, we're still missing the fine-tuning piece!

The community greatly supports your integration of Wav2vec into Nemo.
Any plan for releasing the update ?

Hey! apologies on the radio silence, I haven't had a chance to do the fine-tuning port, but hoping to get some time this week to bring the code as a separate PR. This would complete the code integration however models would still need to be made.

Ties well with this dataset however https://ai.facebook.com/blog/a-new-open-data-set-for-multilingual-speech-research/ :) 

That's excellent !
And you link is an "Ali Baba Cave" !

Hi any updates on this ?

waiting for the implementation ..

This will be added soon in #3297 but we are still training a few models to make sure the convergence is correct.