Basic support for additional audio file formats (.mp3, .flac...etc)
State: closed | Pull request: no
Labels: name: feature

**Is your feature request related to a problem? Please describe.**

It appears that NeMo only works with wav files, but they are 10x larger than equivalent MP3s. Many popular datasets like Mozilla Commonvoice are stored as MP3 and have to be converted before being used. This is time consuming and also makes it harder for those using limited resources like Google Colab. 

I was wondering if there's a specific reason that only WAV is used. 

**Describe the solution you'd like**

For NeMo to work with non wav files. 


Any word on this @titu1994? I'm curious if theres a specific reason. I'm starting to use CommonVoice English and it's 56GB when stored as MP3 so it will be unusable in wav format. Thank you!

@vsl9 Any thoughts about this ? 

Hi

torchaudio developer here. Recently [I am adding support for file-like object in torchaudio's I/O functions](https://github.com/pytorch/audio/pull/1158), and I am wondering what extra features would be required for NeMo to consider using `torchaudio.load`. In #1616, I see that tarfile support is one, which is working now. Do you have any other requirements? What about Windows support?



@mthrok Sorry for the delay, just saw the previous #1616 conversation regarding this. wav, tarfiles via webdataset, and mp3 are currently supported, but it would be nice for torchaudio to also support FLAC if possible, since MLS dataset is in that format - http://www.openslr.org/94/. 

The reason why we don't add torchaudio as an explicit dependency, and instead make it optional for certain modules, is the fact that torchaudio previously pinned it's version to a specific pytorch version. I am unsure of whether this is still the case. 

This is generally fine for day to day use, but interferes with containers that utilize commit hash version of pytorch (Torchaudio pinning forces reinstallation of pytorch). This issue goes away by building torchaudio from main branch, but building torchaudio is a time-consuming process which we don't want to constantly do on the fly. 

One way to alleviate this is to have the Nemo container build torchaudio, but our ci uses pytorch container and pip installs the Nemo library in order to simulate user workflow  - and torchaudios pinned pytorch version interfered with this. 

We would be very happy to add torch audio data loading as an option for various audio formats, but it will have to remain as an optional dependency for users until the pinned version issue is resolved. 

Thanks for the explanation. Please note that torchaudio supports FLAC. (although we get the report that PySoundFile is faster and I think it makes sense as libsox does extra stuff in addition to loading.)

Regarding the pinned versioning, I understand your constraints and unfortunately we cannot do much about it. torchaudio's extension module is coupled with PyTorch's C++ backend so torchaudio cannot be paired with arbitral version of PyTorch. And because torchaudio uses the latest extension mechanism PyTorch provides, often times torchaudio cannot be compiled with the previous release of PyTorch.

If you think of something torchaudio can help, please let us know. 

FLAC is supported by default as it is part of soundfile library. MP3 is specially supported via pydub. We still recommend wav files when training for efficiency reasons - wav file is slightly faster to decode than flac or mp3