Tar file datasets don't allow for "/"in key names for ASR training.
State: closed | Pull request: no
Labels: name: feature

**Is your feature request related to a problem? Please describe.**

Tar file datasets have two problems today that I am aware of:

1. The filenames inside tar files are *required* not to have "/" or "." in them. Disallowing "." is fine, given that it has special meaning in webdataset. Disallowing "/" causes some difficulties. First, if your key is larger than 255 elements and you convert "/" to "_" (as is done here https://github.com/NVIDIA/NeMo/blob/daf08760dde35433f934a5c38ab293bad3c22825/scripts/speech_recognition/convert_to_tarred_audio_dataset.py#L560, then it is not possible to untar the created tar file. You will get an error like this when running `untar`:

```
File name too long: 'gov_DOT_uscourts_DOT_illappct_DOT_1-07-3566-1-08-0702-1-08-0746-1-08-1373-1-08-1982-1-08-1983-1-08-2151-1-08-2152_gov_DOT_uscourts_DOT_illappct_DOT_1-07-3566-1-08-0702-1-08-0746-1-08-1373-1-08-1982-1-08-1983-1-08-2151-1-08-2152_DOT_2009-11-12_DOT_mp3_00000.flac'
```

This is because linux file names are at most 255 characters: https://serverfault.com/a/9548

2. Suppose that your tar dataset contains the following 4 entries:

```
speaker1/utterance1.wav
speaker1/utterance2.wav
speaker2/utterance1.wav
speaker2/utterance2.wav
```

Today, NeMo will convert the key `speaker1/utterance1.wav` into `${parent_dir_of_manifest}/speaker1/utterance1.wav`. That is to say that it believes that this makes. The code which does that is here:

https://github.com/NVIDIA/NeMo/blob/d8785e022e2ccaf8b02a27ca084aea5acd257b18/nemo/collections/common/parts/preprocessing/manifest.py#L92-L104

This code assumes that the keys of tar files have no "/" in them. A more straightforward way is to pass another argument "is_tar_dataset" into this function to accept tar files appropriately. I will show this below.

**Describe the solution you'd like**

I will link to a work-in-progress PR for "A more straightforward way is to pass another argument "is_tar_dataset" into this function to accept tar files appropriately." shortly after creating this issue. Edit: https://github.com/NVIDIA/NeMo/pull/4456

**Describe alternatives you've considered**

I wrote a quick workaround that converts "/" to "_" on the fly after loading the tar file and manifest here: https://github.com/NVIDIA/NeMo/compare/main...galv:tar-remove-slash-workaround

While you could make a "fix" by doing this conversion only when a tarred dataset is being used, this is rather clumsy since the ID changes. This makes debugging and downstream joins hard.

**Additional context**

Why should you make this change? NeMo tarred datasets can currently be created only by its convert_to_tarred_audio_dataset.py script, which cannot be parallelized via something like spark or some other big data processing framework. It is hard for users of big data technology to train on tarred datasets created directly via spark for this reason. Secondly, tar files are kind of the de facto standard by which speech training data is transferred today (see openslr.org, or https://huggingface.co/datasets/MLCommons/peoples_speech). It saves users time and money to be able to train directly on the tar files they have downloaded, instead of having to untar the download, and the retar it with changed names. The existing path basically increases the disk usage required from 2TiB (in the case of People's Speech) to 6TiB.

Additionally, if someone were training on a nemo tarred dataset created by convert_to_tarred_audio_dataset.py and didn't have the underlying data, they would not be able to untar most files simply to listen to them, as their keys will  most likely exceed 255 characters in length. This makes observability hard.