`audio_to_text` doesn't support duplicate files with different offsets
State: closed | Pull request: no
Labels: name: bug

name: stale

When building tarred datasets with `convert_to_tarred_dataset` the script treats duplicate files with concatenating `-sub` to the file name. This allows the manifest to track number of samples while avoiding too many duplicate files in the tar balls. For `audio_to_label` data, this `-sub` is picked up by the class and used to duplicate audio streams.

For `audio_to_text` data, there is not support for this concatenation, the `_loop_offset` method still assumes unique files. As such, anytime you use multiple offsets from the same audio file, the dataloader conflicts with the audio stream, causing hanging.

**Steps/Code to reproduce bug**
- Build a manifest file with multiple offsets from same file
- Run convert_to_tarred
- Run training over tarred script
- Watch run hang

**Expected behavior**

- The `audio_to_text` data file should duplicate audio streams for each instance of `-sub` in the manifest. This can be done by simply duplicating the text in `audio_to_label`

**Environment overview (please complete the following information)**

 - Environment location: [Docker container on cs-oci run]
 - Method of NeMo install: source from main



This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.

Hi, I'm facing the same problem in NeMo 2.0. It took me a long time to see that the '-sub1', 'sub2'.... don't work. This kind a huge issue since with audio lasting up to 1 hour, you don't want to divide them into every segment you have.

I believe you'll be able to avoid these issues by switching to lhotse dataloaders. For ASR recipes,  just add `train_ds.use_lhotse: True`.

> I believe you'll be able to avoid these issues by switching to lhotse dataloaders. For ASR recipes, just add `train_ds.use_lhotse: True`.

Hi, thanks for the tip but when I try to use lhotse dataloaders I have the following error:

```
[NeMo I 2024-11-27 10:24:19 nemo_logging:381] Creating a Lhotse DynamicCutSampler (bucketing is disabled, (max_batch_duration=None max_batch_size=1)
Error executing job with overrides: []
Traceback (most recent call last):
  File "NeMo/nemo/collections/asr/models/ctc_models.py", line 385, in setup_training_data
    * ceil((len(self._train_dl.dataset) / self.world_size) / train_data_config['batch_size'])
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: object of type 'IterableDatasetWrapper' has no len()
```

I have switched to using sharded manifests as it seems to be required for that type of dataloader. I don't know if this is normal and I need to change something else? but what?

I have the following config for my train_ds:
```
train_ds:
    manifest_filepath:
    - - bucket11/sharded_manifests/manifest__OP_0..23_CL_.json
    - - bucket12/sharded_manifests/manifest__OP_0..23_CL_.json
    tarred_audio_filepaths:
    - - bucket11/audio__OP_0..23_CL_.tar
    - - bucket12/audio__OP_0..23_CL_.tar
    sample_rate: ${model.sample_rate}
    batch_size: 1 # must be 1 if using bucketing_batch_size
    shuffle: true
    num_workers: 8
    use_lhotse: true
    pin_memory: true
    max_duration: 30.1
    min_duration: 0.1
    shuffle_n: 2048
    # tarred datasets
    is_tarred: true
    bucketing_strategy: "synced_randomized" # "fixed_order" "synced_randomized" "fully_randomized"
    bucketing_weights: [1, 1]
    bucketing_batch_size: 8
    shard_strategy: scatter
```
@pzelasko