Repository: pytorch
Issue: Do we really need sampler for IterableDataset?
Labels: module: performance,module: dataloader,triaged

### 🚀 The feature, motivation and pitch

I have noticed that "_InfiniteConstantSampler" may be unnecessary for "IterableDataset". The removal of the "_InfiniteConstantSampler" and related logic in  "_IterableDatasetFetcher" could speedup by 27%. 
In detail, the MapDataset use the "RandomSampler" for the shuffle operation, however "IterableDataset" is mutually exclusive with shuffle operaton. This means that we does not need this sampler in the "IterableDataset" case. Otherwise, is there any other purpose of "_InfiniteConstantSampler" for the "IterableDataset"?

**The orginal code:**
```python
class _IterableDatasetFetcher(_BaseDatasetFetcher):
    def __init__(self, dataset, auto_collation, collate_fn, drop_last):
        super(_IterableDatasetFetcher, self).__init__(dataset, auto_collation, collate_fn, drop_last)
        self.dataset_iter = iter(dataset)
        self.ended = False

    def fetch(self, possibly_batched_index):
        if self.ended:
            raise StopIteration
        
        if self.auto_collation:
            data = []
            try:
                data = [next(self.dataset_iter) for _ in range(self.batch_size)]
            except StopIteration:
                self.ended = True
            if len(data) == 0 or (self.drop_last and len(data) < self.batch_size):
                raise StopIteration
        else:
            data = next(self.dataset_iter)
        return self.collate_fn(data)
```

**New code:**
```python
class _IterableDatasetFetcher(_BaseDatasetFetcher):
    def __init__(self, dataset, auto_collation, collate_fn, drop_last, batch_size):
        super(_IterableDatasetFetcher, self).__init__(dataset, auto_collation, collate_fn, drop_last)
        self.dataset_iter = iter(dataset)
        self.batch_size = batch_size # I set 64 in the experiment

    def fetch(self, possibly_batched_index):
        if self.auto_collation:
            data = [next(self.dataset_iter) for _ in range(self.batch_size)]
            if len(data) == 0 or (self.drop_last and len(data) < self.batch_size):
                raise StopIteration
        else:
            data = next(self.dataset_iter)
        return self.collate_fn(data)
```

### Alternatives

_No response_

### Additional context

**My env:**
**os:** Centos
**cpu:** Intel(R) Xeon(R) CPU E5-2682 v4 @ 2.50GHz
**python:** 3.6.8
**PyTorch version:**  1.11

**Test result:**
I have test on 5000000 samples in single process dataloader.

Experiment Name | Duration
-- | --
Original code | 13.02s
New code | 9.63s (-27%)

</div>

**My Test Script:**
```python
# coding: utf-8
import gc
import time
import torch
from torch.utils.data import IterableDataset
from dataloader_utils import MyDataLoader as DataLoader

gc.collect()
torch.cuda.empty_cache()


class RandomIterableDataset(IterableDataset):
    def __init__(self, num=10) -> None:
        self.num = num
        self.sample = {'id': '1',
                       'url': 'https://xxx.com'
                       }

    def get_sample(self):
        return self.sample

    def __iter__(self):
        return self

    def __next__(self):
        while self.num > 0:
            self.num -= 1
            return self.get_sample()
        raise StopIteration


if __name__ == "__main__":
    batch_size = 64
    avg_times = []
    for _ in range(5):
        cnt = 0
        start_time = time.time()
        input_dataset = RandomIterableDataset(num=5000000)
        input_dataloader = DataLoader(
            dataset=input_dataset,
            shuffle=False,
            batch_size=batch_size,
            drop_last=False,
            num_workers=0,
        )

        for data in input_dataloader:
            cnt += 1
            if cnt % 1000 == 0:
                pass

        end_time = time.time()
        duration = (end_time - start_time)
        print(f'Finish task. Use {duration} secs with batch size {batch_size}')
        avg_times.append(duration)
    print(f'The average time spend is {round(sum(avg_times) / len(avg_times), 2)}s')

```

cc @VitalyFedyunin @ngimel @SsnL @ejguan @NivekT