Repository: pytorch
Issue: Training fast with small dataset, slow with large dataset
Labels: needs reproduction,module: performance,triaged,module: data parallel,module: amp (automated mixed precision)

## 🐛 Bug

I'm training a GAN drawing samples from LMDBs. When the number of samples is small - say less than 20K, training speeds are great and consistent. When I increase the number of samples, as high as 65K, I notice much slower training - 5x-10x slower on average per iteration. 

## To Reproduce

Steps to reproduce the behavior:

-This is on a custom dataset/codebase, but I will work with the PyTorch/Nvidia devs over on the PyTorch forum to triage/figure out how best to reproduce. 

https://discuss.pytorch.org/t/training-fast-with-small-dataset-slow-with-large-dataset/95247

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

The training speed in seconds per iteration shouldn't change as the dataset gets bigger. 

## Environment

PyTorch version: 1.6.0
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Red Hat Enterprise Linux Server release 7.7 (Maipo) (x86_64)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
Clang version: 6.0.1 
CMake version: Could not collect

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 0.2.1221


Nvidia driver version: 450.51.06
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] torch==1.6.0
[pip3] torchvision==0.7.0
[conda] Could not collect

## Additional context

I am training a GAN drawing samples from LMDBs. I have multiple LMDBs - I create a dataset for each and concatenate them to make the final one. I was initially training with about 4K samples split across 16 LMDBs. I recently scaled up to about 60K samples, split across 40 LMDBs. I noticed significant slowing in the training when increasing the dataset (after 5 epochs, as I know sometimes the initial epoch is slow due to caching & initialization etc.).

I’m trying to figure out why the training speed (images/second) is 5-10x slower per iteration when using the large dataset than on the smaller one. I observed as I reduce the size of the training set, the training speed progressively gets faster and faster until reaching the expected speed. Using up to ~30K images shows no slowness, and the speeds drop dramatically from there. It doesn’t appear to matter which images are in the reduced dataset - as long as the number of images >30Kish, the training speed drops dramatically.

I’m training on V100’s on a DGX2, using DataParallel() and apex AMP. Using PyTorch 1.4 (and recently upgraded to 1.6 and PyTorch autocast AMP). My minibatches are ~96. Increasing num_workers beyond 6 doesn’t seem to help. Whether I use 1, 2, or 4 GPUs at a time, I still observe the “slower training with more data” phenomenon. The only difference between fast training and slow training in my testing is the number of images included in the dataset. 

**Background:**

This is on an nvidia DGX2. I confirmed that all data is cached on the NVMe cache (as the cache does not grow as training progresses)
I removed DataParallel()/DDP()/AMP, and am training on one GPU as @mcarilli suggested
All hyperparams including minibatch and n_workers for the dataloader are kept the same for all tests
I invoke time.time() after each iteration to get the seconds per iteration.


**Profiling results:**

While training with a small dataset (4k samples), it takes 1.2 seconds per iteration, and that speed is consistent after tens of thousands of iterations

While training with a large dataset (65k samples), it takes an average of 2 seconds per iteration. The way the slowness manifests is a handful of “fast” iteration at 1.2 seconds/iter, followed by a slow iteration that takes 4-10 seconds. It averages out close to 2 seconds/iter. When scaling up to many GPUs, the speed penalty (average seconds/iteration) with the large dataset is 5x or more (and only a negligible speed penalty with a small dataset).

To profile the RAM, I am calling the watch free command. It stays fairly consistent (doesn’t blow up as training progresses):

```
              total        used        free      shared  buff/cache   available
Mem:     1583405972    32676452     1541340      351212  1549188180  1548865308
Swap:       3904508      279808     3624700
```





cc @mcarilli @VitalyFedyunin @ngimel