Repository: pytorch
Issue: Insufficient shared memory (shm) while training
Labels: module: memory usage,triaged

## ❓ Questions and Help

Hi,

I am trying to reproduce the baseline ViLBERT from the Hateful meme challenge (https://github.com/facebookresearch/mmf/tree/master/projects/hateful_memes) and I am running into an issue while training using the following command:
Error:

```
2021-01-14T14:15:39 | mmf_cli.run: Torch version: 1.6.0                                                                                                                                                                                                                         
2021-01-14T14:15:39 | mmf.utils.general: CUDA Device 0 is: GeForce GTX 1080 
2021-01-20T14:15:18 | mmf.utils.general: Total Parameters: 247780354. Trained Parameters: 247780354
2021-01-20T14:15:18 | mmf.trainers.core.training_loop: Starting training...
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
Traceback (most recent call last):
  File "/home/kclabill/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 779, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 107, in get
    if not self._poll(timeout):
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 424, in _poll
    r = wait([self], timeout)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 931, in wait
    ready = selector.select(timeout)
  File "/usr/lib/python3.8/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
  File "/home/kclabill/.local/lib/python3.8/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 61581) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kclabill/.local/bin/mmf_run", line 8, in <module>
    sys.exit(run())
  File "/home/kclabill/.local/lib/python3.8/site-packages/mmf_cli/run.py", line 122, in run
    main(configuration, predict=predict)
  File "/home/kclabill/.local/lib/python3.8/site-packages/mmf_cli/run.py", line 56, in main
    trainer.train()
  File "/home/kclabill/.local/lib/python3.8/site-packages/mmf/trainers/mmf_trainer.py", line 123, in train
    self.training_loop()
  File "/home/kclabill/.local/lib/python3.8/site-packages/mmf/trainers/core/training_loop.py", line 31, in training_loop
    self.run_training_epoch()
  File "/home/kclabill/.local/lib/python3.8/site-packages/mmf/trainers/core/training_loop.py", line 75, in run_training_epoch
    for idx, batch in enumerate(self.train_loader):
  File "/home/kclabill/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 363, in __next__
    data = self._next_data()
  File "/home/kclabill/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 974, in _next_data
    idx, data = self._get_data()
  File "/home/kclabill/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 941, in _get_data
    success, data = self._try_get_data()
  File "/home/kclabill/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 792, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 61581) exited unexpectedly
```

I have increased my shared memory to 100G, my swap memory to 80G, I have 15G of RAM.
 I have tried to set the batch size to 1 and num_workers to 0 but I get the following error when I do:
```
2021-01-14T14:51:02 | mmf.utils.general: Total Parameters: 247780354. Trained Parameters: 247780354
2021-01-14T14:51:02 | mmf.trainers.core.training_loop: Starting training...
./train.sh: line 19: 61816 Bus error               (core dumped) mmf_run config=projects/hateful_memes/configs/vilbert/defaults.yaml model=vilbert dataset=hateful_memes run_type=train_val training.num_workers=0

```
Any idea how to fix this?
Thanks!