Repository: pytorch
Issue: Impossible to raise the limit on number of shared memory tensors
Labels: high priority,module: multiprocessing,triaged

## 🐛 Bug

## To Reproduce

Steps to reproduce the behavior:

1. Allocate a tensor
1. Move it to shared memory
1. Repeat until it fails

<details>
<summary>My little diagnostics script</summary>
<p>

```python
from tqdm import tqdm
from time import sleep 
import torch as ch
import argparse

import os
mem_bytes = os.sysconf('SC_PAGE_SIZE') * os.sysconf('SC_PHYS_PAGES')  # e.g. 4015976448

def run_test( queue, strategy, obj):
    ch.multiprocessing.set_sharing_strategy(strategy)
    if obj == 'count': # Test maximum number of files that we can create
        size = 1024 * 1024 # 1Mb
    if obj == 'size': # Try to test how much memory we can allocate
        size = 1024 * 1024 * 1024 # 1GB

    tot = []
    while True:
        try:
            tot.append(ch.zeros((size // 4, )).share_memory_())
            queue.put(size)
        except Exception as e:
            print(e)
            break

def test(strategy, obj):
    queue = ch.multiprocessing.Queue()

    process = ch.multiprocessing.Process(target=run_test, args=(queue, strategy, obj))
    process.start()

    sizes = []
    with tqdm(total=mem_bytes / 2**30) as pbar:
        while True:
            if queue.qsize() != 0:
                got = queue.get()
                pbar.update(got / 2**30)
                pbar.set_postfix({ 'num_files': len(sizes) })
                sizes.append(got)
            else:
                if not process.is_alive():
                    break
                sleep(0.1)
    print(f"Result for {strategy} and {obj}: {len(sizes)} files, {sum(sizes) / 2**30}GB")


if __name__ == '__main__':
    test('file_system', 'count')
    test('file_descriptor', 'count')
    test('file_descriptor', 'size')
    test('file_system', 'size')
```
</p>
</details>  

## Expected behavior

Being able to create as many shared tensor as  the output of `ulimit -Sn`. Using my diagnostics script I get this Instead:

```
python test_shared_memory.py 
 13%|███████████▎                                                                              | 63.3818359375/503.7639808654785 [00:39<04:22,  1.68it/s, num_files=64902]
[enforce fail at CPUAllocator.cpp:71] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 1048576 bytes. Error code 12 (Cannot allocate memory)
 13%|███████████▎                                                                              | 63.4111328125/503.7639808654785 [00:45<05:13,  1.41it/s, num_files=64932]
Result for file_descriptor and count: 64933 files, 63.4111328125GB
 13%|███████████▎                                                                              | 63.2861328125/503.7639808654785 [00:42<04:37,  1.59it/s, num_files=64804]
[enforce fail at CPUAllocator.cpp:71] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 1048576 bytes. Error code 12 (Cannot allocate memory)
 13%|███████████▋                                                                                 | 63.3984375/503.7639808654785 [00:48<05:39,  1.30it/s, num_files=64919]
Result for file_system and count: 64920 files, 63.3984375GB
99%|██████████████████████████████████████████████████████████████████████████████████████████████████▊ | 498.0/503.7639808654785 [05:13<00:03,  1.59it/s, num_files=497]
Result for file_descriptor and size: 498 files, 498.0GB
 99%|██████████████████████████████████████████████████████████████████████████████████████████████████▊ | 498.0/503.7639808654785 [04:47<00:03,  1.73it/s, num_files=497]
Result for file_system and size: 498 files, 498.0GB
```

As one can see the issue is clearly not running out of physical RAM. When testing for the maximum shared memory I can allocate  reach 498Gb so no issues there. The only problem is the number of files.

The `ulimit` before running the script was:

```
$ ulimit -Sa
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 2063224
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2063224
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
```

Note that if I reduce `ulimit -Sn` then the number of file I can open with `file_descriptor` matches. But passed roughly 2**16 there is something else limiting it.

Another strange thing is that it seems to be completely `pytorch` dependent. Indeed running:
```
for i in $(seq 0 300000); do truncate -s 1M /dev/shm/file_$i; done
```
Works without any issues whatsoever.

## Environment

Machine with 512Gb of RAM. We have to feed 9 A100 with data. Reducing the number of workers is not an option. We really need to be able to create many shared memory tensors.

<details>

<summary>Status of `/dev/shm`</summary>

<p>

```
$ df -h /dev/shm/
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           500G  459G   42G  92% /dev/shm
```

</p>
</details>  

<details>

<summary>Output of ipcs limits</summary>

<p>

```
ipcs -l

------ Messages Limits --------
max queues system wide = 32000
max size of message (bytes) = 8192
default max size of queue (bytes) = 16384

------ Shared Memory Limits --------
max number of segments = 67108864
max seg size (kbytes) = 33554432
max total shared memory (kbytes) = 268435456
min seg size (bytes) = 1

------ Semaphore Limits --------
max number of arrays = 32000
max semaphores per array = 32000
max semaphores system wide = 1024000000
max ops per semop call = 500
semaphore max value = 32767
```
</p>

</details>  



<details>

<summary>Output of ipcs -m is surprisingly empty</summary>

<p>

```
$ sudo ipcs -m

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status  

```
</p>

</details>  




```
Collecting environment information...
PyTorch version: 1.9.0
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.27

Python version: 3.8 (64-bit runtime)
Python platform: Linux-4.15.0-144-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: A100-PCIE-40GB
GPU 1: A100-PCIE-40GB
GPU 2: A100-PCIE-40GB
GPU 3: A100-PCIE-40GB
GPU 4: A100-PCIE-40GB
GPU 5: A100-PCIE-40GB
GPU 6: A100-PCIE-40GB
GPU 7: A100-PCIE-40GB
GPU 8: A100-PCIE-40GB

Nvidia driver version: 460.80
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.2
[pip3] torch==1.9.0
[pip3] torchaudio==0.9.0a0+33b2469
[pip3] torchvision==0.10.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.2.0           h06a4308_296  
[conda] mkl-service               2.3.0            py38h27cfd23_1  
[conda] mkl_fft                   1.3.0            py38h42c9631_2  
[conda] mkl_random                1.2.1            py38ha9443f7_2  
[conda] numpy                     1.20.2           py38h2d18471_0  
[conda] numpy-base                1.20.2           py38hfae3a4d_0  
[conda] pytorch                   1.9.0           py3.8_cuda11.1_cudnn8.0.5_0    pytorch
[conda] torchaudio                0.9.0                      py38    pytorch
[conda] torchvision               0.10.0               py38_cu111    pytorch
```



cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @anjali411