Repository: pytorch
Issue: Multiprocessing hang and queue short circuiting
Labels: module: multiprocessing,triaged,module: deadlock

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->
Here is a short script which produces the behavior, launching two synchronized processes feeding a common queue which are then to consume the items on the queue before ending. First, it seems to result in hanging, which I believe is related to the previously reported bug #50669. Second, both processes seem to short circuit the queue in varying amounts, one typically immediately exiting under the assumption the queue has no items to be consumed, while the other afterwards consumes a majority of the items but eventually jumps to the end. I do understand that the queue size call is not generally meant to be an exact total, but when only a single process is left to consume items, I would expect the call to produce an exact number.

## To Reproduce
Note: I have introduced a sleep call for how long the processes should run before receiving ending signals which helps more clearly illustrate the problem for longer run times.

```
import numpy
import time

import torch
#import multiprocessing as mp
import torch.multiprocessing as mp

import queue

def producerproc(camenderqueue,saveimagequeue,localsyncqueuelist,myownsyncqueue,camindex):
    while True:
        for syncqueue in localsyncqueuelist:
            syncqueue.put(None)
        print("Cam " + str(camindex) + " is waiting for the go signal")
        myownsyncqueue.get(1)
        referencearray=numpy.full((2048,2048),255,dtype=numpy.uint8)
        saveimagequeue.put([3,time.time(),torch.from_numpy(referencearray[:]),camindex])
        #saveimagequeue.put([3,time.time(),referencearray[:],camindex])
        print("Cam " + str(camindex) + " put an image on the queue")
        try:
            camenderqueue.get_nowait()
            print("Cam " + str(camindex)+ " loop is closing")
            while True:
                try:
                    saveimagequeue.get_nowait()
                    print("Cam " + str(camindex) + " sees " +str(saveimagequeue.qsize()) + " items left on the queue")
                except queue.Empty:
                    break
            break
        except queue.Empty:
            pass
    print("Cam " + str(camindex) + " has reached the end")

def rotate(x, i):
    templist=x.copy()
    i %= len(templist)
    templist[:i] = reversed(templist[:i])
    templist[i:] = reversed(templist[i:])
    templist[:] = reversed(templist)
    return templist
    
if __name__ == '__main__':
    mp.freeze_support()

    saveimagequeue=mp.Queue()

    num_cameras = 2

    camenderqueuelist=[]
    for enderindex in range(num_cameras):
        camenderqueuelist.append(mp.Queue())
    
    syncqueuelist=[]
    for syncindex in range(num_cameras):
        syncqueuelist.append(mp.Queue())
        
    camhardlist=[]
    for camindex in range(num_cameras):
        camhardlist.append(mp.Process(target=producerproc,args=(camenderqueuelist[camindex],saveimagequeue,rotate(syncqueuelist,camindex)[1:],rotate(syncqueuelist,camindex)[0],camindex)))
        camhardlist[camindex].start()
        print("Started")

    time.sleep(4)

    i=0
    for camenderqueue in camenderqueuelist:
        camenderqueue.put([True])
        print("I sent an end signal to cam " + str(i))
        i=i+1
    for camhard in camhardlist:
        camhard.join()
    print("Camera has ended")
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
The processes should run to completion without hanging (again, I understand this part is likely still an open bug) and all queues should be properly read and consumed exactly.

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```

 - PyTorch Version (e.g., 1.0):
 - OS (e.g., Linux):
 - How you installed PyTorch (`conda`, `pip`, source):
 - Build command you used (if compiling from source):
 - Python version:
 - CUDA/cuDNN version:
 - GPU models and configuration:
 - Any other relevant information:

```
PyTorch version: 1.10.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.8.12 (default, Oct 12 2021, 03:01:40) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19043-SP0
Is CUDA available: True
CUDA runtime version: 11.5.50
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1060 3GB
Nvidia driver version: 496.49
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] torch==1.10.0
[pip3] torchvision==0.11.1
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.3.1               h59b6b97_2
[conda] mkl                       2021.3.0           haa95532_524
[conda] mkl-service               2.4.0            py38h2bbff1b_0
[conda] mkl_fft                   1.3.1            py38h277e83a_0
[conda] mkl_random                1.2.2            py38hf11a4ad_0
[conda] numpy                     1.21.2           py38hfca59bb_0
[conda] numpy-base                1.21.2           py38h0829f74_0
[conda] pytorch                   1.10.0          py3.8_cuda11.3_cudnn8_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchvision               0.11.1               py38_cu113    pytorch
```
## Additional context

<!-- Add any other context about the problem here. -->


cc @VitalyFedyunin