Repository: pytorch
Issue: Creating Torch tensors slows OpenCV video reading a lot
Labels: module: performance,triaged,module: multithreading

## 🐛 Bug

I am developing a system that samples frames from videos and runs them through a deep network.
When I profiled the code I noticed that OpenCV read videos much slower than expected. Compared to a naive program that only grabs frames and does nothing else, this program spends x2 more time  grabbing frames. 

I managed to figure out a simple condition to trigger this slowing: to create a torch tensor of size >= 100000 whenever I sampled a frame. Something like 
```python
magic_tensor = torch.zeros(100000)
```
I can use any shape (e.g. 100000x1, 320x320), and even any data type (e.g. uint8, float64).
The turning point is exactly 100000, as 99999 won't trigger the slowing. Making a larger tensor or creating more tensors don't deteriorate the speed further.

Creating such an array slows the all cv2.grab() function calls in a short period of time. If I call time.sleep(1) to sleep for 1 second after I create an array, subsequent video reading is not affected at all.

When this experiment is performed in the main thread, video reading takes roughly 1.4x time of the baseline. However, when it is performed in a new thread (this is my use case), the time is about 2.4x of the baseline.

## To Reproduce
The code below will perform three experiments to show the issue.
The `read_video_test(filename, use_magic_code)` function samples every 30th frame from a video.
When `use_magic_code` is `True`, after a frame retrieval of the magic tensor is created, which will slows the grabbing. In reality this is the place I send the images to the deep network.

The three experiments are as follows:
* Baseline. `read_video_test` is performed without creating torch tensor.
* Reproduction in the main thread. This slows the video reading by ~50%.
* Reproduction in a new thread. This slows  the video reading by ~150%.

The test video I use has size of 1920x1080 and is about 20s long. I can reproduce it on other videos too, including videos of different duration and of different resolution (e.g. 360p). Enabling or disabling CUDA seems has not influence on this issue.

```python
import threading
import time

import cv2
import torch

def read_video_test(filename, use_magic_code):
    cap = cv2.VideoCapture(filename)
    assert cap.isOpened()
    
    frame_idx = 0
    total_grab_time = 0
    total_time = time.time()
    while cap.isOpened():
      # --------- Measure the grabbing time ---------
      grab_time = time.time()
      ret = cap.grab()
      grab_time = time.time() - grab_time
      total_grab_time += grab_time
      # ---------------------------------
      if not ret:
        break
      if not frame_idx % 30: # decode every 30th frame
        ret, frame = cap.retrieve()
        if use_magic_code:
          # magic code:
          # trigger condition: size >= 100000, regardless of data type
          magic_tensor = torch.zeros(100000, dtype=torch.int8)
      frame_idx += 1
    total_time = time.time() - total_time
    
    print("total time: {:.4}, cap.grab() took {:.4}s".format(
      total_time, total_grab_time))

if __name__ == '__main__':
    
    test_vid = "data/VID_20190407_170747.mp4"
    
    # 1.0x time  Baseline
    print('Exp#1: baseline, torch code disabled')
    read_video_test(test_vid, False)
    print("------------------------------------")
    
    # 1.4x time
    print('Exp#2: main thread execution, torch code enabled')
    read_video_test(test_vid, True)
    print("------------------------------------")
    
    # 2.4x time
    print('Exp#3: new thread execution, torch code enabled')
    t = threading.Thread(
      target=read_video_test, args=(test_vid, True))
    t.start()
    t.join()
    print("------------------------------------")
```
## Result

```
Exp#1: baseline, torch code disabled
total time: 1.41, cap.grab() took 1.279s
------------------------------------
Exp#2: main thread execution, torch code enabled
total time: 1.963, cap.grab() took 1.841s
------------------------------------
Exp#3: new thread execution, torch code enabled
total time: 3.594, cap.grab() took 3.389s
------------------------------------
```

Ideally there should not be any difference

## Environment

PyTorch version: 1.2.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 16.04.6 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: Tesla P100-PCIE-16GB
Nvidia driver version: 410.48
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.4

Versions of relevant libraries:
opencv                    3.4.2            py37h6fd60c2_1
[pip] numpy==1.17.2
[pip] torch==1.2.0
[pip] torchsummary==1.5.1
[pip] torchvision==0.4.0a0
[conda] _pytorch_select           0.2                       gpu_0  
[conda] blas                      1.0                         mkl  
[conda] mkl                       2019.4                      243  
[conda] mkl-service               2.3.0            py37he904b0f_0  
[conda] mkl_fft                   1.0.14           py37ha843d7b_0  
[conda] mkl_random                1.1.0            py37hd6b4f25_0  
[conda] pytorch                   1.2.0           cuda100py37h938c94c_0  
[conda] torchsummary              1.5.1                    pypi_0    pypi
[conda] torchvision               0.4.0           cuda100py37hecfc37a_0

I am able to reproduce it on a google cloud server and my laptop.
Both have the same software configuration.

## Additional context

<!-- Add any other context about the problem here. -->


cc @VitalyFedyunin @ngimel