Repository: pytorch
Issue: Setting threads number to the number of default by torch.set_num_threads is faster than not setting it
Labels: triaged,module: multithreading

## 🐛 Bug

On CPU and Linux machines, setting the default number of threads is faster than not setting it. But in theory, it should be as fast.

After experimentation, it seems that this problem only occurs in the LSTM layer.

My computer has 8 cores.

This is my code

```
# coding=utf-8
import torch
import torch.nn as nn
import numpy as np
import random
import time

print(torch.cuda.is_available()) # False
print(torch.cuda.device_count()) # 0

class Net(nn.Module):
    def __init__(self, *args, **kwarge):
        super(Net, self).__init__()
        self.lstmLayers = nn.LSTM(510, 32, batch_first=True)
    def forward(self, input):
        x, _ = self.lstmLayers(input)
        return x

class Model():
    def __init__(self):
        super().__init__()
        self.net = Net()
        self.batches = torch.from_numpy(np.random.random((500,5,510))).float()
    
    def test(self):
       # torch.set_num_threads(8) # not setting the number of threads
        print(torch.get_num_threads()) # always print 8
        for _ in range(1000):
            t = time.time()
            self.net(self.batches)
            tt = time.time() - t
            print(tt) # 
        
m = Model()

tottime = time.time()
m.test()
print("tot: ",time.time() - tottime)

```

the time cost is

![image](https://user-images.githubusercontent.com/13199295/88501960-a0d2c880-cfff-11ea-8f99-98a9f6c5a542.png)

but when i change the test function to

```
def test(self):
       torch.set_num_threads(8) # set the number of threads to 8
       print(torch.get_num_threads()) # always print 8
       for _ in range(1000):
           t = time.time()
           self.net(self.batches)
           tt = time.time() - t
           print(tt) # 
```
the time cost is

![image](https://user-images.githubusercontent.com/13199295/88502007-c9f35900-cfff-11ea-9a0a-879c45ec5476.png)

I also tested it on my **32-core machine**, and similar problems also occur. Setting the number of threads will be faster than not setting it, but theoretically it should be as fast. Because they are all the same number of threads.

I am not sure if there is a problem with my code or the way I use torch.set_num_threads! Hope to get your reply, thank you!

**This problem does not appear on windows**.

## Other Experiment

On my 8 cores computer.

When the number of output of LSTM layer is lower then 19, the problem gones.
```
self.lstmLayers = nn.LSTM(510, 18, batch_first=True)
# torch.set_num_threads(8) # not setting
```
![image](https://user-images.githubusercontent.com/13199295/88502412-efcd2d80-d000-11ea-96a7-18328684cd9c.png)

but if i set to 19
```
self.lstmLayers = nn.LSTM(510, 19, batch_first=True)
# torch.set_num_threads(8) # not setting
```
the problem occurs
![image](https://user-images.githubusercontent.com/13199295/88502474-1be8ae80-d001-11ea-9f48-14bb41f9a28f.png)



## Environment

PyTorch version: 1.5.0
Is debug build: No
CUDA used to build PyTorch: 10.2

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: Could not collect

Python version: 3.7
Is CUDA available: No