Repository: pytorch
Issue: TorchScript Performance: 150x gap between TorchScript and Native Python
Labels: triage review,oncall: jit,triaged

## 🐛 Bug

There's a 150x gap in performance for TorchScript ops versus straight Python / C++.
Looping over 100K numbers takes 2+ seconds instead of 18ms or better. 
Please see the benchmarks here: https://github.com/divyekapoor/ml-op-benchmarks

## To Reproduce
https://github.com/divyekapoor/ml-op-benchmarks

Steps to reproduce the behavior:
1. Clone the repo
1. make torchbench

See related TensorFlow issue for context:
https://github.com/tensorflow/tensorflow/issues/34500

## Expected behavior

FizzBuzz Iteration Counts | 100000 |   |   |  
-- | -- | -- | -- | --
  | Raw Latency (ms) | Per Run Latency (usec) | Python Multiplier | C++ Multiplier
PyTorch Python | 4007 | 40.07 | 222.61 | 23851
PyTorch TorchScript Python (from Loaded TorchScript) | 2830 | 28.3 | **157.22** | 16845
PyTorch TorchScript C++ (Native) | 255 | 2.55 | **14.17** | 1518
PyTorch TorchScript C++ (Native + ATen Tensors) | 252 | 2.52 | **14.00** | 1500
Raw Python | 18 | 0.18 | 1.00 | 107
Raw C++ | 0.168 | 0.00168 | 0.01 | 1

Performance similar to raw Python is the expected behavior.

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```

```
$ python3 /tmp/collect_env.py
Collecting environment information...
PyTorch version: 1.3.0.post2
Is debug build: No
CUDA used to build PyTorch: None

OS: Mac OSX 10.14.6
GCC version: Could not collect
CMake version: version 3.15.5

Python version: 3.7
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip3] numpy==1.17.4
[pip3] torch==1.3.0.post2
[conda] Could not collect
```

 - PyTorch Version (e.g., 1.0): 1.3
 - OS (e.g., Linux): Mac OS X
 - How you installed PyTorch (`conda`, `pip`, source): pip
 - Build command you used (if compiling from source): NA
 - Python version: 3.7
 - CUDA/cuDNN version: NA
 - GPU models and configuration: NA
 - Any other relevant information: See performance tables and github repo.

## Additional context

Code: 
```python
class TorchFizzBuzz(torch.nn.Module):
    def __init__(self):
        super(TorchFizzBuzz, self).__init__()
        self.fizz = torch.tensor(0, requires_grad=False)
        self.buzz = torch.tensor(0, requires_grad=False)
        self.fizzbuzz = torch.tensor(0, requires_grad=False)

    def forward(self, n: torch.Tensor):
        i = torch.tensor(0, dtype=torch.int32, requires_grad=False)
        self.fizz = torch.zeros(1)
        self.buzz = torch.zeros(1)
        self.fizzbuzz = torch.zeros(1)
        while i < n:
            if i % 6 == 0:
                self.fizzbuzz += 1
            elif i % 3 == 0:
                self.buzz += 1
            elif i % 2 == 0:
                self.fizz += 1
            i += 1
        return torch.stack([self.fizz, self.buzz, self.fizzbuzz])
```


cc @suo