Repository: pytorch
Issue: torch::zeros is slow for small tensors (C++)
Labels: module: performance,module: cpp,triaged

## 🐛 Bug

For small 1d tensors (smaller than about 4000), setting each element to zero with an accessor is faster than using make_zeros.

Benchmarking code (uses https://github.com/google/benchmark):

```
#pragma clang diagnostic push
#pragma ide diagnostic ignored "cert-err58-cpp"

#include <benchmark/benchmark.h>
#include <torch/torch.h>

static void BM_AccessorZero2d(benchmark::State& state) {
  auto options = torch::TensorOptions().dtype(torch::kFloat32);

  for(auto _ : state) {
    auto empty = torch::empty({state.range(0), state.range(0)}, options);
    auto accessor = empty.accessor<float, 2>();
    for(int i = 0; i < state.range(0); ++i) {
      for(int j = 0; j < state.range(0); ++j) {
        accessor[i][j] = 0;
      }
    }
  }
}
BENCHMARK(BM_AccessorZero2d)->Range(1, 50000);

static void BM_MakeZeros2d(benchmark::State& state) {
  auto options = torch::TensorOptions().dtype(torch::kFloat32);

  for (auto _ : state) {
    torch::zeros({state.range(0), state.range(0)}, options);
  }
}
BENCHMARK(BM_MakeZeros2d)->Range(1, 50000);

static void BM_AccessorZero(benchmark::State& state) {
  auto options = torch::TensorOptions().dtype(torch::kFloat32);

  for (auto _ : state) {
    auto empty = torch::empty({state.range(0)}, options);
    auto accessor = empty.accessor<float, 1>();
    for(int i=0; i<state.range(0); ++i) {
      accessor[i] = 0;
    }
  }
}
BENCHMARK(BM_AccessorZero)->Range(1, 50000);

static void BM_MakeZeros(benchmark::State& state) {
  auto options = torch::TensorOptions().dtype(torch::kFloat32);

  for (auto _ : state) {
    torch::zeros({state.range(0)}, options);
  }
}
BENCHMARK(BM_MakeZeros)->Range(1, 50000);

BENCHMARK_MAIN();
#pragma clang diagnostic pop
```

And here are the results:
```
Results:
Run on (12 X 3600 MHz CPU s)
CPU Caches:
  L1 Data 32K (x6)
  L1 Instruction 32K (x6)
  L2 Unified 256K (x6)
  L3 Unified 15360K (x1)
Load Average: 1.03, 1.64, 2.00
------------------------------------------------------------------
Benchmark                        Time             CPU   Iterations
------------------------------------------------------------------
BM_AccessorZero2d/1            559 ns          559 ns      1221153
BM_AccessorZero2d/8            614 ns          614 ns      1154875
BM_AccessorZero2d/64          2404 ns         2404 ns       290513
BM_AccessorZero2d/512        88699 ns        88698 ns         7957
BM_AccessorZero2d/4096    27946856 ns     27947340 ns           25
BM_AccessorZero2d/32768 1799065774 ns   1799064893 ns            1
BM_AccessorZero2d/50000 4031161980 ns   4031154712 ns            1
BM_MakeZeros2d/1              1503 ns         1503 ns       466054
BM_MakeZeros2d/8              1492 ns         1492 ns       459766
BM_MakeZeros2d/64             1936 ns         1936 ns       358106
BM_MakeZeros2d/512           11142 ns        10945 ns        71892
BM_MakeZeros2d/4096        7404616 ns      7116580 ns           99
BM_MakeZeros2d/32768     388659886 ns    374048021 ns            2
BM_MakeZeros2d/50000     839755289 ns    818242038 ns            1
BM_AccessorZero/1              578 ns          578 ns      1220794
BM_AccessorZero/8              575 ns          575 ns      1186077
BM_AccessorZero/64             620 ns          620 ns      1113125
BM_AccessorZero/512            781 ns          781 ns       914935
BM_AccessorZero/4096          1853 ns         1853 ns       386530
BM_AccessorZero/32768        10718 ns        10719 ns        66042
BM_AccessorZero/50000        16424 ns        16424 ns        42340
BM_MakeZeros/1                1491 ns         1491 ns       451186
BM_MakeZeros/8                1458 ns         1458 ns       479606
BM_MakeZeros/64               1465 ns         1465 ns       473993
BM_MakeZeros/512              1539 ns         1539 ns       455206
BM_MakeZeros/4096             1933 ns         1933 ns       358675
BM_MakeZeros/32768            5305 ns         5305 ns       134382
BM_MakeZeros/50000            7662 ns         7662 ns        90926
```

As you can see, for small 1d tensors make_zero is sometime almost 3x slower than using an accessor. For 2d tensors, make_zeros is still slower for very small tensors.

## Environment
PyTorch version: 1.1.0a0+de582e2
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 18.04.2 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.14.0

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: GeForce GTX TITAN X
Nvidia driver version: 418.56
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.6.0.21
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.5.1

Versions of relevant libraries:
[pip3] numpy==1.13.3
[conda] _tflow_select             2.3.0                       mkl  
[conda] blas                      1.0                         mkl  
[conda] magma-cuda10              2.4.0                         1    cpbotha
[conda] mkl                       2019.3                      199  
[conda] mkl-include               2019.3                      199  
[conda] mkl_fft                   1.0.12           py37ha843d7b_0  
[conda] mkl_random                1.0.2            py37hd81dba3_0  
[conda] mkldnn                    0.16.1                        0    mingfeima
[conda] tensorflow                1.13.1          mkl_py37h54b294f_0  
[conda] tensorflow-base           1.13.1          mkl_py37h7ce6ba3_0  
[conda] torch                     1.1.0a0+de582e2          pypi_0    pypi