Repository: pytorch
Issue: Eigen in Caffe2 doesn't produce vectorized instructions
Labels: caffe2

Compiling with: `CFLAGS=-march=native python setup.py build develop`

Repro code
```
from caffe2.python import core, workspace

test_net = core.Net("layer_norm_test")
test_net.LayerNorm(["input"], ["output", "mean", "stddev"], epsilon=1e-5)

import numpy as np
workspace.FeedBlob('input', np.random.rand(20, 5, 10, 10).astype('f'))
workspace.CreateNet(test_net)
s = time.time()
workspace.RunNet(test_net.Name(), NITER)
t_caffe2 = time.time() - s
print('caffe2 time per iter', t_caffe2 / NITER * 1000)
```

Performance is 11x slower than internal build with BUCK

Verified that vector instructions are not generated with `perf`:

```
caffe2::LayerNormOp<caffe2::CPUContext>::DoRunWithType<float>
       │       nop
       │7d8:   mov    %rcx,%rax
       │       vmovss (%r8,%rcx,4),%xmm0
  1.45 │       cqto
  0.02 │       idiv   %rdi
 26.52 │       mov    %rcx,%rax
  0.15 │       add    %r10,%rdx
  0.03 │       vsubss (%r12,%rdx,4),%xmm0,%xmm0
  6.46 │       cqto
       │       idiv   %rsi
 29.93 │       add    %r11,%rdx
  0.50 │       vdivss 0x0(%r13,%rdx,4),%xmm0,%xmm0
 24.20 │       vmovss %xmm0,(%r9,%rcx,4)
  1.76 │       add    $0x1,%rcx
       │       cmp    %rcx,%rbx
       │     ↑ jg     7d8
       │810:   add    $0x1,%r14
```

Verified from checking `build/build.ninja` that the flags is passed to the file in question (and built on clean setup):
```
build caffe2/CMakeFiles/caffe2.dir/operators/layer_norm_op.cc.o: CXX_COMPILER__caffe2 ../caffe2/operators/layer_norm_op.cc || cmake_order_depends_target_caffe2
  DEFINES = -DCPUINFO_SUPPORTED_PLATFORM=1 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DNNP_CONVOLUTION_ONLY=0 -DNNP_INFERENCE_ONLY=0 -DONNX_NAMESPACE=onnx_torch -DUSE_GCC_ATOMICS=1 -D_FILE_OFFSET_BITS=64 -Dcaffe2_EXPORTS
  DEP_FILE = caffe2/CMakeFiles/caffe2.dir/operators/layer_norm_op.cc.o.d
  FLAGS = --std=c++11  -march=native -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -Wno-unused-but-set-variable -Wno-maybe-uninitialized -DHAVE_AVX_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O2  -fPIC   -DHAVE_GCC_GET_CPUID -DUSE_AVX -DUSE_AVX2 -DTH_HAVE_THREAD -fvisibility=hidden -DCAFFE2_BUILD_MAIN_LIB -O2 -std=gnu++11
  INCLUDES = -I../aten/src -I. -I../ -isystem third_party/gloo -isystem ../cmake/../third_party/gloo -I../third_party/protobuf/src -isystem ../cmake/../third_party/googletest/googletest/include -I../cmake/../third_party/benchmark/include -isystem ../cmake/../third_party/eigen -isystem /usr/include/python2.7 -isystem /data/users/dzhulgakov/pytorch-env/lib/python2.7/site-packages/numpy/core/include -isystem ../cmake/../third_party/pybind11/include -Icaffe2/contrib/aten -I../third_party/onnx -Ithird_party/onnx -Icaffe2/aten/src/TH -I../aten/src/TH -Icaffe2/aten/src -Iaten/src -I../aten/src/THNN -I../aten/src/THCUNN -I../aten/../third_party/catch/single_include -Icaffe2/aten/src/ATen -I../aten/src/ATen/.. -I../caffe2/core/nomnigraph/include -isystem include -I../c10/.. -I../third_party/NNPACK/include -I../third_party/pthreadpool/include -I../third_party/cpuinfo/include -I../third_party/FP16/include
  IN_ABS = /data/users/dzhulgakov/pytorch/caffe2/operators/layer_norm_op.cc
  OBJECT_DIR = caffe2/CMakeFiles/caffe2.dir
  OBJECT_FILE_DIR = caffe2/CMakeFiles/caffe2.dir/operators
  TARGET_COMPILE_PDB = caffe2/CMakeFiles/caffe2.dir/
  TARGET_PDB = lib/libcaffe2.pdb
```

Machine:
```
lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              24
On-line CPU(s) list: 0-23
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s):           24
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               94
Model name:          Intel Core Processor (Skylake)
Stepping:            3
CPU MHz:             2394.452
BogoMIPS:            4788.90
Virtualization:      VT-x
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            4096K
L3 cache:            16384K
NUMA node0 CPU(s):   0-23
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat
```

Discovered as part of #11867