Repository: pytorch
Issue: PyTorch(with mkldnn) in inference mode using non-optimal prop_kind for convolution
Labels: triaged,module: mkldnn

## 🐛 Bug

I'm using ResNet50 with PyTorch+OneDNN in inference mode and observed that "forward_training" prop is invoked for convolution primitives instead of "forward_inference". 
However, oneDNN user guide recommends framework to use "forward_inference" for inference scenarios for better performance.
https://oneapi-src.github.io/oneDNN/dev_guide_inference_and_training_aspects.html

## To Reproduce

Steps to reproduce the behavior:

```
git clone https://github.com/pytorch/pytorch.git
cd pytorch
BUILD_TEST=0 CMAKE_SHARED_LINKER_FLAGS=-Wl,-z,max-page-size=0x10000 USE_MKLDNN=ON python3 setup.py bdist_wheel
cd dist
pip3 install <torch-wheel-created-in-the-above-step>

git clone https://github.com/pytorch/benchmark.git
cd benchmark
python install.py --continue_on_fail
export DNNL_VERBOSE=1
python run.py resnet50 -d cpu -m jit -t eval --profile
```
Observed logs:
<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->
```


dnnl_verbose,info,prim_template:operation,engine,primitive,implementation,prop_kind,memory_descriptors,attributes,auxiliary,problem_desc,exec_time
dnnl_verbose,exec,cpu,convolution,gemm:ref,**forward_training**,src_f32::blocked:abcd:f0 wei_f32::blocked:abcd:f0 bia_undef::undef::f0 dst_f32::blocked:abcd:f0,scratchpad_mode:user;,alg:convolution_direct,mb32_ic3oc64_ih224oh112kh7sh2dh0ph3_iw224ow112kw7sw2dw0pw3,20.947
dnnl_verbose,exec,cpu,reorder,simple:any,undef,src_f32::blocked:abcd:f0 dst_f32::blocked:abcd:f0,,,32x64x112x112,4.79688
dnnl_verbose,exec,cpu,convolution,gemm:ref,forward_training,src_f32::blocked:abcd:f0 wei_f32::blocked:abcd:f0 bia_undef::undef::f0 dst_f32::blocked:abcd:f0,scratchpad_mode:user;,alg:convolution_direct,mb32_ic64oc64_ih56oh56kh1sh1dh0ph0_iw56ow56kw1sw1dw0pw0,2.46313
```

## Expected behavior

Expected logs:
<!-- A clear and concise description of what you expected to happen. -->
`
dnnl_verbose,exec,cpu,convolution,gemm:ref,**forward_inference**,src_f32::blocked:abcd:f0 wei_f32::blocked:abcd:f0 bia_undef::undef::f0 dst_f32::blocked:abcd:f0,scratchpad_mode:user;,alg:convolution_direct,mb32_ic3oc64_ih224oh112kh7sh2dh0ph3_iw224ow112kw7sw2dw0pw3
`
## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```

 - PyTorch Version (e.g., 1.0): v1.9.0
 - OS (e.g., Linux): Linux (Ubuntu 20.04)
 - How you installed PyTorch (`conda`, `pip`, source): source and pip has the same behavior
 - Build command you used (if compiling from source): BUILD_TEST=0 CMAKE_SHARED_LINKER_FLAGS=-Wl,-z,max-page-size=0x10000 USE_MKLDNN=ON python3 setup.py bdist_wheel
 - Python version: 3.8
 - CUDA/cuDNN version: Not applicable
 - GPU models and configuration: Not applicable
 - Any other relevant information: Same behavior observed on aarch64 as well as x86 builds

## Additional context

<!-- Add any other context about the problem here. -->


cc @gujinghui @PenghuiCheng @XiaobingSuper @jianyuh @VitalyFedyunin