Repository: pytorch
Issue: RuntimeError: xnn_status_success == run_status INTERNAL ASSERT FAILED at "/pytorch/aten/src/ATen/native/xnnpack/Linear.cpp":158
Labels: module: android,oncall: mobile,module: ios

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

## To Reproduce

Steps to reproduce the behavior:

1. use c++ to call the downloaded libtorch 1.9 on linux
2. turn off autograd and load model：
...
torch::autograd::AutoGradMode no_autograd_guard{false};
torch::AutoNonVariableTypeMode non_var_guard{true};
torch::jit::GraphOptimizerEnabledGuard no_optimizer_guard{false};
auto module = torch::jit::load(path);
module.eval();
...

3. the function doing forwarding:
torch::NoGradGuard no_grad;
...
std::vector<torch::jit::IValue> inputs;
torch::Tensor tensor_feat = torch::from_blob(feat_mat,{row, col}, torch::kFloat32);
inputs.emplace_back(tensor_feat);
at::Tensor output = module.forward(inputs).toTensor();


4. write a main function calling the above forwarding function to run forwarding on the same file, for 1000 times, with 40 threads on a 24 core/48 thread intel cpu, with 384GB memory; it will crash some times, with the same file. 
(sometimes 16 threads also reproduce the error)

The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
  File "code/__torch__/model/___torch_mangle_390.py", line 126, in forward
    input37 = torch.cat(_51, 1)
    _52 = getattr(self, "prepack_folding._jit_pass_packed_weight_0")
    _53 = ops.prepacked.linear_clamp_run(input37, _52)
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    return _53

Traceback of TorchScript, original code (most recent call last):
        %packed_weight_bias = prepacked::linear_clamp_prepack(
            %weight, %bias, %output_min_max, %output_min_max)
        %res = prepacked::linear_clamp_run(%input, %packed_weight_bias)
               ~~~~~~~~~ <--- HERE
        return (%res)
RuntimeError: xnn_status_success == run_statusINTERNAL ASSERT FAILED at "/pytorch/aten/src/ATen/native/xnnpack/Linear.cpp":158, please report a bug to PyTorch. xnn_run_operator failed!




<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```

 - PyTorch Version (e.g., 1.0): 1.9 
 - OS (e.g., Linux): linux
 - How you installed PyTorch (`conda`, `pip`, source):
 - Build command you used (if compiling from source):
 - Python version:
 - CUDA/cuDNN version:
 - GPU models and configuration:
 - Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


cc @gmagogsfm