Repository: pytorch
Issue: Quantisation results in a "Illegal instruction (core dumped)"
Labels: high priority,triage review,module: crash,oncall: quantization

### 🐛 Describe the bug

I have tried to follow this tutorial: https://pytorch.org/tutorials/recipes/recipes/dynamic_quantization.html .

I have created a fresh venv (Python 3.11.6) and installed torch via pip, output of `pip freeze` here:
```
filelock==3.13.1
fsspec==2023.12.2
Jinja2==3.1.2
MarkupSafe==2.1.3
mpmath==1.3.0
networkx==3.2.1
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==8.9.2.26
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.18.1
nvidia-nvjitlink-cu12==12.3.101
nvidia-nvtx-cu12==12.1.105
sympy==1.12
torch==2.1.1
triton==2.1.0
typing_extensions==4.9.0
```

Then, I run:
```
import torch
import torch.quantization
import torch.nn as nn
import time


class lstm_for_demonstration(nn.Module):
    def __init__(self, in_dim, out_dim, depth):
        super(lstm_for_demonstration, self).__init__()
        self.lstm = nn.LSTM(in_dim, out_dim, depth)

    def forward(self, inputs, hidden):
        out, hidden = self.lstm(inputs, hidden)
        return out, hidden


if __name__ == "__main__":
    torch.manual_seed(29592)  # set the seed for reproducibility
    model_dimension = 80
    sequence_length = 30
    batch_size = 1
    lstm_depth = 5
    inputs = torch.randn(sequence_length, batch_size, model_dimension)
    hidden = (
        torch.randn(lstm_depth, batch_size, model_dimension),
        torch.randn(lstm_depth, batch_size, model_dimension),
    )
    float_lstm = lstm_for_demonstration(model_dimension, model_dimension, lstm_depth)
    quantized_lstm = torch.quantization.quantize_dynamic(
        float_lstm, {nn.LSTM, nn.Linear}, dtype=torch.qint8
    )

    quantized_lstm.forward(inputs, hidden)
```
and I get:
```
xxx/venv/lib64/python3.11/site-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
  device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),
Illegal instruction (core dumped)
```

Output of `lscpu`:
```
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         36 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  4
  On-line CPU(s) list:   0-3
Vendor ID:               GenuineIntel
  BIOS Vendor ID:        Intel
  Model name:            Intel(R) Core(TM) i5-3470S CPU @ 2.90GHz
    BIOS Model name:     Intel(R) Core(TM) i5-3470S CPU @ 2.90GHz  CPU @ 2.9GHz
```

Note that if I follow exactly the same procedure on a different machine, then it works (also Intel x86_64 cpu)... Any idea what could be the problem?

### Versions

Python 3.11.6
torch 2.1.1

cc @ezyang @gchanan @zou3519 @kadeng @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel