Repository: pytorch
Issue: SIGILL, Illegal Instruction from libtorch_cpu.so when callling backward() function in a VM.
Labels: module: dependency bug,module: crash,module: cpu,triaged,module: mkldnn

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->
I try to run a FL project, which uses pytorch. After using some pytorch methods, it crashes when it calls the pytorch "backward()" function.
The code failed and crashed with `Illegal Instruction (Core Dumped)` in an Ubuntu VM, but the same code works on a Windows "normal"/Non-VM machine.

## To Reproduce

Steps to reproduce the behavior:
It's a little bit more complicated to reproduce the behavior, but simply put:

1. I have attempted to test the FL algorithm from: https://github.com/qub-blesson/FedAdapt (but first I had to fix some bugs/adjust some configs, etc. )
2. I set-up some Openstack VMs (1 Master/Server and 5 Clients/Edge devices).
3. I create a fresh python venv and installed packages. 


I used the following line for installing pytorch:
```
pip install torch==1.7.1+cpu torchvision==0.8.2+cpu torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html
```

Ran the FL_training algorithm.

4. Everything works until it reaches the function call _loss.backward()_
```
def train(self, trainloader):
		# Training start
		self.net.to(self.device)
		self.net.train()
		if self.split_layer == (config.model_len -1): # No offloading training
			for batch_idx, (inputs, targets) in enumerate(tqdm.tqdm(trainloader)): ###enumerate?
				inputs, targets = inputs.to(self.device), targets.to(self.device)
				self.optimizer.zero_grad()
				outputs = self.net(inputs)
				loss = self.criterion(outputs, targets)
				loss.backward()
				self.optimizer.step()

```
( More code samples below in _Additional Context_ section )

This results in the following error:

```
Start_FL_training_Client.sh: line 5: 76312 Illegal instruction     (core dumped) python FedAdapt_clientrun.py
```

5. When using gdb for debbuging this is the output:

```
Thread 1 "python" received signal SIGILL, Illegal instruction.
0x00007ffff1fdb0cd in void dnnl::impl::cpu::(anonymous namespace)::block_ker<float, true, false>(long, long, long, float cons                  t*, long, float const*, long, float*, long, float, float, float*, bool) ()
   from /home/ubuntu/diz/FedAdapt/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so


(gdb) backtrace
#0  0x00007ffff1fdb0cd in void dnnl::impl::cpu::(anonymous namespace)::block_ker<float, true, false>(long, long, long, float                   const*, long, float const*, long, float*, long, float, float, float*, bool) ()
   from /home/ubuntu/diz/FedAdapt/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#1  0x00007ffff1fdb2b1 in void dnnl::impl::cpu::(anonymous namespace)::gemm_ithr<float, true, false>(long, long, long, float,                   float const*, long, float const*, long, float, float*, long, bool, float*) ()
   from /home/ubuntu/diz/FedAdapt/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#2  0x00007ffff1fdf07e in dnnl_status_t dnnl::impl::cpu::ref_gemm<float>(char const*, char const*, long const*, long const*,                   long const*, float const*, float const*, long const*, float const*, long const*, float const*, float*, long const*, float con                  st*) () from /home/ubuntu/diz/FedAdapt/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#3  0x00007ffff1f31238 in dnnl::impl::cpu::gemm_convolution_bwd_weights_t::execute_backward_weights_ncsp(dnnl::impl::exec_ctx                  _t const&) const () from /home/ubuntu/diz/FedAdapt/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#4  0x00007ffff13d5e7e in dnnl_primitive::execute(dnnl::impl::exec_ctx_t&) const ()
   from /home/ubuntu/diz/FedAdapt/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#5  0x00007ffff13d612b in dnnl_primitive_execute ()
   from /home/ubuntu/diz/FedAdapt/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#6  0x00007fffe650acaa in dnnl::primitive::execute(dnnl::stream const&, std::unordered_map<int, dnnl::memory, std::hash<int>,                   std::equal_to<int>, std::allocator<std::pair<int const, dnnl::memory> > > const&) const ()
   from /home/ubuntu/diz/FedAdapt/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#7  0x00007fffe6514987 in void ideep::convolution_backward_weights::compute_impl<true>(ideep::tensor const&, ideep::tensor co                  nst&, std::vector<long, std::allocator<long> > const&, ideep::tensor&, ideep::tensor&, std::vector<long, std::allocator<long>                   > const&, std::vector<long, std::allocator<long> > const&, std::vector<long, std::allocator<long> > const&, std::vector<long                  , std::allocator<long> > const&, int, dnnl::memory::data_type, dnnl::algorithm, ideep::engine const&) ()
   from /home/ubuntu/diz/FedAdapt/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#8  0x00007fffe651060f in at::native::mkldnn_convolution_backward_weights(c10::ArrayRef<long>, at::Tensor const&, at::Tensor                   const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool) ()
   from /home/ubuntu/diz/FedAdapt/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#9  0x00007fffe67d69ae in at::TypeDefault::mkldnn_convolution_backward_weights(c10::ArrayRef<long>, at::Tensor const&, at::Te                  nsor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool) ()
   from /home/ubuntu/diz/FedAdapt/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#10 0x00007fffe6811596 in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<std::tup                  le<at::Tensor, at::Tensor> (*)(c10::ArrayRef<long>, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<                  long>, c10::ArrayRef<long>, long, bool), std::tuple<at::Tensor, at::Tensor>, c10::guts::typelist::typelist<c10::ArrayRef<long                  >, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool> >, std::t                  uple<at::Tensor, at::Tensor> (c10::ArrayRef<long>, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<l                  ong>, c10::ArrayRef<long>, long, bool)>::call(c10::OperatorKernel*, c10::ArrayRef<long>, at::Tensor const&, at::Tensor const&                  , c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool) ()
   from /home/ubuntu/diz/FedAdapt/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#11 0x00007fffe6757b9e in std::tuple<at::Tensor, at::Tensor> c10::Dispatcher::callWithDispatchKey<std::tuple<at::Tensor, at::                  Tensor>, c10::ArrayRef<long>, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<l                  ong>, long, bool>(c10::TypedOperatorHandle<std::tuple<at::Tensor, at::Tensor> (c10::ArrayRef<long>, at::Tensor const&, at::Te                  nsor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool)> const&, c10::DispatchKey, c10::Array                  Ref<long>, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool) c                  --Type <RET> for more, q to quit, c to continue without paging--Quit


```
<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

To work on a VM with less/limited CPU instruction sets (I.e without AVX instruction set), and without GPU support.
<!-- A clear and concise description of what you expected to happen. -->

## Environment

Collecting environment information...
PyTorch version: 1.7.1+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.10 (default, Sep 28 2021, 16:10:42)  [GCC 9.3.0] (64-bit runtime)
Python platform: Linux-5.4.0-77-generic-x86_64-with-glibc2.29
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A



 - How you installed PyTorch (`pip`): `pip install torch==1.7.1+cpu torchvision==0.8.2+cpu torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html`
 - The code is returning the error when running on Openstack Nova VM instances.
 - Tried running the code on a physical windows machine with AVX support, but no GPU and it worked.
 - Using `lscpu` to provide info of the Virtual CPU:
 ```
(venv) ubuntu@edge-device-1:~/diz/FedAdapt/FL_training$ lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   40 bits physical, 48 bits virtual
CPU(s):                          1
On-line CPU(s) list:             0
Thread(s) per core:              1
Core(s) per socket:              1
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       AuthenticAMD
CPU family:                      6
Model:                           6
Model name:                      QEMU Virtual CPU version 2.5+
Stepping:                        3
CPU MHz:                         2394.368
BogoMIPS:                        4788.73
Virtualization:                  AMD-V
L1d cache:                       64 KiB
L1i cache:                       64 KiB
L2 cache:                        512 KiB
L3 cache:                        16 MiB
NUMA node0 CPU(s):               0
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full AMD retpoline, STIBP disabled, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm 
                                 no pl cpuid pni cx16 hypervisor lahf_lm svm 3dnowprefetch vmmcall
```

 - Since the CPU doesn't support AVX instructions, I have also tried setting the following environment variables: `TH_NO_AVX=1 TH_NO_AVX2=1 ATEN_CPU_CAPABILITY=default`, as suggested in other issues found on the pytorch github.


## Additional context


It doesn't crash when the _DataLoader() batch_size_ is set to 1. But when the _batch_size_ is anything else, like 2 or 100 it returns the Illegal Instruction error.

I have also tried other small code samples (torch.randn() etc. )to make sure the pytorch works and everything was ok.

Aditional code samples:
```
#Class who has the "train" method.
class Client(Communicator):
	def __init__(self, index, ip_address, server_addr, server_port, datalen, model_name, split_layer):
		super(Client, self).__init__(index, ip_address)
		self.datalen = datalen
		self.device = 'cuda' if torch.cuda.is_available() else 'cpu'
		self.model_name = model_name
		self.uninet = utils.get_model('Unit', self.model_name, config.model_len-1, self.device, config.model_cfg)

		logger.info('Connecting to Server.')
		self.sock.connect((server_addr,server_port))


#How the "trainloader" variable is initialized
def get_local_dataloader(CLIENT_IDEX, cpu_count):
	indices = list(range(N))
	part_tr = indices[int((N/K) * CLIENT_IDEX) : int((N/K) * (CLIENT_IDEX+1))]

	transform_train = transforms.Compose([
	transforms.RandomCrop(32, padding=4),
	transforms.RandomHorizontalFlip(),
	transforms.ToTensor(),
	transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
])
	trainset = torchvision.datasets.CIFAR10(
		root=dataset_path, train=True, download=True, transform=transform_train)
	subset = Subset(trainset, part_tr)
	trainloader = DataLoader(
		subset, batch_size=B, shuffle=True, num_workers=cpu_count)

	classes = ('plane', 'car', 'bird', 'cat', 'deer',
		   'dog', 'frog', 'horse', 'ship', 'truck')
	return trainloader,classes


```
<!-- Add any other context about the problem here. -->

Thank you in advance for your time!

cc @VitalyFedyunin @gujinghui @PenghuiCheng @XiaobingSuper @jianyuh @ezyang @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7