Repository: pytorch
Issue: empty_sparse shouldn't be called with memory layout but is
Labels: module: sparse,triaged,module: memory format,module: tensor creation

Memory format is completely ignored by sparse constructor. It would be good to specify an error if it's specified...

```
Tensor empty_sparse(IntArrayRef size, const TensorOptions& options, c10::optional<MemoryFormat> optional_memory_format) {
  TORCH_CHECK(!options.pinned_memory(), "Only dense CPU tensors can be pinned");
  //TORCH_CHECK(!optional_memory_format.has_value(), "Only dense CPU tensors can have memory format");
  return new_with_dims_sparse(size.size(), 0, size, options);
}
```

But you cannot because this is an override of `empty` and apparently in some situations the memory format is set:

```
Feb 19 16:06:29 ======================================================================
Feb 19 16:06:29 ERROR [0.046s]: test_EmbeddingBag_sparse (__main__.TestNN)
Feb 19 16:06:29 ----------------------------------------------------------------------
Feb 19 16:06:29 Traceback (most recent call last):
Feb 19 16:06:29   File "test_nn.py", line 8499, in <lambda>
Feb 19 16:06:29     add(test_name, lambda self, test=test: test(self))
Feb 19 16:06:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/common_nn.py", line 3664, in __call__
Feb 19 16:06:29     self._do_test(test_case, module, input)
Feb 19 16:06:29   File "test_nn.py", line 285, in _do_test
Feb 19 16:06:29     module.float()
Feb 19 16:06:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 334, in float
Feb 19 16:06:29     return self._apply(lambda t: t.float() if t.is_floating_point() else t)
Feb 19 16:06:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 236, in _apply
Feb 19 16:06:29     grad_applied = fn(param.grad)
Feb 19 16:06:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 334, in <lambda>
Feb 19 16:06:29     return self._apply(lambda t: t.float() if t.is_floating_point() else t)
Feb 19 16:06:29 RuntimeError: Only dense CPU tensors can have memory format
Feb 19 16:06:29 
Feb 19 16:06:29 ======================================================================
Feb 19 16:06:29 ERROR [0.040s]: test_Embedding_sparse (__main__.TestNN)
Feb 19 16:06:29 ----------------------------------------------------------------------
Feb 19 16:06:29 Traceback (most recent call last):
Feb 19 16:06:29   File "test_nn.py", line 8499, in <lambda>
Feb 19 16:06:29     add(test_name, lambda self, test=test: test(self))
Feb 19 16:06:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/common_nn.py", line 3664, in __call__
Feb 19 16:06:29     self._do_test(test_case, module, input)
Feb 19 16:06:29   File "test_nn.py", line 285, in _do_test
Feb 19 16:06:29     module.float()
Feb 19 16:06:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 334, in float
Feb 19 16:06:29     return self._apply(lambda t: t.float() if t.is_floating_point() else t)
Feb 19 16:06:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 236, in _apply
Feb 19 16:06:29     grad_applied = fn(param.grad)
Feb 19 16:06:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 334, in <lambda>
Feb 19 16:06:29     return self._apply(lambda t: t.float() if t.is_floating_point() else t)
Feb 19 16:06:29 RuntimeError: Only dense CPU tensors can have memory format
Feb 19 16:06:29 
```

This seems a bit fishy.

cc @vincentqb @aocsa @nikitaved @pearu @mruberry @VitalyFedyunin @jamesr66a @ppwwyyxx