Repository: pytorch
Issue: CUDA error: no kernel image is available for execution on the device When convert cpu tensor to gpu tensor
Labels: module: build,module: cuda,triaged

i build pytorch1.6 c++ code from source and get a libtorch lib, but a question occurs when i test the lib:

When i test Tesla P40,  all is  ok.
When i test Tesla T4, occur below:

[m@9-186-29-181 ~/Transformer-XL-cpp/build]$ ./example-app ../../models/final_transformer.pt
CUDA available! Training on GPU
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: no kernel image is available for execution on the device
Exception raised from launch_unrolled_kernel at /cfs/soft/pytorch1_6/pytorch/aten/src/ATen/native/cuda/CUDALoops.cuh:157 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x56 (0x7ff2f37bc8d6 in /usr/local/libtorch/lib/libc10.so)
frame #1: void at::native::gpu_kernel_impl<__nv_hdl_wrapper_t<false, true, __nv_dl_tag<void (*)(at::TensorIterator&, bool), &at::native::copy_device_to_device, 9u>, long (long)> >(at::TensorIterator&, __nv_hdl_wrapper_t<false, true, __nv_dl_tag<void (*)(at::TensorIterator&, bool), &at::native::copy_device_to_device, 9u>, long (long)> const&) + 0x1a41 (0x7ff2d7898071 in /usr/local/libtorch/lib/libtorch_cuda.so)
frame #2: void at::native::gpu_kernel<__nv_hdl_wrapper_t<false, true, __nv_dl_tag<void (*)(at::TensorIterator&, bool), &at::native::copy_device_to_device, 9u>, long (long)> >(at::TensorIterator&, __nv_hdl_wrapper_t<false, true, __nv_dl_tag<void (*)(at::TensorIterator&, bool), &at::native::copy_device_to_device, 9u>, long (long)> const&) + 0x137 (0x7ff2d7898677 in /usr/local/libtorch/lib/libtorch_cuda.so)
frame #3: at::native::copy_device_to_device(at::TensorIterator&, bool) + 0x977 (0x7ff2d787cb47 in /usr/local/libtorch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x1e7003c (0x7ff2d787e03c in /usr/local/libtorch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0x9b51ec (0x7ff2ea81f1ec in /usr/local/libtorch/lib/libtorch_cpu.so)
frame #6: <unknown function> + 0x9b25af (0x7ff2ea81c5af in /usr/local/libtorch/lib/libtorch_cpu.so)
frame #7: at::native::copy_(at::Tensor&, at::Tensor const&, bool) + 0x43 (0x7ff2ea81ddd3 in /usr/local/libtorch/lib/libtorch_cpu.so)
frame #8: at::Tensor::copy_(at::Tensor const&, bool) const + 0x100 (0x7ff2eaf504c0 in /usr/local/libtorch/lib/libtorch_cpu.so)
frame #9: <unknown function> + 0x33f1f74 (0x7ff2ed25bf74 in /usr/local/libtorch/lib/libtorch_cpu.so)
frame #10: at::Tensor::copy_(at::Tensor const&, bool) const + 0x100 (0x7ff2eaf504c0 in /usr/local/libtorch/lib/libtorch_cpu.so)
frame #11: at::native::to(at::Tensor const&, c10::ScalarType, bool, bool, c10::optional<c10::MemoryFormat>) + 0x2c9 (0x7ff2eaa84cc9 in /usr/local/libtorch/lib/libtorch_cpu.so)
frame #12: <unknown function> + 0x105ca76 (0x7ff2eaec6a76 in /usr/local/libtorch/lib/libtorch_cpu.so)
frame #13: <unknown function> + 0x2aeda86 (0x7ff2ec957a86 in /usr/local/libtorch/lib/libtorch_cpu.so)
frame #14: <unknown function> + 0x109ddbb (0x7ff2eaf07dbb in /usr/local/libtorch/lib/libtorch_cpu.so)
frame #15: at::Tensor::to(c10::ScalarType, bool, bool, c10::optional<c10::MemoryFormat>) const + 0x10d (0x7ff2eaf7474d in /usr/local/libtorch/lib/libtorch_cpu.so)
frame #16: main + 0x2a5 (0x42961e in ./example-app)
frame #17: __libc_start_main + 0xf5 (0x7ff2d50dab35 in /lib64/libc.so.6)
frame #18: ./example-app() [0x428a29] 

![image](https://user-images.githubusercontent.com/17922949/108706764-095c3680-754a-11eb-8fcb-0eee143dba0c.png)

the example-cpp.cpp is the same in two machine.
![image](https://user-images.githubusercontent.com/17922949/108706907-47595a80-754a-11eb-92aa-d81f149ccac7.png)


Hope to get help！

cc @malfet @seemethere @walterddr @ngimel