Repository: pytorch
Issue: code linking to libtorch cannot use thrust/cub functions
Labels: high priority,module: cpp,module: cuda,triaged

## 🐛 Bug
Hi, I use CPP torch fronted API  with thrust library (but independent). When I call torch function `mask_select` on tensor which is a store at GPU and mask which is store on GPU too (result is store on GPU), I get an error.

`terminate called after throwing an instance of 'thrust::system::system_error'
  what():  radix_sort: failed on 1st step: cudaErrorInvalidDevice: invalid device ordinal` 
which was called by `thrust::sort_by_key` function.

When I comment section with masking tensor, ```thrust::sort_by_keys``` works normally and fine. Or when I move mask tensor to CPU. But it works, I am not sure, if it a good approach to have a mask tensor on CPU and other tensors are store at GPU? (my second questions)

## To Reproduce
I create a small demo program, with the issue.
`main.cu`

```cpp
#include <thrust/execution_policy.h>
#include <torch/torch.h>
#include <iostream>
#include <thrust/host_vector.h>
#include <thrust/device_vector.h>
#include <thrust/sort.h>

int main() {
{
        std::vector<float> numbers = {15.1, 74.9, 16.2, 75.8, 17.3, 76.7};
        torch::Tensor t = torch::from_blob(numbers.data(), {static_cast<long>(numbers.size())},
                                           torch::TensorOptions().dtype(torch::kFloat32)).to(torch::kCUDA);
        torch::Tensor mask = (t < 20);
        std::cout << t.size(0) << " " << t.device() << "\n";
        std::cout << mask.size(0) << " " << mask.device() << "\n";
        auto gpu_result = t.masked_select(mask).view({-1});
        std::cout << gpu_result.size(0) << " " << gpu_result.device() << "\n";
        std::cout << gpu_result << "\n";
    }

    std::cout << "Thrust---\n";
    thrust::host_vector<int> keysOnHost(6);
   //fiill by
    keysOnHost[0] = 1;
    keysOnHost[1] = 4;
    keysOnHost[2] = 2;
    keysOnHost[3] = 8;
    keysOnHost[4] = 5;
    keysOnHost[5] = 7;
    thrust::host_vector<char> valuesOnHost(6);
    valuesOnHost[0] = 'a';
    valuesOnHost[1] = 'b';
    valuesOnHost[2] = 'c';
    valuesOnHost[3] = 'd';
    valuesOnHost[4] = 'e';
    valuesOnHost[5] = 'f';
    thrust::device_vector<int> keys = keysOnHost;
    thrust::device_vector<char> values = valuesOnHost;
    thrust::sort_by_key(keys.begin(), keys.end(), values.begin());
    thrust::copy(keys.begin(), keys.end(), keysOnHost.begin());
    thrust::copy(values.begin(), values.end(), valuesOnHost.begin());
    for (int i = 0; i < 6; ++i) {
        std::cout << keysOnHost[i] << " ";
    }
    std::cout << "\n";
    for (int i = 0; i < 6; ++i) {
        std::cout << valuesOnHost[i] << " ";
    }
    std::cout << "\n";
    std::cout << "END\n";
}
```

Steps to reproduce the behavior:
1. Code above reproduce bad behavior when call  `t.masked_select(mask)`  and after that call `thrust::sort_by_keys` function.
2. When I change row `torch::Tensor mask = (t < 20);` -> `torch::Tensor mask = (t < 20).to(torch::kCPU);` it works when another tensors are store at GPU. But I am not sure if, this approach could throw exception when i use different store tensors. 

## Expected behavior
My target is, that masking tensor on GPU, where the mask is stored at GPU, works the same as now, but not have an effect on the trust::sort_by_key function... 
## Environment

```
Collecting environment information...
PyTorch version: 1.7.1 for cuda 11.0
Is debug build: Yes
CUDA used to build PyTorch: Yes
ROCM used to build PyTorch: No

OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.17.5

Python version: 3.8 (64-bit runtime)
Is CUDA available: Yes, cuda:11.0

CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 2080 Ti
Nvidia driver version: 450.102.04
cuDNN version:  8.0.5.39
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.5
HIP runtime version: No
MIOpen runtime version: No

Versions of relevant libraries:
[pip] Could not collect
[conda] Could not collect
```
I installed the torch by the source to /usr/local.
I note that, I have only one GPU, when I run nvidia-smi command.
Thanks for the help.


cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @anjali411 @yf225 @glaringlee @ngimel