Repository: pytorch
Issue: building torchscript extension results in INVALID TYPE: Only int64_t and bool are supported as an integral argument type  custom_class
Labels: oncall: jit

## 🐛 Bug

Trying to build a torchscript extension results in:
```
Severity	Code	Description	Project	File	Line	Suppression State
Error	C2338	INVALID TYPE: Only int64_t and bool are supported as an integral argument type	PythonManager	....\libtorch-release-latest\libtorch\include\ATen\core\op_registration\infer_schema.h	39	
```
error

## To Reproduce
here is a watered down toy example that demonstrates the issue. 
```cpp
#include <torch/script.h>
#include <string>
#include <vector>
struct PyManager : torch::CustomClassHolder {
    PyManager(std::string model_path, std::string img_bank_folder_root, std::string cache_folder,
        std::string pnet, std::string rnet, std::string onet, bool rebuildCache) 
    {
        std::cout << "ok!";
    }

    int AddUser(std::string user_id, std::string img_fldr_path) 
    {
        std::cout << user_id << img_fldr_path;
        return 0;
    }
    int RemoveUser(std::string user_id) 
    {
        std::cout << user_id ;
        return 0;
    }
    int UpdateUser(std::string user_id, std::string new_images_folder_path)
    {
        std::cout << user_id << new_images_folder_path;
        return 0;
    }
    int UpdateUsersInBulk(std::vector<std::string> userIDs, std::string ID_folders_path, bool is_delete_granted = true, bool show_dbg_info = true)
    {
        std::cout << userIDs.size() << ID_folders_path;
        return 0;
    }
    int RenameId(std::string old_id, std::string new_id)
    {
        std::cout << old_id << new_id;
        return 0;
    }

};

TORCH_LIBRARY(my_classes, m) {
    m.class_<PyManager>("PyManager")
        .def(torch::init<std::string, std::string, std::string, std::string, std::string, std::string, bool>())
        .def("add", &PyManager::AddUser)
        .def("remove", &PyManager::RemoveUser)
        .def("update", &PyManager::UpdateUser)
        .def("update_in_bulk", &PyManager::UpdateUsersInBulk);
}
```

## Expected behavior

Shouldnt be issuing this erro!

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
PS C:\Users\User> python .\co.env.py
Collecting environment information...
PyTorch version: 1.6.0+cpu
Is debug build: No
CUDA used to build PyTorch: None

OS: Microsoft Windows 10 Enterprise
GCC version: Could not collect
CMake version: version 3.17.2

Python version: 3.7
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip] facenet-pytorch==2.2.9
[pip] numpy==1.18.1
[pip] numpydoc==0.9.1
[pip] torch==1.6.0+cpu
[pip] torch-pruning==0.2.2
[pip] torchfile==0.1.0
[pip] torchnet==0.0.4
[pip] torchprof==1.0.0
[pip] torchscope==0.1.0
[pip] torchstat==0.0.7
[pip] torchvision==0.7.0+cpu
[conda] blas                      1.0                         mkl
[conda] facenet-pytorch           2.2.9                    pypi_0    pypi
[conda] mkl                       2019.0                   pypi_0    pypi
[conda] mkl-include               2020.1                      216
[conda] mkl-service               2.3.0            py37hb782905_0
[conda] mkl_fft                   1.0.15           py37h14836fe_0
[conda] mkl_random                1.1.1            py37h47e9c7a_0
[conda] numpy                     1.18.1           py37h93ca92e_0
[conda] numpy-base                1.18.1           py37hc3f5095_1
[conda] numpydoc                  0.9.1                      py_0
[conda] torch                     1.6.0+cpu                pypi_0    pypi
[conda] torch-pruning             0.2.2                    pypi_0    pypi
[conda] torchfile                 0.1.0                    pypi_0    pypi
[conda] torchnet                  0.0.4                    pypi_0    pypi
[conda] torchprof                 1.0.0                    pypi_0    pypi
[conda] torchscope                0.1.0                    pypi_0    pypi
[conda] torchstat                 0.0.7                    pypi_0    pypi
[conda] torchvision               0.4.2+cpu                pypi_0    pypi
```

 - PyTorch Version (e.g., 1.0): 1.6
 - OS (e.g., Linux): windows 10 1909 bbuild 18363
 - How you installed PyTorch (`conda`, `pip`, source): pip
 - Build command you used (if compiling from source): used the prebuilt libs /release
 - Python version: 3.7
 - CUDA/cuDNN version: - None
 - GPU models and configuration: None
 - Any other relevant information: using Visual Studiop 2019 enterprise /c++17, 

## Additional context

<!-- Add any other context about the problem here. -->


cc @gmagogsfm