Repository: pytorch
Issue: ASSERT(initialized()) Debug Error after JIT fusion on Windows
Labels: oncall: jit,module: nvfuser

### Debug Error Explanation
`Assertion failed: initialized(), file C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\c10/util/Optional.h, line 763`

The script runs perfectly in release mode, but crashes in an ASSERT in l. 763 of **Optional.h** during the 2nd or 3rd run in debug mode. More explicitly, in the function FusionExecutorCache::getKernelRuntimeFor in **kernel_cache.cpp**, the variable args from l. 231 args.cache_id->"optional_base".init_ is false and thus hits the above ASSERT(initialized()).

**kernel_cache.cpp**: \torch\csrc\jit\codegen\cuda\kernel_cache.cpp
**Optional.h**: \c10\util\Optional.h

It seems like for some reasons, the JIT fusers don't correctly initialize throughout the first run and thus create this error in one of the next runs. This explains why the error only occurs when using at least two operators as in the model below. Using just one of them (multiplication, division, addition, ...) does not generate any fusions and thus does not generate the error. The example below is a very short example that was found to be part of RAFT ([https://github.com/princeton-vl/RAFT](https://github.com/princeton-vl/RAFT)) which is the network that I actually want to run in the end.
Also, in release mode, the ASSERT will simply be skipped, but I am unsure about whether the fusers did a good job or if they actually hide a bug somewhere, so I don't think simply ignoring the debug error is a good solution.

I generate a very basic model in PyTorch 1.13.1 as follows:

```
import torch
class NeuralNetwork(torch.nn.Module):
    def forward(self, im):
        im = im / 255.0
        im = im * 2.0
        return im

if __name__ == "__main__":
        model = NeuralNetwork()

        # Tracing
        traced_model = torch.jit.trace(model, (torch.rand((1, 3, 184, 320))))
        torch.jit.save(traced_model, "model.pt")

        # Scripting
        scripted_model = torch.jit.script(model)
        torch.jit.save(scripted_model, "model.pt")
```
There is no difference (same error) between the traced and the scripted model.

I then try to run it in C++ using Libtorch 1.13.1 on a GPU (CUDA 11.7, cuDNN 8.6.0) on Windows as follows:

```
#include <torch/script.h>
#include <iostream>

int main()
{
    try
    {
        torch::NoGradGuard no_grad;

        torch::jit::Module model;
        torch::Device device = torch::Device(torch::kCUDA);

        model = torch::jit::load("model.pt");
        model.to(device);
        model.eval();

        std::vector<int64_t> dims = { 1, 3, 184, 320 };
        for (auto i = 0; i < 100; i++)
        {
            torch::Tensor t = torch::ones(dims).to(device);
            std::vector<torch::jit::IValue> inputs;
            inputs.push_back(t);

            auto modelOutput = m_model.forward(inputs);
            std::cout << i << std::endl;
        }
    }
    catch (const c10::Error e)
    {
        std::string str = std::string("Error: ") + std::string(e.what());
        return 1;
    }
    catch (const std::exception e)
    {
        std::string str = std::string("Error: ") + std::string(e.what());
        return 1;
    }    
     return 0;
}

```

### Versions

PyTorch 1.13.1 / LibTorch 1.13.1
CUDA 11.7.0
cuDNN 8.6.0
on Windows 10

_I don't think the results from the Python-only dependencies from your script are too important here, but there you are:_
PyTorch version: 1.13.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Enterprise
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.23.0-rc2
Libc version: N/A

Python version: 3.9.10 (tags/v3.9.10:f2f3f53, Jan 17 2022, 15:14:21) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19044-SP0
Is CUDA available: True
CUDA runtime version: 11.7.64
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA RTX A3000 Laptop GPU
Nvidia driver version: 516.01
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin\cudnn_ops_train64_8.dll
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True


cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel @kevinstephano @jjsjann123