Repository: pytorch
Issue: [bazel] [ci] `//:module_test` CUDA error: CUDA driver version is insufficient for CUDA runtime version
Labels: module: cuda,triaged,module: bazel

### 🐛 Describe the bug

I made a run of bazel in CI with GPU enabled and a long timeout.
PR: https://github.com/pytorch/pytorch/pull/79844

The logs contain all errors that CI hit https://github.com/pytorch/pytorch/runs/6955918249?check_suite_focus=true

If you search for `Test output for //:module_test` you will find the following error

```
==================== Test output for //:module_test:
Running main() from gmock_main.cc
[==========] Running 58 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 58 tests from ModuleTest
[ RUN      ] ModuleTest.CanEnableAndDisableTrainingMode
[       OK ] ModuleTest.CanEnableAndDisableTrainingMode (21 ms)
[ RUN      ] ModuleTest.ZeroGrad
[       OK ] ModuleTest.ZeroGrad (3 ms)
[ RUN      ] ModuleTest.ZeroGradWithUndefined
[       OK ] ModuleTest.ZeroGradWithUndefined (1 ms)
[ RUN      ] ModuleTest.RegisterModuleThrowsForEmptyOrDottedName
[       OK ] ModuleTest.RegisterModuleThrowsForEmptyOrDottedName (0 ms)
[ RUN      ] ModuleTest.RegisterModuleThrowsForDuplicateModuleName
[       OK ] ModuleTest.RegisterModuleThrowsForDuplicateModuleName (0 ms)
[ RUN      ] ModuleTest.ReplaceModuleThrowsForUnknownModuleName
[       OK ] ModuleTest.ReplaceModuleThrowsForUnknownModuleName (0 ms)
[ RUN      ] ModuleTest.ReplaceModule
[       OK ] ModuleTest.ReplaceModule (0 ms)
[ RUN      ] ModuleTest.UnregisterModule
[       OK ] ModuleTest.UnregisterModule (0 ms)
[ RUN      ] ModuleTest.RegisterParameterThrowsForEmptyOrDottedName
[       OK ] ModuleTest.RegisterParameterThrowsForEmptyOrDottedName (0 ms)
[ RUN      ] ModuleTest.RegisterParameterThrowsForDuplicateModuleName
[       OK ] ModuleTest.RegisterParameterThrowsForDuplicateModuleName (0 ms)
[ RUN      ] ModuleTest.RegisterParameterUndefinedTensor
[       OK ] ModuleTest.RegisterParameterUndefinedTensor (0 ms)
[ RUN      ] ModuleTest.RegisterBufferThrowsForEmptyOrDottedName
[       OK ] ModuleTest.RegisterBufferThrowsForEmptyOrDottedName (0 ms)
[ RUN      ] ModuleTest.RegisterBufferThrowsForDuplicateModuleName
[       OK ] ModuleTest.RegisterBufferThrowsForDuplicateModuleName (0 ms)
[ RUN      ] ModuleTest.CanGetName
[       OK ] ModuleTest.CanGetName (0 ms)
[ RUN      ] ModuleTest.AsCastsModulesCorrectly
[       OK ] ModuleTest.AsCastsModulesCorrectly (0 ms)
[ RUN      ] ModuleTest.DeviceOrDtypeConversionSkipsUndefinedTensor
[       OK ] ModuleTest.DeviceOrDtypeConversionSkipsUndefinedTensor (0 ms)
[ RUN      ] ModuleTest.DeviceOrDtypeConversionSkipsUndefinedTensor_CUDA
unknown file: Failure
C++ exception with description "CUDA error: CUDA driver version is insufficient for CUDA runtime version
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Exception raised from getDevice at ./c10/cuda/impl/CUDAGuardImpl.h:39 (most recent call first):
(no backtrace available)" thrown in the test body.
[  FAILED  ] ModuleTest.DeviceOrDtypeConversionSkipsUndefinedTensor_CUDA (0 ms)
[ RUN      ] ModuleTest.ParametersAndBuffersAccessorSkipsUndefinedTensor
[       OK ] ModuleTest.ParametersAndBuffersAccessorSkipsUndefinedTensor (0 ms)
[ RUN      ] ModuleTest.Conversion_MultiCUDA
unknown file: Failure
C++ exception with description "Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx
Exception raised from device_count_impl at c10/cuda/CUDAFunctions.cpp:44 (most recent call first):
(no backtrace available)" thrown in the test body.
[  FAILED  ] ModuleTest.Conversion_MultiCUDA (1 ms)
[ RUN      ] ModuleTest.CallingCloneOnModuleThatDoesNotOverrideCloneThrows
[       OK ] ModuleTest.CallingCloneOnModuleThatDoesNotOverrideCloneThrows (0 ms)
[ RUN      ] ModuleTest.CallingCloneOnModuleThatDoesOverrideCloneDoesNotThrow
[       OK ] ModuleTest.CallingCloneOnModuleThatDoesOverrideCloneDoesNotThrow (0 ms)
[ RUN      ] ModuleTest.CloneCreatesDistinctParameters
[       OK ] ModuleTest.CloneCreatesDistinctParameters (15 ms)
[ RUN      ] ModuleTest.CloneCreatesDistinctParametersExplicitDevice_CUDA
-- Test timed out at 2022-06-19 17:03:11 UTC --
```

Despite the timeout at the end, the test fails earlier.

This is strange (how the driver version works out for the Cmake build in this case?) but I seen the same error when running on RBE too. I don't see this error when I run locally on driver `470.82.01` and CUDA `11.2`.

### Versions

master `7d17e3b884c83a11b2453b8ab13f054fda160474`

cc @ngimel