Repository: pytorch
Issue: [bazel] [ci] `//:lazy_tests` Could not run 'aten::mul.Tensor' with arguments from the 'Lazy' backend
Labels: module: cuda,triaged,lazy,module: lazy,module: bazel

### 🐛 Describe the bug

I made a run of bazel in CI with GPU enabled and a long timeout.
PR: https://github.com/pytorch/pytorch/pull/79844

The logs contain all errors that CI hit https://github.com/pytorch/pytorch/runs/6955918249?check_suite_focus=true

If you search for Test output for //:lazy_tests you will find the following error

```
022-06-19T17:05:01.1555929Z [1A[K==================== Test output for //:lazy_tests:
2022-06-19T17:05:01.1556291Z Running main() from gmock_main.cc
2022-06-19T17:05:01.1556774Z [==========] Running 33 tests from 9 test suites.
2022-06-19T17:05:01.1557168Z [----------] Global test environment set-up.
2022-06-19T17:05:01.1557549Z [----------] 10 tests from BackendDeviceTest
2022-06-19T17:05:01.1557887Z [ RUN      ] BackendDeviceTest.BackendDeviceType
2022-06-19T17:05:01.1558414Z [       OK ] BackendDeviceTest.BackendDeviceType (0 ms)
2022-06-19T17:05:01.1559014Z [ RUN      ] BackendDeviceTest.Basic1
2022-06-19T17:05:01.1559325Z [       OK ] BackendDeviceTest.Basic1 (0 ms)
2022-06-19T17:05:01.1559646Z [ RUN      ] BackendDeviceTest.Basic2
2022-06-19T17:05:01.1560147Z [       OK ] BackendDeviceTest.Basic2 (0 ms)
2022-06-19T17:05:01.1560793Z [ RUN      ] BackendDeviceTest.Basic3
2022-06-19T17:05:01.1561473Z [       OK ] BackendDeviceTest.Basic3 (0 ms)
2022-06-19T17:05:01.1561902Z [ RUN      ] BackendDeviceTest.Compare
2022-06-19T17:05:01.1562260Z [       OK ] BackendDeviceTest.Compare (0 ms)
2022-06-19T17:05:01.1562756Z [ RUN      ] BackendDeviceTest.Ostream
2022-06-19T17:05:01.1563398Z [       OK ] BackendDeviceTest.Ostream (0 ms)
2022-06-19T17:05:01.1564041Z [ RUN      ] BackendDeviceTest.FromAten
2022-06-19T17:05:01.1564657Z [       OK ] BackendDeviceTest.FromAten (0 ms)
2022-06-19T17:05:01.1565255Z [ RUN      ] BackendDeviceTest.ToAten
2022-06-19T17:05:01.1565804Z [       OK ] BackendDeviceTest.ToAten (0 ms)
2022-06-19T17:05:01.1566379Z [ RUN      ] BackendDeviceTest.GetBackendDevice1
2022-06-19T17:05:01.1567030Z [       OK ] BackendDeviceTest.GetBackendDevice1 (18 ms)
2022-06-19T17:05:01.1567608Z [ RUN      ] BackendDeviceTest.GetBackendDevice2
2022-06-19T17:05:01.1567999Z [       OK ] BackendDeviceTest.GetBackendDevice2 (0 ms)
2022-06-19T17:05:01.1568483Z [----------] 10 tests from BackendDeviceTest (18 ms total)
2022-06-19T17:05:01.1568706Z 
2022-06-19T17:05:01.1568866Z [----------] 2 tests from CacheTest
2022-06-19T17:05:01.1569308Z [ RUN      ] CacheTest.BasicTest
2022-06-19T17:05:01.1569618Z [       OK ] CacheTest.BasicTest (0 ms)
2022-06-19T17:05:01.1569988Z [ RUN      ] CacheTest.ShapeCacheTestForDynamicShape
2022-06-19T17:05:01.1570392Z [       OK ] CacheTest.ShapeCacheTestForDynamicShape (0 ms)
2022-06-19T17:05:01.1570832Z [----------] 2 tests from CacheTest (0 ms total)
2022-06-19T17:05:01.1571032Z 
2022-06-19T17:05:01.1571215Z [----------] 2 tests from IrUtilTest
2022-06-19T17:05:01.1571489Z [ RUN      ] IrUtilTest.BasicTest
2022-06-19T17:05:01.1571793Z [       OK ] IrUtilTest.BasicTest (0 ms)
2022-06-19T17:05:01.1572099Z [ RUN      ] IrUtilTest.TestCircle
2022-06-19T17:05:01.1572392Z [       OK ] IrUtilTest.TestCircle (0 ms)
2022-06-19T17:05:01.1572781Z [----------] 2 tests from IrUtilTest (0 ms total)
2022-06-19T17:05:01.1572986Z 
2022-06-19T17:05:01.1573163Z [----------] 2 tests from HashTest
2022-06-19T17:05:01.1573425Z [ RUN      ] HashTest.Scalar
2022-06-19T17:05:01.1573705Z [       OK ] HashTest.Scalar (0 ms)
2022-06-19T17:05:01.1573996Z [ RUN      ] HashTest.Sanity
2022-06-19T17:05:01.1574280Z [       OK ] HashTest.Sanity (0 ms)
2022-06-19T17:05:01.1574621Z [----------] 2 tests from HashTest (0 ms total)
2022-06-19T17:05:01.1574819Z 
2022-06-19T17:05:01.1575034Z [----------] 3 tests from PermutationUtilTest
2022-06-19T17:05:01.1575414Z [ RUN      ] PermutationUtilTest.TestInversePermutation
2022-06-19T17:05:01.1575856Z [       OK ] PermutationUtilTest.TestInversePermutation (0 ms)
2022-06-19T17:05:01.1576265Z [ RUN      ] PermutationUtilTest.TestIsPermutation
2022-06-19T17:05:01.1576663Z [       OK ] PermutationUtilTest.TestIsPermutation (0 ms)
2022-06-19T17:05:01.1577047Z [ RUN      ] PermutationUtilTest.TestPermute
2022-06-19T17:05:01.1577400Z [       OK ] PermutationUtilTest.TestPermute (0 ms)
2022-06-19T17:05:01.1577847Z [----------] 3 tests from PermutationUtilTest (0 ms total)
2022-06-19T17:05:01.1578067Z 
2022-06-19T17:05:01.1578249Z [----------] 7 tests from ShapeTest
2022-06-19T17:05:01.1578520Z [ RUN      ] ShapeTest.Basic1
2022-06-19T17:05:01.1578802Z [       OK ] ShapeTest.Basic1 (0 ms)
2022-06-19T17:05:01.1579087Z [ RUN      ] ShapeTest.Basic2
2022-06-19T17:05:01.1579354Z [       OK ] ShapeTest.Basic2 (0 ms)
2022-06-19T17:05:01.1579621Z [ RUN      ] ShapeTest.Basic3
2022-06-19T17:05:01.1579901Z [       OK ] ShapeTest.Basic3 (0 ms)
2022-06-19T17:05:01.1580180Z [ RUN      ] ShapeTest.SetScalarType
2022-06-19T17:05:01.1580509Z [       OK ] ShapeTest.SetScalarType (0 ms)
2022-06-19T17:05:01.1580802Z [ RUN      ] ShapeTest.SetSize
2022-06-19T17:05:01.1581086Z [       OK ] ShapeTest.SetSize (0 ms)
2022-06-19T17:05:01.1581354Z [ RUN      ] ShapeTest.Equal
2022-06-19T17:05:01.1581633Z [       OK ] ShapeTest.Equal (0 ms)
2022-06-19T17:05:01.1581911Z [ RUN      ] ShapeTest.Ostream
2022-06-19T17:05:01.1582178Z [       OK ] ShapeTest.Ostream (0 ms)
2022-06-19T17:05:01.1582558Z [----------] 7 tests from ShapeTest (0 ms total)
2022-06-19T17:05:01.1582755Z 
2022-06-19T17:05:01.1582945Z [----------] 2 tests from LazyShapeTest
2022-06-19T17:05:01.1583246Z [ RUN      ] LazyShapeTest.TestMulBasic
2022-06-19T17:05:01.1583600Z unknown file: Failure
2022-06-19T17:05:01.1587689Z C++ exception with description "Could not run 'aten::mul.Tensor' with arguments from the 'Lazy' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::mul.Tensor' is only available for these backends: [Dense, FPGA, ORT, Metal, Quantized, CustomRNGKeyId, MkldnnCPU, Sparse, SparseCsrCUDA, NestedTensor, BackendSelect, Python, Fake, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, AutogradOther, AutogradFunctionality, AutogradNestedTensor, Tracer, AutocastCPU, Autocast, FuncTorchBatched, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, DeferredInit, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, TESTING_ONLY_GenericMode, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, CPU, CUDA, XLA, MPS, IPU, XPU, HPU, VE, Lazy, Meta, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, SparseCPU, SparseCUDA, SparseHIP, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, SparseXPU, UNKNOWN_TENSOR_TYPE_ID, SparseVE, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, NestedTensorCPU, NestedTensorCUDA, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID].
2022-06-19T17:05:01.1590281Z 
2022-06-19T17:05:01.1590789Z Undefined: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1591481Z CPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCPU.cpp:39984 [kernel]
2022-06-19T17:05:01.1592051Z CUDA: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCUDA.cpp:55699 [kernel]
2022-06-19T17:05:01.1592772Z HIP: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1593598Z MPS: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1594425Z IPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1595252Z XPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1596078Z HPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1596895Z VE: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1597569Z Meta: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterMeta.cpp:34244 [kernel]
2022-06-19T17:05:01.1598307Z PrivateUse1: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1599168Z PrivateUse2: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1600013Z PrivateUse3: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1600832Z FPGA: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1601757Z ORT: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1602616Z Vulkan: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1603458Z Metal: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1604287Z QuantizedCPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1605153Z QuantizedCUDA: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1606038Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1607019Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1607885Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1608760Z QuantizedXPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1609641Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1610515Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1611374Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1612259Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1613127Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1614001Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1614873Z CustomRNGKeyId: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1615573Z MkldnnCPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterMkldnnCPU.cpp:691 [kernel]
2022-06-19T17:05:01.1616191Z SparseCPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterSparseCPU.cpp:1859 [kernel]
2022-06-19T17:05:01.1616808Z SparseCUDA: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterSparseCUDA.cpp:2019 [kernel]
2022-06-19T17:05:01.1617564Z SparseHIP: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1618431Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1619312Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1620182Z SparseXPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1621132Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1621993Z SparseVE: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1622869Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1623747Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1624620Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1625472Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1626291Z SparseCsrCPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterSparseCsrCPU.cpp:1508 [kernel]
2022-06-19T17:05:01.1627359Z SparseCsrCUDA: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterSparseCsrCUDA.cpp:1658 [kernel]
2022-06-19T17:05:01.1628047Z NestedTensorCPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterNestedTensorCPU.cpp:386 [kernel]
2022-06-19T17:05:01.1628702Z NestedTensorCUDA: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterNestedTensorCUDA.cpp:466 [kernel]
2022-06-19T17:05:01.1629280Z BackendSelect: fallthrough registered at aten/src/ATen/core/BackendSelectFallbackKernel.cpp:3 [backend fallback]
2022-06-19T17:05:01.1629804Z Python: registered at aten/src/ATen/core/PythonFallbackKernel.cpp:133 [backend fallback]
2022-06-19T17:05:01.1630307Z Functionalize: registered at aten/src/ATen/FunctionalizeFallbackKernel.cpp:174 [backend fallback]
2022-06-19T17:05:01.1630794Z Named: fallthrough registered at aten/src/ATen/core/NamedRegistrations.cpp:11 [kernel]
2022-06-19T17:05:01.1631258Z Conjugate: registered at aten/src/ATen/ConjugateFallback.cpp:18 [backend fallback]
2022-06-19T17:05:01.1631712Z Negative: registered at aten/src/ATen/native/NegateFallback.cpp:18 [backend fallback]
2022-06-19T17:05:01.1632284Z ZeroTensor: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterZeroTensor.cpp:199 [kernel]
2022-06-19T17:05:01.1632827Z ADInplaceOrView: fallthrough registered at aten/src/ATen/core/VariableFallbackKernel.cpp:64 [backend fallback]
2022-06-19T17:05:01.1633509Z AutogradOther: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1634184Z AutogradCPU: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1634845Z AutogradCUDA: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1635557Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1636247Z AutogradXLA: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1636919Z AutogradMPS: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1637568Z AutogradIPU: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1638229Z AutogradXPU: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1638880Z AutogradHPU: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1639671Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1640367Z AutogradLazy: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1641039Z AutogradMeta: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1641728Z AutogradPrivateUse1: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1642440Z AutogradPrivateUse2: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1643122Z AutogradPrivateUse3: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1643778Z Tracer: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/TraceType_0.cpp:14370 [kernel]
2022-06-19T17:05:01.1644360Z AutocastCPU: fallthrough registered at aten/src/ATen/autocast_mode.cpp:482 [backend fallback]
2022-06-19T17:05:01.1644826Z Autocast: fallthrough registered at aten/src/ATen/autocast_mode.cpp:324 [backend fallback]
2022-06-19T17:05:01.1645259Z Batched: registered at aten/src/ATen/BatchingRegistrations.cpp:1068 [kernel]
2022-06-19T17:05:01.1645742Z VmapMode: fallthrough registered at aten/src/ATen/VmapModeRegistrations.cpp:33 [backend fallback]
2022-06-19T17:05:01.1646263Z PythonTLSSnapshot: registered at aten/src/ATen/core/PythonFallbackKernel.cpp:137 [backend fallback]
2022-06-19T17:05:01.1646551Z 
2022-06-19T17:05:01.1646812Z Exception raised from reportError at aten/src/ATen/core/dispatch/OperatorEntry.cpp:474 (most recent call first):
2022-06-19T17:05:01.1647222Z (no backtrace available)" thrown in the test body.
2022-06-19T17:05:01.1647572Z [  FAILED  ] LazyShapeTest.TestMulBasic (1 ms)
2022-06-19T17:05:01.1647913Z [ RUN      ] LazyShapeTest.TestCatBasic
2022-06-19T17:05:01.1648185Z unknown file: Failure
2022-06-19T17:05:01.1651720Z C++ exception with description "Could not run 'aten::cat' with arguments from the 'Lazy' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::cat' is only available for these backends: [Dense, FPGA, ORT, Metal, Quantized, CustomRNGKeyId, MkldnnCPU, Sparse, SparseCsrCUDA, NestedTensor, BackendSelect, Python, Fake, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, AutogradOther, AutogradFunctionality, AutogradNestedTensor, Tracer, AutocastCPU, Autocast, FuncTorchBatched, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, DeferredInit, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, TESTING_ONLY_GenericMode, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, CPU, CUDA, XLA, MPS, IPU, XPU, HPU, VE, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, SparseCPU, SparseCUDA, SparseHIP, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, SparseXPU, UNKNOWN_TENSOR_TYPE_ID, SparseVE, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, NestedTensorCPU, NestedTensorCUDA, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID].
2022-06-19T17:05:01.1654016Z 
2022-06-19T17:05:01.1654519Z Undefined: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1655206Z CPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCPU.cpp:39984 [kernel]
2022-06-19T17:05:01.1655832Z CUDA: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCUDA.cpp:55699 [kernel]
2022-06-19T17:05:01.1656547Z HIP: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1657383Z MPS: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1658213Z IPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1659038Z XPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1659845Z HPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1660740Z VE: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1661418Z Meta: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterMeta.cpp:34244 [kernel]
2022-06-19T17:05:01.1662150Z PrivateUse1: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1662995Z PrivateUse2: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1663850Z PrivateUse3: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1664692Z FPGA: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1665525Z ORT: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1666358Z Vulkan: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1667548Z Metal: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1668293Z QuantizedCPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterQuantizedCPU.cpp:1330 [kernel]
2022-06-19T17:05:01.1669065Z QuantizedCUDA: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1669948Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1670818Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1671690Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1672568Z QuantizedXPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1673452Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1674312Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1675191Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1676192Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1677103Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1677979Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1678836Z CustomRNGKeyId: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1679691Z MkldnnCPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1680405Z SparseCPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterSparseCPU.cpp:1859 [kernel]
2022-06-19T17:05:01.1681110Z SparseCUDA: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterSparseCUDA.cpp:2019 [kernel]
2022-06-19T17:05:01.1681844Z SparseHIP: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1682712Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1683592Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1684463Z SparseXPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1685321Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1686187Z SparseVE: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1687063Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1688002Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1688861Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1689735Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1690605Z SparseCsrCPU: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1691471Z SparseCsrCUDA: registered at bazel-out/k8-fastbuild/bin/aten/src/ATen/RegisterCompositeExplicitAutogradNonFunctional.cpp:26815 [default backend kernel]
2022-06-19T17:05:01.1692111Z BackendSelect: fallthrough registered at aten/src/ATen/core/BackendSelectFallbackKernel.cpp:3 [backend fallback]
2022-06-19T17:05:01.1692641Z Python: registered at aten/src/ATen/core/PythonFallbackKernel.cpp:133 [backend fallback]
2022-06-19T17:05:01.1693142Z Functionalize: registered at aten/src/ATen/FunctionalizeFallbackKernel.cpp:174 [backend fallback]
2022-06-19T17:05:01.1693635Z Named: fallthrough registered at aten/src/ATen/core/NamedRegistrations.cpp:11 [kernel]
2022-06-19T17:05:01.1694081Z Conjugate: registered at aten/src/ATen/ConjugateFallback.cpp:18 [backend fallback]
2022-06-19T17:05:01.1694536Z Negative: registered at aten/src/ATen/native/NegateFallback.cpp:18 [backend fallback]
2022-06-19T17:05:01.1695058Z ZeroTensor: registered at aten/src/ATen/ZeroTensorFallback.cpp:86 [backend fallback]
2022-06-19T17:05:01.1695587Z ADInplaceOrView: fallthrough registered at aten/src/ATen/core/VariableFallbackKernel.cpp:64 [backend fallback]
2022-06-19T17:05:01.1696267Z AutogradOther: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1696950Z AutogradCPU: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1697628Z AutogradCUDA: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1698325Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1699072Z AutogradXLA: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1699741Z AutogradMPS: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1700404Z AutogradIPU: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1701082Z AutogradXPU: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1701734Z AutogradHPU: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1702420Z UNKNOWN_TENSOR_TYPE_ID: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1703110Z AutogradLazy: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1703785Z AutogradMeta: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1704465Z AutogradPrivateUse1: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1705178Z AutogradPrivateUse2: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1705875Z AutogradPrivateUse3: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/VariableType_0.cpp:14136 [autograd kernel]
2022-06-19T17:05:01.1706534Z Tracer: registered at bazel-out/k8-fastbuild/bin/torch/csrc/autograd/generated/TraceType_0.cpp:14370 [kernel]
2022-06-19T17:05:01.1707378Z AutocastCPU: registered at aten/src/ATen/autocast_mode.cpp:486 [kernel]
2022-06-19T17:05:01.1707832Z Autocast: fallthrough registered at aten/src/ATen/autocast_mode.cpp:324 [backend fallback]
2022-06-19T17:05:01.1708295Z Batched: registered at aten/src/ATen/BatchingRegistrations.cpp:1068 [kernel]
2022-06-19T17:05:01.1708763Z VmapMode: fallthrough registered at aten/src/ATen/VmapModeRegistrations.cpp:33 [backend fallback]
2022-06-19T17:05:01.1709284Z PythonTLSSnapshot: registered at aten/src/ATen/core/PythonFallbackKernel.cpp:137 [backend fallback]
2022-06-19T17:05:01.1709573Z 
2022-06-19T17:05:01.1709834Z Exception raised from reportError at aten/src/ATen/core/dispatch/OperatorEntry.cpp:474 (most recent call first):
2022-06-19T17:05:01.1710263Z (no backtrace available)" thrown in the test body.
2022-06-19T17:05:01.1710599Z [  FAILED  ] LazyShapeTest.TestCatBasic (0 ms)
2022-06-19T17:05:01.1711033Z [----------] 2 tests from LazyShapeTest (1 ms total)
2022-06-19T17:05:01.1711239Z 
2022-06-19T17:05:01.1711430Z [----------] 2 tests from TrieCacheTest
2022-06-19T17:05:01.1711736Z [ RUN      ] TrieCacheTest.TestSinglePath
2022-06-19T17:05:01.1712087Z [       OK ] TrieCacheTest.TestSinglePath (0 ms)
2022-06-19T17:05:01.1712431Z [ RUN      ] TrieCacheTest.TestTwoPaths
2022-06-19T17:05:01.1712843Z [       OK ] TrieCacheTest.TestTwoPaths (0 ms)
2022-06-19T17:05:01.1713277Z [----------] 2 tests from TrieCacheTest (0 ms total)
2022-06-19T17:05:01.1713484Z 
2022-06-19T17:05:01.1713658Z [----------] 3 tests from UtilTest
2022-06-19T17:05:01.1713956Z [ RUN      ] UtilTest.ExceptionCleanup
2022-06-19T17:05:01.1714273Z [       OK ] UtilTest.ExceptionCleanup (0 ms)
2022-06-19T17:05:01.1714576Z [ RUN      ] UtilTest.MaybeRef
2022-06-19T17:05:01.1714862Z [       OK ] UtilTest.MaybeRef (0 ms)
2022-06-19T17:05:01.1715121Z [ RUN      ] UtilTest.Iota
2022-06-19T17:05:01.1715388Z [       OK ] UtilTest.Iota (0 ms)
2022-06-19T17:05:01.1715755Z [----------] 3 tests from UtilTest (0 ms total)
2022-06-19T17:05:01.1715948Z 
2022-06-19T17:05:01.1716140Z [----------] Global test environment tear-down
2022-06-19T17:05:01.1716473Z [==========] 33 tests from 9 test suites ran. (21 ms total)
2022-06-19T17:05:01.1716844Z [  PASSED  ] 31 tests.
2022-06-19T17:05:01.1717097Z [  FAILED  ] 2 tests, listed below:
2022-06-19T17:05:01.1717410Z [  FAILED  ] LazyShapeTest.TestMulBasic
2022-06-19T17:05:01.1717734Z [  FAILED  ] LazyShapeTest.TestCatBasic
2022-06-19T17:05:01.1717929Z 
2022-06-19T17:05:01.1718033Z  2 FAILED TESTS
```

This error repros locally for me.

### Versions

master 7d17e3b884c83a11b2453b8ab13f054fda160474

cc @ngimel