Repository: pytorch
Issue: Floating point exception in mkl_vml_serv_GetMinN () on a specific computer
Labels: high priority,needs reproduction,module: binaries,module: crash,triaged,module: mkl

## 🐛 Bug

Hey! We bought a new server at our research institute and every complex PyTorch-based library that we want to run crashes with floating point exceptions, without traceback or anything.

## To Reproduce

Steps to reproduce the behavior:

1. pip install torchaudio speechbrain
2. git clone https://github.com/speechbrain/speechbrain.git
3. cd speechbrain/tests/integration/neural_networks/ASR_CTC/
4. python example_asr_ctc_experiment.py hyperparams.yaml 

The output of the command on the specific computer:

`[1]    3135660 floating point exception (core dumped)  python example_asr_ctc_experiment.py hyperparams.yaml`

And if I enter: `gdb --args python example_asr_ctc_experiment.py hyperparams.yaml`, run it, then backtrace it:

```
Thread 1 "python" received signal SIGFPE, Arithmetic exception.                                                                     [205/1898]
0x00007fffe61aac77 in mkl_vml_serv_GetMinN ()
   from /home/levaid/.miniconda3/envs/sb_debug/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
(gdb) backtrace
#0  0x00007fffe61aac77 in mkl_vml_serv_GetMinN ()
   from /home/levaid/.miniconda3/envs/sb_debug/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#1  0x00007fffe61a651d in mkl_vml_serv_threader_s_1i_1o ()
   from /home/levaid/.miniconda3/envs/sb_debug/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#2  0x00007fffe616dad8 in vmsCos () from /home/levaid/.miniconda3/envs/sb_debug/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#3  0x00007fffe5349b7d in at::native::AVX2::cos_kernel(at::TensorIteratorBase&)::{lambda()#1}::operator()() const::{lambda()#4}::operator()() 
const::{lambda(char**, long const*, long)#1}::operator()({lambda()#4}, char**, long) const [clone .isra.4282] ()
   from /home/levaid/.miniconda3/envs/sb_debug/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#4  0x00007fffe540b8c8 in void c10::function_ref<void (char**, long const*, long, long)>::callback_fn<at::TensorIteratorBase::loop_2d_from_1d<
at::native::AVX2::cos_kernel(at::TensorIteratorBase&)::{lambda()#1}::operator()() const::{lambda()#4}::operator()() const::{lambda(char**, lon
g const*, long)#1}>(at::native::AVX2::cos_kernel(at::TensorIteratorBase&)::{lambda()#1}::operator()() const::{lambda()#4}::operator()() const:
:{lambda(char**, long const*, long)#1} const&)::{lambda(char**, long const*, long, long)#1}>(long, char**, long const*, long, long) ()
   from /home/levaid/.miniconda3/envs/sb_debug/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#5  0x00007fffe1ea8837 in at::TensorIteratorBase::serial_for_each(c10::function_ref<void (char**, long const*, long, long)>, at::Range) const
    () from /home/levaid/.miniconda3/envs/sb_debug/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#6  0x00007fffe5382dd7 in at::native::AVX2::cos_kernel(at::TensorIteratorBase&) ()
   from /home/levaid/.miniconda3/envs/sb_debug/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#7  0x00007fffe2a2a608 in at::(anonymous namespace)::wrapper_cos_(at::Tensor&) ()
   from /home/levaid/.miniconda3/envs/sb_debug/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#8  0x00007fffe2cdf018 in at::Tensor::cos_() const ()
   from /home/levaid/.miniconda3/envs/sb_debug/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#9  0x00007fffe22a6bae in at::native::hamming_window(long, bool, double, double, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c
10::optional<c10::Device>, c10::optional<bool>) ()
   from /home/levaid/.miniconda3/envs/sb_debug/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#10 0x00007fffe22a6cfe in at::native::hamming_window(long, bool, double, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::opti
onal<c10::Device>, c10::optional<bool>) () from /home/levaid/.miniconda3/envs/sb_debug/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#11 0x00007fffe22a6d2e in at::native::hamming_window(long, bool, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10
::Device>, c10::optional<bool>) () from /home/levaid/.miniconda3/envs/sb_debug/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#12 0x00007fffe22a6d5c in at::native::hamming_window(long, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Devi
ce>, c10::optional<bool>) () from /home/levaid/.miniconda3/envs/sb_debug/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#13 0x00007fffe2b7216e in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<
at::Tensor (long, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>), &at::(anonymou
s namespace)::(anonymous namespace)::wrapper_hamming_window>, at::Tensor, c10::guts::typelist::typelist<long, c10::optional<c10::ScalarType>, 
c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool> > >, at::Tensor (long, c10::optional<c10::ScalarType>, c10::option
al<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>)>::call(c10::OperatorKernel*, c10::DispatchKeySet, long, c10::optional<c10::S
calarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) ()
   from /home/levaid/.miniconda3/envs/sb_debug/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#14 0x00007fffe29ce58a in at::Tensor c10::Dispatcher::redispatch<at::Tensor, long, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>,
 c10::optional<c10::Device>, c10::optional<bool> >(c10::TypedOperatorHandle<at::Tensor (long, c10::optional<c10::ScalarType>, c10::optional<c1
0::Layout>, c10::optional<c10::Device>, c10::optional<bool>)> const&, c10::DispatchKeySet, long, c10::optional<c10::ScalarType>, c10::optional
<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) const ()
   from /home/levaid/.miniconda3/envs/sb_debug/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#15 0x00007fffe29da140 in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<
at::Tensor (long, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>), &at::(anonymou
s namespace)::hamming_window>, at::Tensor, c10::guts::typelist::typelist<long, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10
::optional<c10::Device>, c10::optional<bool> > >, at::Tensor (long, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<
c10::Device>, c10::optional<bool>)>::call(c10::OperatorKernel*, c10::DispatchKeySet, long, c10::optional<c10::ScalarType>, c10::optional<c10::
Layout>, c10::optional<c10::Device>, c10::optional<bool>) ()
   from /home/levaid/.miniconda3/envs/sb_debug/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#16 0x00007fffe256eb85 in at::hamming_window(long, c10::TensorOptions) ()
   from /home/levaid/.miniconda3/envs/sb_debug/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so
#17 0x00007ffff4ef0d68 in torch::autograd::THPVariable_hamming_window(_object*, _object*, _object*) ()
--Type <RET> for more, q to quit, c to continue without paging--
   from /home/levaid/.miniconda3/envs/sb_debug/lib/python3.7/site-packages/torch/lib/libtorch_python.so
#18 0x000055555569dac4 in _PyMethodDef_RawFastCallKeywords (method=0x7ffff5eef6e0 <torch::autograd::torch_functions+14016>, self=0x0, 
    args=0x5555583530b8, nargs=<optimized out>, kwnames=<optimized out>) at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:693
#19 0x000055555571c42e in _PyCFunction_FastCallKeywords (kwnames=0x0, nargs=1, args=0x5555583530b8, func=0x7fff9e36cbe0)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:732
#20 call_function (kwnames=0x0, oparg=1, pp_stack=<synthetic pointer>) at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:4568
#21 _PyEval_EvalFrameDefault (f=0x555558352ef0, throwflag=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3124
#22 0x000055555566d0de in PyEval_EvalFrameEx (throwflag=0, f=0x555558352ef0)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:547
#23 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=<optimized out>, kwnames=<optimized out>, kwargs=0x7ffef98b5a70, kwcount=<optimized out>, kwstep=2, defs=0x7ffef98c4368, 
    defcount=8, kwdefs=0x0, closure=0x7ffef98c3f50, name=0x7ffff76964f0, qualname=0x7ffef98b93b0)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3930
#24 0x000055555566e4ae in _PyFunction_FastCallDict (kwargs=<optimized out>, nargs=<optimized out>, args=0x7fffffffbcc0, func=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:376
#25 _PyObject_FastCallDict (callable=<optimized out>, args=0x7fffffffbcc0, nargs=<optimized out>, kwargs=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:98
#26 0x00005555556dabef in _PyObject_Call_Prepend (kwargs=0x7ffef98ba230, args=0x7ffff7696050, obj=<optimized out>, callable=0x7ffef98c75f0)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:906
#27 slot_tp_init (self=<optimized out>, args=0x7ffff7696050, kwds=0x7ffef98ba230)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/typeobject.c:6636
#28 0x00005555556d3c7c in type_call (kwds=<optimized out>, args=0x7ffff7696050, type=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/typeobject.c:971
#29 _PyObject_FastCallKeywords (callable=0x5555588484f0, stack=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:199
#30 0x00005555556d42d1 in call_function (pp_stack=0x7fffffffbe60, oparg=<optimized out>, kwnames=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:4619
#31 0x00005555557181bc in _PyEval_EvalFrameDefault (f=0x555558852c60, throwflag=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3139
#32 0x000055555566d0de in PyEval_EvalFrameEx (throwflag=0, f=0x555558852c60)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:547
#33 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>,                     
    argcount=<optimized out>, kwnames=<optimized out>, kwargs=0x0, kwcount=<optimized out>, kwstep=2, defs=0x7ffef9909f68, defcount=16,       
    kwdefs=0x0, closure=0x7ffef98b9850, name=0x7ffff76964f0, qualname=0x7ffef98b91b0)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3930
#34 0x000055555566e202 in _PyFunction_FastCallDict (kwargs=<optimized out>, nargs=1, args=0x7fffffffc0b0, func=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:376
#35 _PyObject_FastCallDict (callable=<optimized out>, args=0x7fffffffc0b0, nargs=1, kwargs=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:98
#36 0x00005555556dabef in _PyObject_Call_Prepend (kwargs=0x7ffef9908dc0, args=0x7ffff7696050, obj=<optimized out>, callable=0x7ffef98c0680)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:906
#37 slot_tp_init (self=<optimized out>, args=0x7ffff7696050, kwds=0x7ffef9908dc0)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/typeobject.c:6636
#38 0x000055555568e8e7 in type_call (type=<optimized out>, args=0x7ffff7696050, kwds=0x7ffef9908dc0)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/typeobject.c:971
#39 0x000055555566e7b6 in PyObject_Call (callable=0x555558851e50, args=<optimized out>, kwargs=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:245
#40 0x0000555555718a6d in do_call_core (kwdict=0x7ffef9908dc0, callargs=0x7ffff7696050, func=0x555558851e50)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:4645
#41 _PyEval_EvalFrameDefault (f=0x5555588492f0, throwflag=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3191
--Type <RET> for more, q to quit, c to continue without paging--
#42 0x000055555568cd17 in PyEval_EvalFrameEx (throwflag=0, f=0x5555588492f0)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:547
#43 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:283
#44 _PyFunction_FastCallKeywords (func=<optimized out>, stack=0x555558842f90, nargs=<optimized out>, kwnames=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:408
#45 0x0000555555717155 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:4616
#46 _PyEval_EvalFrameDefault (f=0x555558842dd0, throwflag=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3124
#47 0x000055555566cbb3 in PyEval_EvalFrameEx (throwflag=0, f=0x555558842dd0)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:547
#48 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=<optimized out>, kwnames=0x7ffff73b76a8, kwargs=0x7ffef98e2810, kwcount=<optimized out>, kwstep=1, defs=0x7ffefabd36a8, 
    defcount=1, kwdefs=0x0, closure=0x0, name=0x7ffff73b65d0, qualname=0x7ffff73b1b10)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3930
#49 0x000055555568d223 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=0x7ffef98e2800, nargs=<optimized out>, 
    kwnames=<optimized out>) at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:433
#50 0x00005555556d40c5 in call_function (pp_stack=0x7fffffffc690, oparg=<optimized out>, kwnames=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:4616
#51 0x00005555557181bc in _PyEval_EvalFrameDefault (f=0x7ffef98e2650, throwflag=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3139
#52 0x000055555566cbb3 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffef98e2650)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:547
#53 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>,            [55/1882]
    argcount=<optimized out>, kwnames=0x7ffff73b7aa8, kwargs=0x7ffef988b3b8, kwcount=<optimized out>, kwstep=1, defs=0x7ffff73b78e8, 
    defcount=1, kwdefs=0x0, closure=0x0, name=0x7ffff73b67b0, qualname=0x7ffff73b1e70)                           
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3930                                                   
#54 0x000055555568d223 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=0x7ffef988b3a8, nargs=<optimized out>, 
    kwnames=<optimized out>) at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:433                 
#55 0x00005555556d40c5 in call_function (pp_stack=0x7fffffffc970, oparg=<optimized out>, kwnames=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:4616                        
#56 0x00005555557181bc in _PyEval_EvalFrameDefault (f=0x7ffef988b220, throwflag=<optimized out>)       
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3139    
#57 0x000055555566d0de in PyEval_EvalFrameEx (throwflag=0, f=0x7ffef988b220)                                                                  
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:547                                                                       
#58 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>,             
    argcount=<optimized out>, kwnames=<optimized out>, kwargs=0x7ffef988b1f0, kwcount=<optimized out>, kwstep=1, defs=0x7ffff73bd928, 
    defcount=1, kwdefs=0x0, closure=0x7ffff73e2710, name=0x7ffff73b67b0, qualname=0x7ffff73ba210)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3930                                                   
#59 0x000055555568d223 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=0x7ffef988b1e0, nargs=<optimized out>,            
    kwnames=<optimized out>) at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:433                 
#60 0x00005555556d40c5 in call_function (pp_stack=0x7fffffffcc48, oparg=<optimized out>, kwnames=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:4616                                                   
#61 0x0000555555717381 in _PyEval_EvalFrameDefault (f=0x7ffef988b050, throwflag=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3110                                         
#62 0x00005555556d2ffc in PyEval_EvalFrameEx (throwflag=<optimized out>, f=0x7ffef988b050)                                                    
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:547                                                                       
#63 gen_send_ex (gen=0x7ffef99354d0, arg=<optimized out>, exc=<optimized out>, closing=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/genobject.c:221                    
#64 0x00005555557177a2 in _PyEval_EvalFrameDefault (f=0x55555883fdf0, throwflag=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:2809
--Type <RET> for more, q to quit, c to continue without paging--                                                                              
#65 0x000055555566cbb3 in PyEval_EvalFrameEx (throwflag=0, f=0x55555883fdf0)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:547                                                                       
#66 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>,             
    argcount=<optimized out>, kwnames=0x0, kwargs=0x7ffef98ec3e0, kwcount=<optimized out>, kwstep=1, defs=0x7ffefabd36a8, defcount=1, 
    kwdefs=0x0, closure=0x0, name=0x7ffff73b65d0, qualname=0x7ffff73b1b10)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3930                                                   
#67 0x000055555568d1c1 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=0x7ffef98ec3d0, nargs=<optimized out>, 
    kwnames=<optimized out>) at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:433                 
#68 0x00005555556d40c5 in call_function (pp_stack=0x7fffffffd098, oparg=<optimized out>, kwnames=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:4616
#69 0x0000555555717381 in _PyEval_EvalFrameDefault (f=0x7ffef98ec230, throwflag=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3110
#70 0x000055555568cd17 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffef98ec230)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:547
#71 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:283
#72 _PyFunction_FastCallKeywords (func=<optimized out>, stack=0x7ffef98fa8d0, nargs=<optimized out>, kwnames=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:408
#73 0x00005555556d40c5 in call_function (pp_stack=0x7fffffffd288, oparg=<optimized out>, kwnames=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:4616
#74 0x0000555555717381 in _PyEval_EvalFrameDefault (f=0x7ffef98fa750, throwflag=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3110
#75 0x000055555568cd17 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffef98fa750)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:547
#76 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:283
#77 _PyFunction_FastCallKeywords (func=<optimized out>, stack=0x555558834028, nargs=<optimized out>, kwnames=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:408
#78 0x00005555556d40c5 in call_function (pp_stack=0x7fffffffd478, oparg=<optimized out>, kwnames=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:4616
#79 0x0000555555717381 in _PyEval_EvalFrameDefault (f=0x555558833ea0, throwflag=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3110
#80 0x000055555566cbb3 in PyEval_EvalFrameEx (throwflag=0, f=0x555558833ea0)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:547
#81 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=<optimized out>, kwnames=0x7ffefab85468, kwargs=0x7ffef9e6fc00, kwcount=<optimized out>, kwstep=1, defs=0x7ffff73fa628, 
    defcount=1, kwdefs=0x0, closure=0x0, name=0x7ffff763e1f0, qualname=0x7ffff763e1f0)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3930
#82 0x000055555568d223 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=0x7ffef9e6fbf8, nargs=<optimized out>, 
    kwnames=<optimized out>) at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:433
#83 0x00005555556d40c5 in call_function (pp_stack=0x7fffffffd760, oparg=<optimized out>, kwnames=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:4616
#84 0x00005555557181bc in _PyEval_EvalFrameDefault (f=0x7ffef9e6fa50, throwflag=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3139
#85 0x000055555566cbb3 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffef9e6fa50)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:547
#86 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=<optimized out>, kwnames=0x0, kwargs=0x555555971f20, kwcount=<optimized out>, kwstep=1, defs=0x7ffefab74928, defcount=2, 
    kwdefs=0x0, closure=0x0, name=0x7ffff74f8df0, qualname=0x7ffff74f8df0)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3930
#87 0x000055555568d1c1 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=0x555555971f18, nargs=<optimized out>, 
    kwnames=<optimized out>) at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:433
--Type <RET> for more, q to quit, c to continue without paging--
#88 0x0000555555717155 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:4616
#89 _PyEval_EvalFrameDefault (f=0x555555971d60, throwflag=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3124
#90 0x000055555568cd17 in PyEval_EvalFrameEx (throwflag=0, f=0x555555971d60)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:547
#91 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:283
#92 _PyFunction_FastCallKeywords (func=<optimized out>, stack=0x7ffff75e25c0, nargs=<optimized out>, kwnames=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Objects/call.c:408
#93 0x0000555555717155 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:4616
#94 _PyEval_EvalFrameDefault (f=0x7ffff75e2450, throwflag=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3124
#95 0x000055555566cbb3 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffff75e2450)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:547
#96 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=<optimized out>, kwnames=0x0, kwargs=0x0, kwcount=<optimized out>, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, 
    name=0x0, qualname=0x0) at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3930
#97 0x000055555566dee3 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, 
    locals=<optimized out>, globals=<optimized out>, _co=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:3959
#98 PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Python/ceval.c:524
#99 0x000055555577b802 in run_mod (mod=<optimized out>, filename=<optimized out>, globals=0x7ffff764dd20, locals=0x7ffff764dd20, 
    flags=<optimized out>, arena=<optimized out>) at /tmp/build/80754af9/python_1627392990942/work/Python/pythonrun.c:1037
#100 0x000055555578594e in PyRun_FileExFlags (fp=0x5555558be1e0, filename_str=<optimized out>, start=<optimized out>, 
    globals=0x7ffff764dd20, locals=0x7ffff764dd20, closeit=1, flags=0x7fffffffde20)
    at /tmp/build/80754af9/python_1627392990942/work/Python/pythonrun.c:990
#101 0x0000555555785b3b in PyRun_SimpleFileExFlags (fp=0x5555558be1e0, filename=<optimized out>, closeit=1, flags=0x7fffffffde20)
    at /tmp/build/80754af9/python_1627392990942/work/Python/pythonrun.c:429
#102 0x0000555555786c3a in pymain_run_file (p_cf=0x7fffffffde20, filename=0x5555558bd8d0 L"example_asr_ctc_experiment.py", fp=0x5555558be1e0)
    at /tmp/build/80754af9/python_1627392990942/work/Modules/main.c:462
#103 pymain_run_filename (cf=0x7fffffffde20, pymain=0x7fffffffdf30) at /tmp/build/80754af9/python_1627392990942/work/Modules/main.c:1652
#104 pymain_run_python (pymain=0x7fffffffdf30) at /tmp/build/80754af9/python_1627392990942/work/Modules/main.c:2913
#105 pymain_main (pymain=0x7fffffffdf30) at /tmp/build/80754af9/python_1627392990942/work/Modules/main.c:3460
#106 0x0000555555786ccc in _Py_UnixMain (argc=<optimized out>, argv=<optimized out>)
    at /tmp/build/80754af9/python_1627392990942/work/Modules/main.c:3495
#107 0x00007ffff7c6f0b3 in __libc_start_main (main=0x555555645340 <main>, argc=3, argv=0x7fffffffe098, init=<optimized out>, 
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe088) at ../csu/libc-start.c:308
#108 0x000055555572b555 in _start ()
```

## Expected behavior

I expected it to run and start outputting metrics, like it does on every single other computer.

## Environment

```
PyTorch version: 1.9.1+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.17

Python version: 3.7.11 (default, Jul 27 2021, 14:32:16)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-74-generic-x86_64-with-debian-bullseye-sid
Is CUDA available: True
CUDA runtime version: 11.1.105
GPU models and configuration: 
GPU 0: A100-PCIE-40GB
GPU 1: A100-PCIE-40GB
GPU 2: A100-PCIE-40GB
GPU 3: A100-PCIE-40GB
GPU 4: A100-PCIE-40GB
GPU 5: A100-PCIE-40GB
GPU 6: A100-PCIE-40GB
GPU 7: A100-PCIE-40GB

Nvidia driver version: 460.84
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] torch==1.9.1
[pip3] torchaudio==0.9.1
[conda] numpy                     1.21.2                   pypi_0    pypi
[conda] torch                     1.9.1                    pypi_0    pypi
[conda] torchaudio                0.9.1                    pypi_0    pypi
```

## Additional context

Edit: The code does not crash on the GPU. I am sorry I don't have a smaller example, but the smaller examples all work without hiccup. I suspect there is a package missing/corrupted on the host machine, but I can't figure out which and I don't see a clear-cut requirements as to what PC can run PyTorch.

cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @seemethere @malfet