GPT fp8 pretrain not working on H20: Caught signal 8 (Floating point exception: integer divide by zero)
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

We are trying to run GPT fp8 pretraining on NVIDIA H20 according to [NeMo docs](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/nemo_megatron/gpt/gpt_training.html) on single node with 8 GPUs

However, we get the error below:
```
Sanity Checking DataLoader 0:   0%|                                                                                                                                      | 0/8 [00:00<?, ?it/s][n154-161-133:121882:0:121882] Caught signal 8 (Floating point exception: integer divide by zero)
[n154-161-133:121876:0:121876] Caught signal 8 (Floating point exception: integer divide by zero)
[n154-161-133:121880:0:121880] Caught signal 8 (Floating point exception: integer divide by zero)
[n154-161-133:121878:0:121878] Caught signal 8 (Floating point exception: integer divide by zero)
[n154-161-133:121664:0:121664] Caught signal 8 (Floating point exception: integer divide by zero)
[n154-161-133:121879:0:121879] Caught signal 8 (Floating point exception: integer divide by zero)
[n154-161-133:121877:0:121877] Caught signal 8 (Floating point exception: integer divide by zero)
[n154-161-133:121881:0:121881] Caught signal 8 (Floating point exception: integer divide by zero)
```

**Steps/Code to reproduce bug**


1. Bootstrapping the NeMo docker container from NGC, we use `23.10`

```
docker run -d --name nemo \
    -u root \
    -v /data:/data \
    --privileged  \
    --network=host \
    --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 \
    --rm --gpus all \
    nvcr.io/nvidia/nemo:23.10 \
    sleep infinity
```

2. Download and preprocess the enwiki dataset according the [NeMo docs](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/nemo_megatron/gpt/gpt_training.html) using HuggingFace GPT2 tokenizer files under path `/data`

```
docker exec -it nemo bash
cd /data
python /workspace/nemo/scripts/nlp_language_modeling/preprocess_data_for_megatron.py \
--input=train_data.jsonl \
--json-keys=text \
--tokenizer-library=megatron \
--vocab gpt2-vocab.json \
--dataset-impl mmap \
--tokenizer-type GPT2BPETokenizer \
--merge-file gpt2-merges.txt \
--output-prefix=hfbpe_gpt_training_data \
--append-eod \
--workers=96
```

3. Launch the Single Node 8 GPU job in the container which enabling fp8

```
python /workspace/nemo/examples/nlp/language_modeling/megatron_gpt_pretraining.py  \
    --config-path=/workspace/nemo/examples/nlp/language_modeling/conf \
    --config-name=megatron_gpt_config \
    trainer.devices=8 \
    trainer.num_nodes=1 \
    trainer.max_epochs=null \
    trainer.max_steps=300000 \
    trainer.val_check_interval=300 \
    trainer.log_every_n_steps=50 \
    trainer.limit_val_batches=50 \
    trainer.limit_test_batches=50 \
    trainer.accumulate_grad_batches=1 \
    trainer.precision=bf16 \
    model.transformer_engine=True \
    model.fp8=True \
    model.micro_batch_size=6 \
    model.global_batch_size=192 \
    model.tensor_model_parallel_size=1 \
    model.pipeline_model_parallel_size=1 \
    model.max_position_embeddings=1024 \
    model.encoder_seq_length=1024 \
    model.hidden_size=768 \
    model.ffn_hidden_size=3072 \
    model.num_layers=12 \
    model.num_attention_heads=12 \
    model.init_method_std=0.021 \
    model.hidden_dropout=0.1 \
    model.layernorm_epsilon=1e-5 \
    model.tokenizer.vocab_file=gpt2-vocab.json \
    model.tokenizer.merge_file=gpt2-merges.txt \
    model.data.data_prefix=[1.0,hfbpe_gpt_training_data_text_document] \
    model.data.num_workers=2 \
    model.data.seq_length=1024 \
    model.data.splits_string=\'980,10,10\' \
    model.optim.name=fused_adam \
    model.optim.lr=6e-4 \
    model.optim.betas=[0.9,0.95] \
    model.optim.weight_decay=0.1 \
    model.optim.sched.name=CosineAnnealing \
    model.optim.sched.warmup_steps=750 \
    model.optim.sched.constant_steps=80000 \
    model.optim.sched.min_lr=6e-5 \
    exp_manager.resume_if_exists=True \
    exp_manager.resume_ignore_no_checkpoint=True \
    exp_manager.create_checkpoint_callback=True \
    exp_manager.checkpoint_callback_params.monitor=val_loss \
    exp_manager.checkpoint_callback_params.save_top_k=3 \
    exp_manager.checkpoint_callback_params.mode=min \
    exp_manager.checkpoint_callback_params.always_save_nemo=False
```


**Expected behavior**

The job should launch successfully and start training.

We execute the same experiment on NVIDIA H100, and it works
```
Epoch 0: :   5%|▋             | 15900/301500 [1:22:Epoch 0, global step 15900: 'val_loss' reached 3.04824 (best 3.04824), saving model to '/data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=3.05-step=15900-consumed_samples=3052800.0.ckpt' as top 3
[NeMo I 2024-02-26 06:39:06 nlp_overrides:412] Removing checkpoint: /data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=3.05-step=15000-consumed_samples=2880000.0.ckpt
[NeMo I 2024-02-26 06:39:09 nlp_overrides:412] Removing checkpoint: /data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=3.05-step=15600-consumed_samples=2995200.0-last.ckpt
Epoch 0: :   5%|▊             | 16200/301500 [1:24:Epoch 0, global step 16200: 'val_loss' reached 3.04690 (best 3.04690), saving model to '/data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=3.05-step=16200-consumed_samples=3110400.0.ckpt' as top 3
[NeMo I 2024-02-26 06:40:41 nlp_overrides:412] Removing checkpoint: /data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=3.05-step=15300-consumed_samples=2937600.0.ckpt
[NeMo I 2024-02-26 06:40:44 nlp_overrides:412] Removing checkpoint: /data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=3.05-step=15900-consumed_samples=3052800.0-last.ckpt
Epoch 0: :   5%|▊             | 16500/301500 [1:25:Epoch 0, global step 16500: 'val_loss' reached 3.04502 (best 3.04502), saving model to '/data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=3.05-step=16500-consumed_samples=3168000.0.ckpt' as top 3
[NeMo I 2024-02-26 06:42:15 nlp_overrides:412] Removing checkpoint: /data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=3.05-step=15600-consumed_samples=2995200.0.ckpt
[NeMo I 2024-02-26 06:42:18 nlp_overrides:412] Removing checkpoint: /data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=3.05-step=16200-consumed_samples=3110400.0-last.ckpt
Epoch 0: :   6%|▊             | 16800/301500 [1:27:Epoch 0, global step 16800: 'val_loss' reached 3.04356 (best 3.04356), saving model to '/data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=3.04-step=16800-consumed_samples=3225600.0.ckpt' as top 3
[NeMo I 2024-02-26 06:43:50 nlp_overrides:412] Removing checkpoint: /data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=3.05-step=15900-consumed_samples=3052800.0.ckpt
[NeMo I 2024-02-26 06:43:53 nlp_overrides:412] Removing checkpoint: /data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=3.05-step=16500-consumed_samples=3168000.0-last.ckpt
Epoch 0: :   6%|▊             | 17100/301500 [1:29:Epoch 0, global step 17100: 'val_loss' reached 3.04004 (best 3.04004), saving model to '/data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=3.04-step=17100-consumed_samples=3283200.0.ckpt' as top 3
```

we also test bf16 training on NVIDIA H20, it also works

```
Epoch 0: :   0%|                    | 300/301500 [0Epoch 0, global step 300: 'val_loss' reached 5.98933 (best 5.98933), saving model to '/data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=5.99-step=300-consumed_samples=57600.0.ckpt' as top 3
Epoch 0: :   0%|                    | 600/301500 [0Epoch 0, global step 600: 'val_loss' reached 4.94614 (best 4.94614), saving model to '/data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=4.95-step=600-consumed_samples=115200.0.ckpt' as top 3
[NeMo I 2024-02-26 06:48:45 nlp_overrides:412] Removing checkpoint: /data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=5.99-step=300-consumed_samples=57600.0-last.ckpt
Epoch 0: :   0%|                    | 900/301500 [0Epoch 0, global step 900: 'val_loss' reached 4.18590 (best 4.18590), saving model to '/data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=4.19-step=900-consumed_samples=172800.0.ckpt' as top 3
[NeMo I 2024-02-26 06:50:43 nlp_overrides:412] Removing checkpoint: /data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=4.95-step=600-consumed_samples=115200.0-last.ckpt
Epoch 0: :   0%|                   | 1200/301500 [0Epoch 0, global step 1200: 'val_loss' reached 3.88466 (best 3.88466), saving model to '/data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=3.88-step=1200-consumed_samples=230400.0.ckpt' as top 3
[NeMo I 2024-02-26 06:52:38 nlp_overrides:412] Removing checkpoint: /data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=5.99-step=300-consumed_samples=57600.0.ckpt
[NeMo I 2024-02-26 06:52:40 nlp_overrides:412] Removing checkpoint: /data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=4.19-step=900-consumed_samples=172800.0-last.ckpt
Epoch 0: :   0%|                  | 1500/301500 [09Epoch 0, global step 1500: 'val_loss' reached 3.73976 (best 3.73976), saving model to '/data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=3.74-step=1500-consumed_samples=288000.0.ckpt' as top 3
[NeMo I 2024-02-26 06:54:36 nlp_overrides:412] Removing checkpoint: /data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=4.95-step=600-consumed_samples=115200.0.ckpt
[NeMo I 2024-02-26 06:54:39 nlp_overrides:412] Removing checkpoint: /data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=3.88-step=1200-consumed_samples=230400.0-last.ckpt
Epoch 0: :   1%|                  | 1800/301500 [11Epoch 0, global step 1800: 'val_loss' reached 3.63803 (best 3.63803), saving model to '/data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=3.64-step=1800-consumed_samples=345600.0.ckpt' as top 3
[NeMo I 2024-02-26 06:56:35 nlp_overrides:412] Removing checkpoint: /data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=4.19-step=900-consumed_samples=172800.0.ckpt
[NeMo I 2024-02-26 06:56:38 nlp_overrides:412] Removing checkpoint: /data/nemo_experiments/megatron_gpt/checkpoints/megatron_gpt--val_loss=3.74-step=1500-consumed_samples=288000.0-last.ckpt
Epoch 0: :   1%|                  | 1829/301500 [12:02<32:53:20, v_num=0, reduced_train_loss=3.670, global_step=1828.0, consumed_samples=3.51e+5, train_step_timing in s=0.351, val_loss=3.640]
```


**Environment overview (please complete the following information)**

 - Environment location: Docker singularity containers created from NVIDIA's NGC NeMo containers 23.10
 - Method of NeMo install: Pre-installed in container.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

```
# docker pull 
docker pull nvcr.io/nvidia/nemo:23.10
# docker run already shown above
```


**Additional context**

Add any other context about the problem here.
Example: GPU model H20

Backtrace for H20

```
==== backtrace (tid: 121876) ====
 0 0x0000000000042520 __sigaction()  ???:0
 1 0x0000000000b83824 cublasLt_for_cublas_ZZZ()  ???:0
 2 0x0000000000897f37 cublasLt_for_cublas_ZZZ()  ???:0
 3 0x000000000072c917 cublasLtLegacyGemmUtilizationZZZ()  ???:0
 4 0x00000000007ff3de cublasLtMatmulAlgoCheck()  ???:0
 5 0x0000000000800399 cublasLtMatmulAlgoCheck()  ???:0
 6 0x0000000000800ecc cublasLtMatmulAlgoCheck()  ???:0
 7 0x0000000000829a9d cublasLtMatmulAlgoGetHeuristic()  ???:0
 8 0x0000000000273439 transformer_engine::cublas_gemm()  ???:0
 9 0x0000000000273904 nvte_cublas_gemm()  ???:0
10 0x0000000000057c56 te_gemm()  ???:0
11 0x00000000000bb968 te_gemm_ts()  /workspace/TransformerEngine/transformer_engine/pytorch/csrc/ts_fp8_op.cpp:265
12 0x00000000000be875 c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >::operator()()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/WrapFunctionIntoRuntimeFunctor.h:18
13 0x00000000000be875 c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, at::Tensor (at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long)>::call()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:463
14 0x00000000000bfaca c10::impl::call_functor_with_args_from_stack_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul, 9ul, 10ul, 11ul, 12ul, 13ul, 14ul, 15ul, 16ul, 17ul, 18ul, 19ul, 20ul, 21ul, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long>()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:501
15 0x00000000000bfaca c10::impl::call_functor_with_args_from_stack<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false>()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:513
16 0x00000000000bfaca c10::impl::make_boxed_from_unboxed_functor<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false>::call()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:579
17 0x0000000004817624 c10::Dispatcher::callBoxed()  register_c10_ops.cpp:0
18 0x000000000085fdb4 torch::jit::invokeOperatorFromPython()  ???:0
19 0x000000000086067e torch::jit::_get_operation_for_overload_or_packet()  ???:0
20 0x00000000007459e6 pybind11::cpp_function::initialize<torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#194}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args, pybind11::kwargs)#1}, pybind11::object, pybind11::args, pybind11::kwargs, pybind11::name, pybind11::doc>(torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#194}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args, pybind11::kwargs)#1}&&, pybind11::object (*)(pybind11::args, pybind11::kwargs), pybind11::name const&, pybind11::doc const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN()  init.cpp:0
21 0x000000000039c407 pybind11::cpp_function::dispatcher()  :0
22 0x000000000015a10e PyObject_CallFunctionObjArgs()  ???:0
23 0x000000000016942b PyObject_Call()  ???:0
24 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
25 0x000000000014fc14 _PyObject_FastCallDictTstate()  ???:0
26 0x00000000001658d1 _PyObject_Call_Prepend()  ???:0
27 0x0000000000280700 PyInit__datetime()  ???:0
28 0x000000000016942b PyObject_Call()  ???:0
29 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
30 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
31 0x000000000014453c _PyEval_EvalFrameDefault()  ???:0
32 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
33 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
34 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
35 0x0000000000169492 PyObject_Call()  ???:0
36 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
37 0x00000000001687f1 PyMethod_New()  ???:0
38 0x0000000000169492 PyObject_Call()  ???:0
39 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
40 0x00000000001687f1 PyMethod_New()  ???:0
41 0x0000000000169492 PyObject_Call()  ???:0
42 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
43 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
44 0x000000000014fcbd _PyObject_FastCallDictTstate()  ???:0
45 0x000000000016586c _PyObject_Call_Prepend()  ???:0
46 0x0000000000280700 PyInit__datetime()  ???:0
47 0x0000000000150a7b _PyObject_MakeTpCall()  ???:0
48 0x000000000014a150 _PyEval_EvalFrameDefault()  ???:0
49 0x00000000001687f1 PyMethod_New()  ???:0
50 0x0000000000169492 PyObject_Call()  ???:0
51 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
52 0x00000000001687f1 PyMethod_New()  ???:0
53 0x0000000000169492 PyObject_Call()  ???:0
54 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
55 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
56 0x000000000014fcbd _PyObject_FastCallDictTstate()  ???:0
=================================
==== backtrace (tid: 121878) ====
 0 0x0000000000042520 __sigaction()  ???:0
 1 0x0000000000b83824 cublasLt_for_cublas_ZZZ()  ???:0
 2 0x0000000000897f37 cublasLt_for_cublas_ZZZ()  ???:0
 3 0x000000000072c917 cublasLtLegacyGemmUtilizationZZZ()  ???:0
 4 0x00000000007ff3de cublasLtMatmulAlgoCheck()  ???:0
 5 0x0000000000800399 cublasLtMatmulAlgoCheck()  ???:0
 6 0x0000000000800ecc cublasLtMatmulAlgoCheck()  ???:0
 7 0x0000000000829a9d cublasLtMatmulAlgoGetHeuristic()  ???:0
 8 0x0000000000273439 transformer_engine::cublas_gemm()  ???:0
==== backtrace (tid: 121880) ====
 9 0x0000000000273904 nvte_cublas_gemm()  ???:0
10 0x0000000000057c56 te_gemm()  ???:0
 0 0x0000000000042520 __sigaction()  ???:0
11 0x00000000000bb968 te_gemm_ts()  /workspace/TransformerEngine/transformer_engine/pytorch/csrc/ts_fp8_op.cpp:265
 1 0x0000000000b83824 cublasLt_for_cublas_ZZZ()  ???:0
12 0x00000000000be875 c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >::operator()()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/WrapFunctionIntoRuntimeFunctor.h:18
 2 0x0000000000897f37 cublasLt_for_cublas_ZZZ()  ???:0
13 0x00000000000be875 c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, at::Tensor (at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long)>::call()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:463
 3 0x000000000072c917 cublasLtLegacyGemmUtilizationZZZ()  ???:0
14 0x00000000000bfaca c10::impl::call_functor_with_args_from_stack_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul, 9ul, 10ul, 11ul, 12ul, 13ul, 14ul, 15ul, 16ul, 17ul, 18ul, 19ul, 20ul, 21ul, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long>()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:501
 4 0x00000000007ff3de cublasLtMatmulAlgoCheck()  ???:0
15 0x00000000000bfaca c10::impl::call_functor_with_args_from_stack<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false>()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:513
 5 0x0000000000800399 cublasLtMatmulAlgoCheck()  ???:0
16 0x00000000000bfaca c10::impl::make_boxed_from_unboxed_functor<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false>::call()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:579
 6 0x0000000000800ecc cublasLtMatmulAlgoCheck()  ???:0
17 0x0000000004817624 c10::Dispatcher::callBoxed()  register_c10_ops.cpp:0
 7 0x0000000000829a9d cublasLtMatmulAlgoGetHeuristic()  ???:0
18 0x000000000085fdb4 torch::jit::invokeOperatorFromPython()  ???:0
 8 0x0000000000273439 transformer_engine::cublas_gemm()  ???:0
19 0x000000000086067e torch::jit::_get_operation_for_overload_or_packet()  ???:0
 9 0x0000000000273904 nvte_cublas_gemm()  ???:0
20 0x00000000007459e6 pybind11::cpp_function::initialize<torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#194}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args, pybind11::kwargs)#1}, pybind11::object, pybind11::args, pybind11::kwargs, pybind11::name, pybind11::doc>(torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#194}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args, pybind11::kwargs)#1}&&, pybind11::object (*)(pybind11::args, pybind11::kwargs), pybind11::name const&, pybind11::doc const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN()  init.cpp:0
10 0x0000000000057c56 te_gemm()  ???:0
21 0x000000000039c407 pybind11::cpp_function::dispatcher()  :0
11 0x00000000000bb968 te_gemm_ts()  /workspace/TransformerEngine/transformer_engine/pytorch/csrc/ts_fp8_op.cpp:265
22 0x000000000015a10e PyObject_CallFunctionObjArgs()  ???:0
12 0x00000000000be875 c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >::operator()()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/WrapFunctionIntoRuntimeFunctor.h:18
23 0x000000000016942b PyObject_Call()  ???:0
13 0x00000000000be875 c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, at::Tensor (at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long)>::call()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:463
24 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
14 0x00000000000bfaca c10::impl::call_functor_with_args_from_stack_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul, 9ul, 10ul, 11ul, 12ul, 13ul, 14ul, 15ul, 16ul, 17ul, 18ul, 19ul, 20ul, 21ul, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long>()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:501
25 0x000000000014fc14 _PyObject_FastCallDictTstate()  ???:0
15 0x00000000000bfaca c10::impl::call_functor_with_args_from_stack<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false>()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:513
26 0x00000000001658d1 _PyObject_Call_Prepend()  ???:0
16 0x00000000000bfaca c10::impl::make_boxed_from_unboxed_functor<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false>::call()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:579
27 0x0000000000280700 PyInit__datetime()  ???:0
17 0x0000000004817624 c10::Dispatcher::callBoxed()  register_c10_ops.cpp:0
28 0x000000000016942b PyObject_Call()  ???:0
18 0x000000000085fdb4 torch::jit::invokeOperatorFromPython()  ???:0
29 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
19 0x000000000086067e torch::jit::_get_operation_for_overload_or_packet()  ???:0
30 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
20 0x00000000007459e6 pybind11::cpp_function::initialize<torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#194}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args, pybind11::kwargs)#1}, pybind11::object, pybind11::args, pybind11::kwargs, pybind11::name, pybind11::doc>(torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#194}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args, pybind11::kwargs)#1}&&, pybind11::object (*)(pybind11::args, pybind11::kwargs), pybind11::name const&, pybind11::doc const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN()  init.cpp:0
31 0x000000000014453c _PyEval_EvalFrameDefault()  ???:0
21 0x000000000039c407 pybind11::cpp_function::dispatcher()  :0
32 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
22 0x000000000015a10e PyObject_CallFunctionObjArgs()  ???:0
33 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
23 0x000000000016942b PyObject_Call()  ???:0
34 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
24 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
35 0x0000000000169492 PyObject_Call()  ???:0
25 0x000000000014fc14 _PyObject_FastCallDictTstate()  ???:0
36 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
26 0x00000000001658d1 _PyObject_Call_Prepend()  ???:0
37 0x00000000001687f1 PyMethod_New()  ???:0
27 0x0000000000280700 PyInit__datetime()  ???:0
38 0x0000000000169492 PyObject_Call()  ???:0
28 0x000000000016942b PyObject_Call()  ???:0
39 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
29 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
40 0x00000000001687f1 PyMethod_New()  ???:0
30 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
41 0x0000000000169492 PyObject_Call()  ???:0
31 0x000000000014453c _PyEval_EvalFrameDefault()  ???:0
42 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
32 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
43 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
33 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
44 0x000000000014fcbd _PyObject_FastCallDictTstate()  ???:0
34 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
45 0x000000000016586c _PyObject_Call_Prepend()  ???:0
35 0x0000000000169492 PyObject_Call()  ???:0
46 0x0000000000280700 PyInit__datetime()  ???:0
36 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
47 0x0000000000150a7b _PyObject_MakeTpCall()  ???:0
37 0x00000000001687f1 PyMethod_New()  ???:0
48 0x000000000014a150 _PyEval_EvalFrameDefault()  ???:0
38 0x0000000000169492 PyObject_Call()  ???:0
49 0x00000000001687f1 PyMethod_New()  ???:0
39 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
50 0x0000000000169492 PyObject_Call()  ???:0
40 0x00000000001687f1 PyMethod_New()  ???:0
51 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
41 0x0000000000169492 PyObject_Call()  ???:0
52 0x00000000001687f1 PyMethod_New()  ???:0
42 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
53 0x0000000000169492 PyObject_Call()  ???:0
43 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
54 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
44 0x000000000014fcbd _PyObject_FastCallDictTstate()  ???:0
55 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
45 0x000000000016586c _PyObject_Call_Prepend()  ???:0
56 0x000000000014fcbd _PyObject_FastCallDictTstate()  ???:0
46 0x0000000000280700 PyInit__datetime()  ???:0
=================================
47 0x0000000000150a7b _PyObject_MakeTpCall()  ???:0
48 0x000000000014a150 _PyEval_EvalFrameDefault()  ???:0
49 0x00000000001687f1 PyMethod_New()  ???:0
50 0x0000000000169492 PyObject_Call()  ???:0
51 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
52 0x00000000001687f1 PyMethod_New()  ???:0
53 0x0000000000169492 PyObject_Call()  ???:0
54 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
55 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
56 0x000000000014fcbd _PyObject_FastCallDictTstate()  ???:0
=================================
==== backtrace (tid: 121881) ====
 0 0x0000000000042520 __sigaction()  ???:0
 1 0x0000000000b83824 cublasLt_for_cublas_ZZZ()  ???:0
 2 0x0000000000897f37 cublasLt_for_cublas_ZZZ()  ???:0
 3 0x000000000072c917 cublasLtLegacyGemmUtilizationZZZ()  ???:0
 4 0x00000000007ff3de cublasLtMatmulAlgoCheck()  ???:0
 5 0x0000000000800399 cublasLtMatmulAlgoCheck()  ???:0
 6 0x0000000000800ecc cublasLtMatmulAlgoCheck()  ???:0
 7 0x0000000000829a9d cublasLtMatmulAlgoGetHeuristic()  ???:0
 8 0x0000000000273439 transformer_engine::cublas_gemm()  ???:0
 9 0x0000000000273904 nvte_cublas_gemm()  ???:0
10 0x0000000000057c56 te_gemm()  ???:0
11 0x00000000000bb968 te_gemm_ts()  /workspace/TransformerEngine/transformer_engine/pytorch/csrc/ts_fp8_op.cpp:265
12 0x00000000000be875 c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >::operator()()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/WrapFunctionIntoRuntimeFunctor.h:18
13 0x00000000000be875 c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, at::Tensor (at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long)>::call()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:463
14 0x00000000000bfaca c10::impl::call_functor_with_args_from_stack_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul, 9ul, 10ul, 11ul, 12ul, 13ul, 14ul, 15ul, 16ul, 17ul, 18ul, 19ul, 20ul, 21ul, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long>()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:501
15 0x00000000000bfaca c10::impl::call_functor_with_args_from_stack<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false>()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:513
16 0x00000000000bfaca c10::impl::make_boxed_from_unboxed_functor<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false>::call()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:579
17 0x0000000004817624 c10::Dispatcher::callBoxed()  register_c10_ops.cpp:0
18 0x000000000085fdb4 torch::jit::invokeOperatorFromPython()  ???:0
19 0x000000000086067e torch::jit::_get_operation_for_overload_or_packet()  ???:0
20 0x00000000007459e6 pybind11::cpp_function::initialize<torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#194}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args, pybind11::kwargs)#1}, pybind11::object, pybind11::args, pybind11::kwargs, pybind11::name, pybind11::doc>(torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#194}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args, pybind11::kwargs)#1}&&, pybind11::object (*)(pybind11::args, pybind11::kwargs), pybind11::name const&, pybind11::doc const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN()  init.cpp:0
21 0x000000000039c407 pybind11::cpp_function::dispatcher()  :0
22 0x000000000015a10e PyObject_CallFunctionObjArgs()  ???:0
23 0x000000000016942b PyObject_Call()  ???:0
24 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
25 0x000000000014fc14 _PyObject_FastCallDictTstate()  ???:0
26 0x00000000001658d1 _PyObject_Call_Prepend()  ???:0
27 0x0000000000280700 PyInit__datetime()  ???:0
28 0x000000000016942b PyObject_Call()  ???:0
29 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
30 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
31 0x000000000014453c _PyEval_EvalFrameDefault()  ???:0
32 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
33 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
34 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
35 0x0000000000169492 PyObject_Call()  ???:0
36 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
37 0x00000000001687f1 PyMethod_New()  ???:0
38 0x0000000000169492 PyObject_Call()  ???:0
39 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
40 0x00000000001687f1 PyMethod_New()  ???:0
41 0x0000000000169492 PyObject_Call()  ???:0
42 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
43 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
44 0x000000000014fcbd _PyObject_FastCallDictTstate()  ???:0
45 0x000000000016586c _PyObject_Call_Prepend()  ???:0
46 0x0000000000280700 PyInit__datetime()  ???:0
47 0x0000000000150a7b _PyObject_MakeTpCall()  ???:0
48 0x000000000014a150 _PyEval_EvalFrameDefault()  ???:0
49 0x00000000001687f1 PyMethod_New()  ???:0
50 0x0000000000169492 PyObject_Call()  ???:0
51 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
52 0x00000000001687f1 PyMethod_New()  ???:0
53 0x0000000000169492 PyObject_Call()  ???:0
54 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
55 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
56 0x000000000014fcbd _PyObject_FastCallDictTstate()  ???:0
=================================
==== backtrace (tid: 121879) ====
 0 0x0000000000042520 __sigaction()  ???:0
 1 0x0000000000b83824 cublasLt_for_cublas_ZZZ()  ???:0
 2 0x0000000000897f37 cublasLt_for_cublas_ZZZ()  ???:0
 3 0x000000000072c917 cublasLtLegacyGemmUtilizationZZZ()  ???:0
 4 0x00000000007ff3de cublasLtMatmulAlgoCheck()  ???:0
 5 0x0000000000800399 cublasLtMatmulAlgoCheck()  ???:0
 6 0x0000000000800ecc cublasLtMatmulAlgoCheck()  ???:0
 7 0x0000000000829a9d cublasLtMatmulAlgoGetHeuristic()  ???:0
 8 0x0000000000273439 transformer_engine::cublas_gemm()  ???:0
 9 0x0000000000273904 nvte_cublas_gemm()  ???:0
10 0x0000000000057c56 te_gemm()  ???:0
11 0x00000000000bb968 te_gemm_ts()  /workspace/TransformerEngine/transformer_engine/pytorch/csrc/ts_fp8_op.cpp:265
12 0x00000000000be875 c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >::operator()()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/WrapFunctionIntoRuntimeFunctor.h:18
13 0x00000000000be875 c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, at::Tensor (at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long)>::call()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:463
14 0x00000000000bfaca c10::impl::call_functor_with_args_from_stack_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul, 9ul, 10ul, 11ul, 12ul, 13ul, 14ul, 15ul, 16ul, 17ul, 18ul, 19ul, 20ul, 21ul, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long>()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:501
15 0x00000000000bfaca c10::impl::call_functor_with_args_from_stack<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false>()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:513
16 0x00000000000bfaca c10::impl::make_boxed_from_unboxed_functor<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false>::call()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:579
17 0x0000000004817624 c10::Dispatcher::callBoxed()  register_c10_ops.cpp:0
18 0x000000000085fdb4 torch::jit::invokeOperatorFromPython()  ???:0
19 0x000000000086067e torch::jit::_get_operation_for_overload_or_packet()  ???:0
20 0x00000000007459e6 pybind11::cpp_function::initialize<torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#194}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args, pybind11::kwargs)#1}, pybind11::object, pybind11::args, pybind11::kwargs, pybind11::name, pybind11::doc>(torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#194}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args, pybind11::kwargs)#1}&&, pybind11::object (*)(pybind11::args, pybind11::kwargs), pybind11::name const&, pybind11::doc const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN()  init.cpp:0
21 0x000000000039c407 pybind11::cpp_function::dispatcher()  :0
22 0x000000000015a10e PyObject_CallFunctionObjArgs()  ???:0
23 0x000000000016942b PyObject_Call()  ???:0
24 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
25 0x000000000014fc14 _PyObject_FastCallDictTstate()  ???:0
26 0x00000000001658d1 _PyObject_Call_Prepend()  ???:0
27 0x0000000000280700 PyInit__datetime()  ???:0
28 0x000000000016942b PyObject_Call()  ???:0
29 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
30 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
31 0x000000000014453c _PyEval_EvalFrameDefault()  ???:0
32 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
33 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
34 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
35 0x0000000000169492 PyObject_Call()  ???:0
36 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
37 0x00000000001687f1 PyMethod_New()  ???:0
38 0x0000000000169492 PyObject_Call()  ???:0
39 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
40 0x00000000001687f1 PyMethod_New()  ???:0
41 0x0000000000169492 PyObject_Call()  ???:0
42 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
43 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
44 0x000000000014fcbd _PyObject_FastCallDictTstate()  ???:0
45 0x000000000016586c _PyObject_Call_Prepend()  ???:0
46 0x0000000000280700 PyInit__datetime()  ???:0
47 0x0000000000150a7b _PyObject_MakeTpCall()  ???:0
48 0x000000000014a150 _PyEval_EvalFrameDefault()  ???:0
49 0x00000000001687f1 PyMethod_New()  ???:0
50 0x0000000000169492 PyObject_Call()  ???:0
51 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
52 0x00000000001687f1 PyMethod_New()  ???:0
53 0x0000000000169492 PyObject_Call()  ???:0
54 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
55 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
56 0x000000000014fcbd _PyObject_FastCallDictTstate()  ???:0
=================================
==== backtrace (tid: 121877) ====
 0 0x0000000000042520 __sigaction()  ???:0
 1 0x0000000000b83824 cublasLt_for_cublas_ZZZ()  ???:0
 2 0x0000000000897f37 cublasLt_for_cublas_ZZZ()  ???:0
 3 0x000000000072c917 cublasLtLegacyGemmUtilizationZZZ()  ???:0
 4 0x00000000007ff3de cublasLtMatmulAlgoCheck()  ???:0
 5 0x0000000000800399 cublasLtMatmulAlgoCheck()  ???:0
 6 0x0000000000800ecc cublasLtMatmulAlgoCheck()  ???:0
 7 0x0000000000829a9d cublasLtMatmulAlgoGetHeuristic()  ???:0
 8 0x0000000000273439 transformer_engine::cublas_gemm()  ???:0
 9 0x0000000000273904 nvte_cublas_gemm()  ???:0
10 0x0000000000057c56 te_gemm()  ???:0
11 0x00000000000bb968 te_gemm_ts()  /workspace/TransformerEngine/transformer_engine/pytorch/csrc/ts_fp8_op.cpp:265
12 0x00000000000be875 c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >::operator()()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/WrapFunctionIntoRuntimeFunctor.h:18
13 0x00000000000be875 c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, at::Tensor (at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long)>::call()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:463
14 0x00000000000bfaca c10::impl::call_functor_with_args_from_stack_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul, 9ul, 10ul, 11ul, 12ul, 13ul, 14ul, 15ul, 16ul, 17ul, 18ul, 19ul, 20ul, 21ul, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long>()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:501
15 0x00000000000bfaca c10::impl::call_functor_with_args_from_stack<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false>()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:513
16 0x00000000000bfaca c10::impl::make_boxed_from_unboxed_functor<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false>::call()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:579
17 0x0000000004817624 c10::Dispatcher::callBoxed()  register_c10_ops.cpp:0
18 0x000000000085fdb4 torch::jit::invokeOperatorFromPython()  ???:0
19 0x000000000086067e torch::jit::_get_operation_for_overload_or_packet()  ???:0
20 0x00000000007459e6 pybind11::cpp_function::initialize<torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#194}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args, pybind11::kwargs)#1}, pybind11::object, pybind11::args, pybind11::kwargs, pybind11::name, pybind11::doc>(torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#194}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args, pybind11::kwargs)#1}&&, pybind11::object (*)(pybind11::args, pybind11::kwargs), pybind11::name const&, pybind11::doc const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN()  init.cpp:0
21 0x000000000039c407 pybind11::cpp_function::dispatcher()  :0
22 0x000000000015a10e PyObject_CallFunctionObjArgs()  ???:0
23 0x000000000016942b PyObject_Call()  ???:0
24 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
25 0x000000000014fc14 _PyObject_FastCallDictTstate()  ???:0
26 0x00000000001658d1 _PyObject_Call_Prepend()  ???:0
27 0x0000000000280700 PyInit__datetime()  ???:0
28 0x000000000016942b PyObject_Call()  ???:0
29 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
30 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
31 0x000000000014453c _PyEval_EvalFrameDefault()  ???:0
32 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
33 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
34 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
35 0x0000000000169492 PyObject_Call()  ???:0
36 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
37 0x00000000001687f1 PyMethod_New()  ???:0
38 0x0000000000169492 PyObject_Call()  ???:0
39 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
40 0x00000000001687f1 PyMethod_New()  ???:0
41 0x0000000000169492 PyObject_Call()  ???:0
42 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
43 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
44 0x000000000014fcbd _PyObject_FastCallDictTstate()  ???:0
45 0x000000000016586c _PyObject_Call_Prepend()  ???:0
46 0x0000000000280700 PyInit__datetime()  ???:0
47 0x0000000000150a7b _PyObject_MakeTpCall()  ???:0
48 0x000000000014a150 _PyEval_EvalFrameDefault()  ???:0
49 0x00000000001687f1 PyMethod_New()  ???:0
50 0x0000000000169492 PyObject_Call()  ???:0
51 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
52 0x00000000001687f1 PyMethod_New()  ???:0
53 0x0000000000169492 PyObject_Call()  ???:0
54 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
55 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
56 0x000000000014fcbd _PyObject_FastCallDictTstate()  ???:0
=================================
==== backtrace (tid: 121664) ====
 0 0x0000000000042520 __sigaction()  ???:0
 1 0x0000000000b83824 cublasLt_for_cublas_ZZZ()  ???:0
 2 0x0000000000897f37 cublasLt_for_cublas_ZZZ()  ???:0
 3 0x000000000072c917 cublasLtLegacyGemmUtilizationZZZ()  ???:0
 4 0x00000000007ff3de cublasLtMatmulAlgoCheck()  ???:0
 5 0x0000000000800399 cublasLtMatmulAlgoCheck()  ???:0
 6 0x0000000000800ecc cublasLtMatmulAlgoCheck()  ???:0
 7 0x0000000000829a9d cublasLtMatmulAlgoGetHeuristic()  ???:0
 8 0x0000000000273439 transformer_engine::cublas_gemm()  ???:0
 9 0x0000000000273904 nvte_cublas_gemm()  ???:0
10 0x0000000000057c56 te_gemm()  ???:0
11 0x00000000000bb968 te_gemm_ts()  /workspace/TransformerEngine/transformer_engine/pytorch/csrc/ts_fp8_op.cpp:265
12 0x00000000000be875 c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >::operator()()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/WrapFunctionIntoRuntimeFunctor.h:18
13 0x00000000000be875 c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, at::Tensor (at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long)>::call()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:463
14 0x00000000000bfaca c10::impl::call_functor_with_args_from_stack_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul, 9ul, 10ul, 11ul, 12ul, 13ul, 14ul, 15ul, 16ul, 17ul, 18ul, 19ul, 20ul, 21ul, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long>()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:501
15 0x00000000000bfaca c10::impl::call_functor_with_args_from_stack<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false>()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:513
16 0x00000000000bfaca c10::impl::make_boxed_from_unboxed_functor<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false>::call()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:579
17 0x0000000004817624 c10::Dispatcher::callBoxed()  register_c10_ops.cpp:0
18 0x000000000085fdb4 torch::jit::invokeOperatorFromPython()  ???:0
19 0x000000000086067e torch::jit::_get_operation_for_overload_or_packet()  ???:0
20 0x00000000007459e6 pybind11::cpp_function::initialize<torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#194}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args, pybind11::kwargs)#1}, pybind11::object, pybind11::args, pybind11::kwargs, pybind11::name, pybind11::doc>(torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#194}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args, pybind11::kwargs)#1}&&, pybind11::object (*)(pybind11::args, pybind11::kwargs), pybind11::name const&, pybind11::doc const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN()  init.cpp:0
21 0x000000000039c407 pybind11::cpp_function::dispatcher()  :0
22 0x000000000015a10e PyObject_CallFunctionObjArgs()  ???:0
23 0x000000000016942b PyObject_Call()  ???:0
24 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
25 0x000000000014fc14 _PyObject_FastCallDictTstate()  ???:0
26 0x00000000001658d1 _PyObject_Call_Prepend()  ???:0
27 0x0000000000280700 PyInit__datetime()  ???:0
28 0x000000000016942b PyObject_Call()  ???:0
29 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
30 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
31 0x000000000014453c _PyEval_EvalFrameDefault()  ???:0
32 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
33 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
34 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
35 0x0000000000169492 PyObject_Call()  ???:0
36 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
37 0x00000000001687f1 PyMethod_New()  ???:0
38 0x0000000000169492 PyObject_Call()  ???:0
39 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
40 0x00000000001687f1 PyMethod_New()  ???:0
41 0x0000000000169492 PyObject_Call()  ???:0
42 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
43 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
44 0x000000000014fcbd _PyObject_FastCallDictTstate()  ???:0
45 0x000000000016586c _PyObject_Call_Prepend()  ???:0
46 0x0000000000280700 PyInit__datetime()  ???:0
47 0x0000000000150a7b _PyObject_MakeTpCall()  ???:0
48 0x000000000014a150 _PyEval_EvalFrameDefault()  ???:0
49 0x00000000001687f1 PyMethod_New()  ???:0
50 0x0000000000169492 PyObject_Call()  ???:0
51 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
52 0x00000000001687f1 PyMethod_New()  ???:0
53 0x0000000000169492 PyObject_Call()  ???:0
54 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
55 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
56 0x000000000014fcbd _PyObject_FastCallDictTstate()  ???:0
=================================
Floating point exception (core dumped)
root@n154-161-133:/data# ==== backtrace (tid: 121882) ====
 0 0x0000000000042520 __sigaction()  ???:0
 1 0x0000000000b83824 cublasLt_for_cublas_ZZZ()  ???:0
 2 0x0000000000897f37 cublasLt_for_cublas_ZZZ()  ???:0
 3 0x000000000072c917 cublasLtLegacyGemmUtilizationZZZ()  ???:0
 4 0x00000000007ff3de cublasLtMatmulAlgoCheck()  ???:0
 5 0x0000000000800399 cublasLtMatmulAlgoCheck()  ???:0
 6 0x0000000000800ecc cublasLtMatmulAlgoCheck()  ???:0
 7 0x0000000000829a9d cublasLtMatmulAlgoGetHeuristic()  ???:0
 8 0x0000000000273439 transformer_engine::cublas_gemm()  ???:0
 9 0x0000000000273904 nvte_cublas_gemm()  ???:0
10 0x0000000000057c56 te_gemm()  ???:0
11 0x00000000000bb968 te_gemm_ts()  /workspace/TransformerEngine/transformer_engine/pytorch/csrc/ts_fp8_op.cpp:265
12 0x00000000000be875 c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >::operator()()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/WrapFunctionIntoRuntimeFunctor.h:18
13 0x00000000000be875 c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, at::Tensor (at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long)>::call()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:463
14 0x00000000000bfaca c10::impl::call_functor_with_args_from_stack_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul, 9ul, 10ul, 11ul, 12ul, 13ul, 14ul, 15ul, 16ul, 17ul, 18ul, 19ul, 20ul, 21ul, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long>()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:501
15 0x00000000000bfaca c10::impl::call_functor_with_args_from_stack<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false>()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:513
16 0x00000000000bfaca c10::impl::make_boxed_from_unboxed_functor<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long), at::Tensor, c10::guts::typelist::typelist<at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, long, long, at::Tensor, at::Tensor, long, at::Tensor, at::Tensor, long, at::Tensor, long, at::Tensor, long, long, long> >, false>::call()  /usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/boxing/impl/make_boxed_from_unboxed_functor.h:579
17 0x0000000004817624 c10::Dispatcher::callBoxed()  register_c10_ops.cpp:0
18 0x000000000085fdb4 torch::jit::invokeOperatorFromPython()  ???:0
19 0x000000000086067e torch::jit::_get_operation_for_overload_or_packet()  ???:0
20 0x00000000007459e6 pybind11::cpp_function::initialize<torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#194}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args, pybind11::kwargs)#1}, pybind11::object, pybind11::args, pybind11::kwargs, pybind11::name, pybind11::doc>(torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#194}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args, pybind11::kwargs)#1}&&, pybind11::object (*)(pybind11::args, pybind11::kwargs), pybind11::name const&, pybind11::doc const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN()  init.cpp:0
21 0x000000000039c407 pybind11::cpp_function::dispatcher()  :0
22 0x000000000015a10e PyObject_CallFunctionObjArgs()  ???:0
23 0x000000000016942b PyObject_Call()  ???:0
24 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
25 0x000000000014fc14 _PyObject_FastCallDictTstate()  ???:0
26 0x00000000001658d1 _PyObject_Call_Prepend()  ???:0
27 0x0000000000280700 PyInit__datetime()  ???:0
28 0x000000000016942b PyObject_Call()  ???:0
29 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
30 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
31 0x000000000014453c _PyEval_EvalFrameDefault()  ???:0
32 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
33 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
34 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
35 0x0000000000169492 PyObject_Call()  ???:0
36 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
37 0x00000000001687f1 PyMethod_New()  ???:0
38 0x0000000000169492 PyObject_Call()  ???:0
39 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
40 0x00000000001687f1 PyMethod_New()  ???:0
41 0x0000000000169492 PyObject_Call()  ???:0
42 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
43 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
44 0x000000000014fcbd _PyObject_FastCallDictTstate()  ???:0
45 0x000000000016586c _PyObject_Call_Prepend()  ???:0
46 0x0000000000280700 PyInit__datetime()  ???:0
47 0x0000000000150a7b _PyObject_MakeTpCall()  ???:0
48 0x000000000014a150 _PyEval_EvalFrameDefault()  ???:0
49 0x00000000001687f1 PyMethod_New()  ???:0
50 0x0000000000169492 PyObject_Call()  ???:0
51 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
52 0x00000000001687f1 PyMethod_New()  ???:0
53 0x0000000000169492 PyObject_Call()  ???:0
54 0x00000000001455d7 _PyEval_EvalFrameDefault()  ???:0
55 0x000000000015a9fc _PyFunction_Vectorcall()  ???:0
56 0x000000000014fcbd _PyObject_FastCallDictTstate()  ???:0
=================================
```

Any suggestions would be appreciated, Thanks


It seems the issue is due to CUDA 12.2 support for FP8 cuBLASLt matmul operations. After I use NeMo 24.01.framework `nvcr.io/nvidia/nemo:24.01.framework` which use CUDA 12.3, the issue is resolved.

ref: https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html

@houminz Hi, houminz. Could you tell me what driver version is installed on your H20 machine? I have met the same problem even though use CUDA 12.3 in docker.I'm really looking forward to your reply and thank you very much.

@Hap-Zhang FYI, we use 535.161.03

@houminz Got it. Thank you very much.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.