========================
START TIME: Sat Jul  6 09:43:03 UTC 2024
python3 version = Python 3.10.14
========================
The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.
Token is valid (permission: write).
Your token has been saved to /admin/home/ferdinand_mom/.cache/huggingface/token
Login successful
Already on 'bench_cluster'
M	examples/config_tiny_llama.py
M	examples/config_tiny_llama.yaml
M	examples/train_tiny_llama.sh
Your branch is up to date with 'origin/bench_cluster'.
Job status: RUNNING
[2024-07-06 09:43:05,580] torch.distributed.run: [WARNING] 
[2024-07-06 09:43:05,580] torch.distributed.run: [WARNING] *****************************************
[2024-07-06 09:43:05,580] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
[2024-07-06 09:43:05,580] torch.distributed.run: [WARNING] *****************************************
[2024-07-06 09:43:05,648] torch.distributed.run: [WARNING] 
[2024-07-06 09:43:05,648] torch.distributed.run: [WARNING] *****************************************
[2024-07-06 09:43:05,648] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
[2024-07-06 09:43:05,648] torch.distributed.run: [WARNING] *****************************************
[2024-07-06 09:43:05,678] torch.distributed.run: [WARNING] 
[2024-07-06 09:43:05,678] torch.distributed.run: [WARNING] *****************************************
[2024-07-06 09:43:05,678] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
[2024-07-06 09:43:05,678] torch.distributed.run: [WARNING] *****************************************
[2024-07-06 09:43:05,726] torch.distributed.run: [WARNING] 
[2024-07-06 09:43:05,726] torch.distributed.run: [WARNING] *****************************************
[2024-07-06 09:43:05,726] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
[2024-07-06 09:43:05,726] torch.distributed.run: [WARNING] *****************************************
[2024-07-06 09:43:05,742] torch.distributed.run: [WARNING] 
[2024-07-06 09:43:05,742] torch.distributed.run: [WARNING] *****************************************
[2024-07-06 09:43:05,742] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
[2024-07-06 09:43:05,742] torch.distributed.run: [WARNING] *****************************************
[2024-07-06 09:43:05,808] torch.distributed.run: [WARNING] 
[2024-07-06 09:43:05,808] torch.distributed.run: [WARNING] *****************************************
[2024-07-06 09:43:05,808] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
[2024-07-06 09:43:05,808] torch.distributed.run: [WARNING] *****************************************
[2024-07-06 09:43:05,823] torch.distributed.run: [WARNING] 
[2024-07-06 09:43:05,823] torch.distributed.run: [WARNING] *****************************************
[2024-07-06 09:43:05,823] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
[2024-07-06 09:43:05,823] torch.distributed.run: [WARNING] *****************************************
[2024-07-06 09:43:05,954] torch.distributed.run: [WARNING] 
[2024-07-06 09:43:05,954] torch.distributed.run: [WARNING] *****************************************
[2024-07-06 09:43:05,954] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
[2024-07-06 09:43:05,954] torch.distributed.run: [WARNING] *****************************************
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Config:
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Config(general=GeneralArgs(project='bench_cluster',
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            run='%date_%jobid',
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            seed=42,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            step=None,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            consumed_train_samples=None,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            benchmark_csv_path=None,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            ignore_sanity_checks=True),
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        parallelism=ParallelismArgs(dp=1,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    pp=64,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    tp=1,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    pp_engine=<nanotron.parallel.pipeline_parallel.engine.AllForwardAllBackwardPipelineEngine object at 0x7f741c2a88b0>,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    tp_mode=<TensorParallelLinearMode.REDUCE_SCATTER: 2>,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    tp_linear_async_communication=False,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    expert_parallel_size=1),
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        model=ModelArgs(model_config=LlamaConfig(bos_token_id=1,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 eos_token_id=2,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 hidden_act='silu',
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 hidden_size=2048,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 initializer_range=0.02,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 intermediate_size=4096,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 is_llama_config=True,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 max_position_embeddings=4096,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 num_attention_heads=32,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 num_hidden_layers=24,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 num_key_value_heads=32,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 pad_token_id=None,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 pretraining_tp=1,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 rms_norm_eps=1e-05,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 rope_scaling=None,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 rope_theta=10000.0,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 tie_word_embeddings=True,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 use_cache=True,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 vocab_size=50257),
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                        init_method=RandomInit(std=0.025),
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                        dtype=torch.bfloat16,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                        make_vocab_size_divisible_by=1,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                        ddp_bucket_cap_mb=25),
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        tokenizer=TokenizerArgs(tokenizer_name_or_path='openai-community/gpt2',
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                tokenizer_revision=None,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                tokenizer_max_length=None),
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        checkpoints=CheckpointsArgs(checkpoints_path=PosixPath('/dev/null'),
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    checkpoint_interval=100000,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    save_initial_state=False,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    resume_checkpoint_path=None,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    checkpoints_path_is_shared_file_system=False),
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        logging=LoggingArgs(log_level='info',
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            log_level_replica='info',
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            iteration_step_info_interval=1),
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        tokens=TokensArgs(sequence_length=4096,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                          train_steps=20,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                          micro_batch_size=16,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                          batch_accumulation_per_replica=64,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                          val_check_interval=-1,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                          limit_val_batches=0,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                          limit_test_batches=0),
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        optimizer=OptimizerArgs(optimizer_factory=AdamWOptimizerArgs(adam_eps=1e-08,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                     adam_beta1=0.9,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                     adam_beta2=0.95,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                     torch_adam_is_fused=True,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                     name='adamW'),
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                zero_stage=1,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                weight_decay=0.01,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                clip_grad=1.0,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                accumulate_grad_in_fp32=True,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                learning_rate_scheduler=LRSchedulerArgs(learning_rate=0.0001,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                        lr_warmup_steps=1,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                        lr_warmup_style='linear',
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                        lr_decay_style='linear',
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                        lr_decay_steps=19,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                        lr_decay_starting_step=None,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                        min_decay_lr=1e-05)),
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        data_stages=[DatasetStageArgs(name='Training Stage',
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                      start_training_step=1,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                      data=DataArgs(dataset=PretrainDatasetsArgs(hf_dataset_or_datasets='roneneldan/TinyStories',
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                                 hf_dataset_splits='train',
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                                 hf_dataset_config_name=None,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                                 dataset_processing_num_proc_per_process=64,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                                 dataset_overwrite_cache=False,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                                 text_column_name='text'),
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                    seed=42,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                    num_loading_workers=0))],
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        profiler=ProfilerArgs(profiler_export_path=PosixPath('/fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/64_GPUS/dp-1_tp-1_pp-64_mbz-16')),
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        lighteval=None)
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Model Config:
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: LlamaConfig(bos_token_id=1,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             eos_token_id=2,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             hidden_act='silu',
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             hidden_size=2048,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             initializer_range=0.02,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             intermediate_size=4096,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             is_llama_config=True,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             max_position_embeddings=4096,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             num_attention_heads=32,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             num_hidden_layers=24,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             num_key_value_heads=32,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             pad_token_id=None,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             pretraining_tp=1,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             rms_norm_eps=1e-05,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             rope_scaling=None,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             rope_theta=10000.0,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             tie_word_embeddings=True,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             use_cache=True,
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             vocab_size=50257)
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Building model..
[default0]:07/06/2024 09:43:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Setting PP block ranks...
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=19|TP=0|ip-26-0-169-139]: Local number of parameters: 41.9M (80.01MiB)
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=19|TP=0|ip-26-0-169-139]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=18|TP=0|ip-26-0-169-139]: Local number of parameters: 41.9M (80.01MiB)
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=18|TP=0|ip-26-0-169-139]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=8|TP=0|ip-26-0-169-132]: Local number of parameters: 41.9M (80.01MiB)
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=8|TP=0|ip-26-0-169-132]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=8|TP=0|ip-26-0-169-132]: No checkpoint path provided.
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=45|TP=0|ip-26-0-169-247]: Local number of parameters: 0 (0.00MiB)
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=45|TP=0|ip-26-0-169-247]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=45|TP=0|ip-26-0-169-247]: No checkpoint path provided.
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=53|TP=0|ip-26-0-169-86]: Local number of parameters: 0 (0.00MiB)
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=53|TP=0|ip-26-0-169-86]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=53|TP=0|ip-26-0-169-86]: No checkpoint path provided.
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=4|TP=0|ip-26-0-160-192]: Local number of parameters: 41.9M (80.01MiB)
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=4|TP=0|ip-26-0-160-192]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=4|TP=0|ip-26-0-160-192]: No checkpoint path provided.
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=19|TP=0|ip-26-0-169-139]: No checkpoint path provided.
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=18|TP=0|ip-26-0-169-139]: No checkpoint path provided.
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=14|TP=0|ip-26-0-169-132]: Local number of parameters: 41.9M (80.01MiB)
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=14|TP=0|ip-26-0-169-132]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=14|TP=0|ip-26-0-169-132]: No checkpoint path provided.
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=41|TP=0|ip-26-0-169-247]: Local number of parameters: 0 (0.00MiB)
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=51|TP=0|ip-26-0-169-86]: Local number of parameters: 0 (0.00MiB)
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=51|TP=0|ip-26-0-169-86]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=51|TP=0|ip-26-0-169-86]: No checkpoint path provided.
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=2|TP=0|ip-26-0-160-192]: Local number of parameters: 41.9M (80.01MiB)
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=23|TP=0|ip-26-0-169-139]: Local number of parameters: 41.9M (80.01MiB)
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=23|TP=0|ip-26-0-169-139]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=10|TP=0|ip-26-0-169-132]: Local number of parameters: 41.9M (80.01MiB)
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=10|TP=0|ip-26-0-169-132]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=10|TP=0|ip-26-0-169-132]: No checkpoint path provided.
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=41|TP=0|ip-26-0-169-247]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=48|TP=0|ip-26-0-169-86]: Local number of parameters: 0 (0.00MiB)
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=48|TP=0|ip-26-0-169-86]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=48|TP=0|ip-26-0-169-86]: No checkpoint path provided.
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=32|TP=0|ip-26-0-169-239]: Local number of parameters: 0 (0.00MiB)
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=32|TP=0|ip-26-0-169-239]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=32|TP=0|ip-26-0-169-239]: No checkpoint path provided.
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=2|TP=0|ip-26-0-160-192]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=23|TP=0|ip-26-0-169-139]: No checkpoint path provided.
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=13|TP=0|ip-26-0-169-132]: Local number of parameters: 41.9M (80.01MiB)
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=13|TP=0|ip-26-0-169-132]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=41|TP=0|ip-26-0-169-247]: No checkpoint path provided.
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=49|TP=0|ip-26-0-169-86]: Local number of parameters: 0 (0.00MiB)
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=35|TP=0|ip-26-0-169-239]: Local number of parameters: 0 (0.00MiB)
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=2|TP=0|ip-26-0-160-192]: No checkpoint path provided.
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=22|TP=0|ip-26-0-169-139]: Local number of parameters: 41.9M (80.01MiB)
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=22|TP=0|ip-26-0-169-139]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=13|TP=0|ip-26-0-169-132]: No checkpoint path provided.
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=42|TP=0|ip-26-0-169-247]: Local number of parameters: 0 (0.00MiB)
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=49|TP=0|ip-26-0-169-86]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=49|TP=0|ip-26-0-169-86]: No checkpoint path provided.
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=35|TP=0|ip-26-0-169-239]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=34|TP=0|ip-26-0-169-239]: Local number of parameters: 0 (0.00MiB)
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=57|TP=0|ip-26-0-170-31]: Local number of parameters: 0 (0.00MiB)
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=1|TP=0|ip-26-0-160-192]: Local number of parameters: 41.9M (80.01MiB)
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=22|TP=0|ip-26-0-169-139]: No checkpoint path provided.
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=12|TP=0|ip-26-0-169-132]: Local number of parameters: 41.9M (80.01MiB)
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=42|TP=0|ip-26-0-169-247]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=42|TP=0|ip-26-0-169-247]: No checkpoint path provided.
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=50|TP=0|ip-26-0-169-86]: Local number of parameters: 0 (0.00MiB)
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=50|TP=0|ip-26-0-169-86]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=35|TP=0|ip-26-0-169-239]: No checkpoint path provided.
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=57|TP=0|ip-26-0-170-31]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=1|TP=0|ip-26-0-160-192]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=1|TP=0|ip-26-0-160-192]: No checkpoint path provided.
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=20|TP=0|ip-26-0-169-139]: Local number of parameters: 41.9M (80.01MiB)
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=20|TP=0|ip-26-0-169-139]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=20|TP=0|ip-26-0-169-139]: No checkpoint path provided.
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=11|TP=0|ip-26-0-169-132]: Local number of parameters: 41.9M (80.01MiB)
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=26|TP=0|ip-26-0-169-207]: Local number of parameters: 0 (0.00MiB)
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=43|TP=0|ip-26-0-169-247]: Local number of parameters: 0 (0.00MiB)
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=50|TP=0|ip-26-0-169-86]: No checkpoint path provided.
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=34|TP=0|ip-26-0-169-239]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=34|TP=0|ip-26-0-169-239]: No checkpoint path provided.
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=57|TP=0|ip-26-0-170-31]: No checkpoint path provided.
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=59|TP=0|ip-26-0-170-31]: Local number of parameters: 0 (0.00MiB)
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=59|TP=0|ip-26-0-170-31]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=3|TP=0|ip-26-0-160-192]: Local number of parameters: 41.9M (80.01MiB)
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=3|TP=0|ip-26-0-160-192]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=3|TP=0|ip-26-0-160-192]: No checkpoint path provided.
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=17|TP=0|ip-26-0-169-139]: Local number of parameters: 41.9M (80.01MiB)
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=17|TP=0|ip-26-0-169-139]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=17|TP=0|ip-26-0-169-139]: No checkpoint path provided.
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=11|TP=0|ip-26-0-169-132]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=11|TP=0|ip-26-0-169-132]: No checkpoint path provided.
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=26|TP=0|ip-26-0-169-207]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=26|TP=0|ip-26-0-169-207]: No checkpoint path provided.
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=43|TP=0|ip-26-0-169-247]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=43|TP=0|ip-26-0-169-247]: No checkpoint path provided.
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=54|TP=0|ip-26-0-169-86]: Local number of parameters: 0 (0.00MiB)
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=54|TP=0|ip-26-0-169-86]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=54|TP=0|ip-26-0-169-86]: No checkpoint path provided.
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=33|TP=0|ip-26-0-169-239]: Local number of parameters: 0 (0.00MiB)
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=62|TP=0|ip-26-0-170-31]: Local number of parameters: 0 (0.00MiB)
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=7|TP=0|ip-26-0-160-192]: Local number of parameters: 41.9M (80.01MiB)
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=7|TP=0|ip-26-0-160-192]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=7|TP=0|ip-26-0-160-192]: No checkpoint path provided.
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=16|TP=0|ip-26-0-169-139]: Local number of parameters: 41.9M (80.01MiB)
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=12|TP=0|ip-26-0-169-132]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=12|TP=0|ip-26-0-169-132]: No checkpoint path provided.
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=31|TP=0|ip-26-0-169-207]: Local number of parameters: 0 (0.00MiB)
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=31|TP=0|ip-26-0-169-207]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=31|TP=0|ip-26-0-169-207]: No checkpoint path provided.
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=44|TP=0|ip-26-0-169-247]: Local number of parameters: 0 (0.00MiB)
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=44|TP=0|ip-26-0-169-247]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=44|TP=0|ip-26-0-169-247]: No checkpoint path provided.
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=52|TP=0|ip-26-0-169-86]: Local number of parameters: 0 (0.00MiB)
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=52|TP=0|ip-26-0-169-86]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=52|TP=0|ip-26-0-169-86]: No checkpoint path provided.
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=33|TP=0|ip-26-0-169-239]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=33|TP=0|ip-26-0-169-239]: No checkpoint path provided.
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=62|TP=0|ip-26-0-170-31]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=59|TP=0|ip-26-0-170-31]: No checkpoint path provided.
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=63|TP=0|ip-26-0-170-31]: Local number of parameters: 0 (0.00MiB)
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Total number of parameters: 1.21G (2312.82MiB)
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Local number of parameters: 145M (276.32MiB)
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [After model building] Memory usage: 277.33MiB. Peak allocated: 279.36MiB Peak reserved: 294.00MiB
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=16|TP=0|ip-26-0-169-139]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=16|TP=0|ip-26-0-169-139]: No checkpoint path provided.
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=9|TP=0|ip-26-0-169-132]: Local number of parameters: 41.9M (80.01MiB)
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=9|TP=0|ip-26-0-169-132]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=25|TP=0|ip-26-0-169-207]: Local number of parameters: 103M (196.32MiB)
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=46|TP=0|ip-26-0-169-247]: Local number of parameters: 0 (0.00MiB)
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=39|TP=0|ip-26-0-169-239]: Local number of parameters: 0 (0.00MiB)
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=39|TP=0|ip-26-0-169-239]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=39|TP=0|ip-26-0-169-239]: No checkpoint path provided.
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=63|TP=0|ip-26-0-170-31]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=62|TP=0|ip-26-0-170-31]: No checkpoint path provided.
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: No checkpoint path provided.
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Parametrizing model parameters using StandardParametrizator
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=21|TP=0|ip-26-0-169-139]: Local number of parameters: 41.9M (80.01MiB)
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=21|TP=0|ip-26-0-169-139]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=9|TP=0|ip-26-0-169-132]: No checkpoint path provided.
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=28|TP=0|ip-26-0-169-207]: Local number of parameters: 0 (0.00MiB)
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=40|TP=0|ip-26-0-169-247]: Local number of parameters: 0 (0.00MiB)
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=40|TP=0|ip-26-0-169-247]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=36|TP=0|ip-26-0-169-239]: Local number of parameters: 0 (0.00MiB)
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=36|TP=0|ip-26-0-169-239]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=63|TP=0|ip-26-0-170-31]: No checkpoint path provided.
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=6|TP=0|ip-26-0-160-192]: Local number of parameters: 41.9M (80.01MiB)
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=6|TP=0|ip-26-0-160-192]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=6|TP=0|ip-26-0-160-192]: No checkpoint path provided.
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=21|TP=0|ip-26-0-169-139]: No checkpoint path provided.
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=15|TP=0|ip-26-0-169-132]: Local number of parameters: 41.9M (80.01MiB)
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=15|TP=0|ip-26-0-169-132]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=28|TP=0|ip-26-0-169-207]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=25|TP=0|ip-26-0-169-207]: [After model building] Memory usage: 196.33MiB. Peak allocated: 196.35MiB Peak reserved: 200.00MiB
[default1]:07/06/2024 09:43:43 [INFO|DP=0|PP=25|TP=0|ip-26-0-169-207]: No checkpoint path provided.
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=40|TP=0|ip-26-0-169-247]: No checkpoint path provided.
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=36|TP=0|ip-26-0-169-239]: No checkpoint path provided.
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=56|TP=0|ip-26-0-170-31]: Local number of parameters: 0 (0.00MiB)
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=56|TP=0|ip-26-0-170-31]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=56|TP=0|ip-26-0-170-31]: No checkpoint path provided.
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=15|TP=0|ip-26-0-169-132]: No checkpoint path provided.
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=28|TP=0|ip-26-0-169-207]: No checkpoint path provided.
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=46|TP=0|ip-26-0-169-247]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=46|TP=0|ip-26-0-169-247]: No checkpoint path provided.
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=55|TP=0|ip-26-0-169-86]: Local number of parameters: 0 (0.00MiB)
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=55|TP=0|ip-26-0-169-86]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=37|TP=0|ip-26-0-169-239]: Local number of parameters: 0 (0.00MiB)
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=61|TP=0|ip-26-0-170-31]: Local number of parameters: 0 (0.00MiB)
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=24|TP=0|ip-26-0-169-207]: Local number of parameters: 2.05K (0.00MiB)
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=47|TP=0|ip-26-0-169-247]: Local number of parameters: 0 (0.00MiB)
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=55|TP=0|ip-26-0-169-86]: No checkpoint path provided.
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=37|TP=0|ip-26-0-169-239]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=37|TP=0|ip-26-0-169-239]: No checkpoint path provided.
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=61|TP=0|ip-26-0-170-31]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=5|TP=0|ip-26-0-160-192]: Local number of parameters: 41.9M (80.01MiB)
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=5|TP=0|ip-26-0-160-192]: [After model building] Memory usage: 81.02MiB. Peak allocated: 83.05MiB Peak reserved: 96.00MiB
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=5|TP=0|ip-26-0-160-192]: No checkpoint path provided.
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=30|TP=0|ip-26-0-169-207]: Local number of parameters: 0 (0.00MiB)
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=47|TP=0|ip-26-0-169-247]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default7]:07/06/2024 09:43:43 [INFO|DP=0|PP=47|TP=0|ip-26-0-169-247]: No checkpoint path provided.
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=38|TP=0|ip-26-0-169-239]: Local number of parameters: 0 (0.00MiB)
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=38|TP=0|ip-26-0-169-239]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=61|TP=0|ip-26-0-170-31]: No checkpoint path provided.
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=30|TP=0|ip-26-0-169-207]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=24|TP=0|ip-26-0-169-207]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=38|TP=0|ip-26-0-169-239]: No checkpoint path provided.
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=58|TP=0|ip-26-0-170-31]: Local number of parameters: 0 (0.00MiB)
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=58|TP=0|ip-26-0-170-31]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default0]:07/06/2024 09:43:43 [INFO|DP=0|PP=24|TP=0|ip-26-0-169-207]: No checkpoint path provided.
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=60|TP=0|ip-26-0-170-31]: Local number of parameters: 0 (0.00MiB)
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=29|TP=0|ip-26-0-169-207]: Local number of parameters: 0 (0.00MiB)
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=60|TP=0|ip-26-0-170-31]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default2]:07/06/2024 09:43:43 [INFO|DP=0|PP=58|TP=0|ip-26-0-170-31]: No checkpoint path provided.
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=29|TP=0|ip-26-0-169-207]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default4]:07/06/2024 09:43:43 [INFO|DP=0|PP=60|TP=0|ip-26-0-170-31]: No checkpoint path provided.
[default6]:07/06/2024 09:43:43 [INFO|DP=0|PP=30|TP=0|ip-26-0-169-207]: No checkpoint path provided.
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=27|TP=0|ip-26-0-169-207]: Local number of parameters: 0 (0.00MiB)
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=27|TP=0|ip-26-0-169-207]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB
[default5]:07/06/2024 09:43:43 [INFO|DP=0|PP=29|TP=0|ip-26-0-169-207]: No checkpoint path provided.
[default3]:07/06/2024 09:43:43 [INFO|DP=0|PP=27|TP=0|ip-26-0-169-207]: No checkpoint path provided.
[default0]:07/06/2024 09:43:44 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [Optimizer Building] Using LearningRateForSP as learning rate
[default0]:07/06/2024 09:43:44 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [ZeRO sharding] Size of optimizer params per rank:
[default0]:07/06/2024 09:43:44 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [ZeRO sharding] DP Rank 0 has 145M out of 145M (100.00%) params' optimizer states
[default1]:Traceback (most recent call last):
[default1]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in <module>
[default1]:    trainer = DistributedTrainer(config_file)
[default1]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 185, in __init__
[default1]:    self.optimizer, self.grad_accumulator = init_optimizer_and_grad_accumulator(
[default1]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/helpers.py", line 401, in init_optimizer_and_grad_accumulator
[default1]:    param = model.get_parameter(optim_model_param_name)
[default1]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 714, in get_parameter
[default1]:    mod: torch.nn.Module = self.get_submodule(module_path)
[default1]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 681, in get_submodule
[default1]:    raise AttributeError(mod._get_name() + " has no "
[default1]:AttributeError: PipelineBlock has no attribute `pp_block`
[default0]:07/06/2024 09:43:45 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [Training Plan] Stage Training Stage has 19 remaining training steps and has consumed 0 samples
[default0]:07/06/2024 09:43:45 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Using `datasets` library
[default0]:07/06/2024 09:43:45 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Loading tokenizer from openai-community/gpt2 and transformers/hf_hub versions ('4.41.2', '0.23.4')
[default0]:07/06/2024 09:43:45 [WARNING|DP=0|PP=0|TP=0|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[2024-07-06 09:43:47,984] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 3368787 closing signal SIGTERM
[2024-07-06 09:43:47,984] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 3368789 closing signal SIGTERM
[2024-07-06 09:43:47,985] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 3368790 closing signal SIGTERM
[2024-07-06 09:43:47,985] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 3368791 closing signal SIGTERM
[2024-07-06 09:43:47,986] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 3368792 closing signal SIGTERM
[2024-07-06 09:43:47,986] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 3368793 closing signal SIGTERM
[2024-07-06 09:43:47,986] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 3368794 closing signal SIGTERM
[2024-07-06 09:43:50,097] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 1 (pid: 3368788) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-06_09:43:47
  host      : ip-26-0-169-207.ec2.internal
  rank      : 25 (local_rank: 1)
  exitcode  : 1 (pid: 3368788)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
srun: error: ip-26-0-169-207: task 4: Exited with exit code 1
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/06/2024 09:43:50 [WARNING|DP=0|PP=8|TP=0|ip-26-0-169-132]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/06/2024 09:43:50 [WARNING|DP=0|PP=13|TP=0|ip-26-0-169-132]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/06/2024 09:43:50 [WARNING|DP=0|PP=18|TP=0|ip-26-0-169-139]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/06/2024 09:43:50 [WARNING|DP=0|PP=10|TP=0|ip-26-0-169-132]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/06/2024 09:43:50 [WARNING|DP=0|PP=19|TP=0|ip-26-0-169-139]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/06/2024 09:43:50 [WARNING|DP=0|PP=9|TP=0|ip-26-0-169-132]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/06/2024 09:43:50 [WARNING|DP=0|PP=23|TP=0|ip-26-0-169-139]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/06/2024 09:43:50 [WARNING|DP=0|PP=11|TP=0|ip-26-0-169-132]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/06/2024 09:43:50 [WARNING|DP=0|PP=12|TP=0|ip-26-0-169-132]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/06/2024 09:43:50 [WARNING|DP=0|PP=20|TP=0|ip-26-0-169-139]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/06/2024 09:43:50 [WARNING|DP=0|PP=15|TP=0|ip-26-0-169-132]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/06/2024 09:43:50 [WARNING|DP=0|PP=22|TP=0|ip-26-0-169-139]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/06/2024 09:43:50 [WARNING|DP=0|PP=17|TP=0|ip-26-0-169-139]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/06/2024 09:43:50 [WARNING|DP=0|PP=21|TP=0|ip-26-0-169-139]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/06/2024 09:43:50 [WARNING|DP=0|PP=16|TP=0|ip-26-0-169-139]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/06/2024 09:43:50 [WARNING|DP=0|PP=14|TP=0|ip-26-0-169-132]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:[rank16]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[default0]:[rank16]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down.
[default0]:[rank16]:[E ProcessGroupNCCL.cpp:1182] [Rank 16] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3
[default0]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.
[default0]:Last error:
[default0]:socketProgress: Connection closed by remote peer ip-26-0-169-207.ec2.internal<58894>
[default0]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first):
[default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f5e98e20d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector<std::shared_ptr<c10d::NCCLComm>, std::allocator<std::shared_ptr<c10d::NCCLComm> > > const&) + 0x2f3 (0x7f5e99fc7fa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7f5e99fc827b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7f5e99fcbc1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f5e99fcc839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #5: <unknown function> + 0xd3e95 (0x7f5ee3cd0e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6)
[default0]:frame #6: <unknown function> + 0x8609 (0x7f5ee8dd8609 in /lib/x86_64-linux-gnu/libpthread.so.0)
[default0]:frame #7: clone + 0x43 (0x7f5ee8ba3353 in /lib/x86_64-linux-gnu/libc.so.6)
[default0]:
[default0]:terminate called after throwing an instance of 'c10::DistBackendError'
[default0]:  what():  [Rank 16] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3
[default0]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.
[default0]:Last error:
[default0]:socketProgress: Connection closed by remote peer ip-26-0-169-207.ec2.internal<58894>
[default0]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first):
[default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f5e98e20d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector<std::shared_ptr<c10d::NCCLComm>, std::allocator<std::shared_ptr<c10d::NCCLComm> > > const&) + 0x2f3 (0x7f5e99fc7fa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7f5e99fc827b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7f5e99fcbc1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f5e99fcc839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #5: <unknown function> + 0xd3e95 (0x7f5ee3cd0e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6)
[default0]:frame #6: <unknown function> + 0x8609 (0x7f5ee8dd8609 in /lib/x86_64-linux-gnu/libpthread.so.0)
[default0]:frame #7: clone + 0x43 (0x7f5ee8ba3353 in /lib/x86_64-linux-gnu/libc.so.6)
[default0]:
[default0]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1186 (most recent call first):
[default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f5e98e20d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:frame #1: <unknown function> + 0xdf6b11 (0x7f5e99d22b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #2: <unknown function> + 0xd3e95 (0x7f5ee3cd0e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6)
[default0]:frame #3: <unknown function> + 0x8609 (0x7f5ee8dd8609 in /lib/x86_64-linux-gnu/libpthread.so.0)
[default0]:frame #4: clone + 0x43 (0x7f5ee8ba3353 in /lib/x86_64-linux-gnu/libc.so.6)
[default0]:
[default0]:07/06/2024 09:43:56 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [Training Plan] There are 1 training stages 
[default0]:07/06/2024 09:43:56 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [Stage Training Stage] start from step 1 
[default0]:07/06/2024 09:43:56 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: 
[default0]:07/06/2024 09:43:56 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [Start training] datetime: 2024-07-06 09:43:56.379157 | mbs: 16 | grad_accum: 64 | global_batch_size: 1024 | sequence_length: 4096 | train_steps: 20 | start_iteration_step: 0 | consumed_train_samples: 0
[default2]:07/06/2024 09:43:56 [WARNING|DP=0|PP=42|TP=0|ip-26-0-169-247]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/06/2024 09:43:56 [WARNING|DP=0|PP=45|TP=0|ip-26-0-169-247]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/06/2024 09:43:56 [WARNING|DP=0|PP=41|TP=0|ip-26-0-169-247]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/06/2024 09:43:56 [WARNING|DP=0|PP=43|TP=0|ip-26-0-169-247]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/06/2024 09:43:56 [WARNING|DP=0|PP=44|TP=0|ip-26-0-169-247]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/06/2024 09:43:56 [WARNING|DP=0|PP=51|TP=0|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/06/2024 09:43:56 [WARNING|DP=0|PP=53|TP=0|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/06/2024 09:43:56 [WARNING|DP=0|PP=2|TP=0|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/06/2024 09:43:56 [WARNING|DP=0|PP=47|TP=0|ip-26-0-169-247]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/06/2024 09:43:56 [WARNING|DP=0|PP=7|TP=0|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/06/2024 09:43:56 [WARNING|DP=0|PP=50|TP=0|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/06/2024 09:43:56 [WARNING|DP=0|PP=54|TP=0|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/06/2024 09:43:56 [WARNING|DP=0|PP=48|TP=0|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/06/2024 09:43:56 [WARNING|DP=0|PP=3|TP=0|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/06/2024 09:43:56 [WARNING|DP=0|PP=40|TP=0|ip-26-0-169-247]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/06/2024 09:43:56 [WARNING|DP=0|PP=4|TP=0|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/06/2024 09:43:56 [WARNING|DP=0|PP=1|TP=0|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/06/2024 09:43:56 [WARNING|DP=0|PP=57|TP=0|ip-26-0-170-31]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/06/2024 09:43:56 [WARNING|DP=0|PP=52|TP=0|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/06/2024 09:43:56 [WARNING|DP=0|PP=6|TP=0|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/06/2024 09:43:56 [WARNING|DP=0|PP=63|TP=0|ip-26-0-170-31]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/06/2024 09:43:56 [WARNING|DP=0|PP=62|TP=0|ip-26-0-170-31]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/06/2024 09:43:56 [WARNING|DP=0|PP=59|TP=0|ip-26-0-170-31]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/06/2024 09:43:56 [WARNING|DP=0|PP=56|TP=0|ip-26-0-170-31]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/06/2024 09:43:56 [WARNING|DP=0|PP=61|TP=0|ip-26-0-170-31]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/06/2024 09:43:56 [WARNING|DP=0|PP=55|TP=0|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/06/2024 09:43:56 [WARNING|DP=0|PP=58|TP=0|ip-26-0-170-31]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/06/2024 09:43:56 [WARNING|DP=0|PP=60|TP=0|ip-26-0-170-31]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/06/2024 09:43:56 [WARNING|DP=0|PP=5|TP=0|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/06/2024 09:43:56 [WARNING|DP=0|PP=46|TP=0|ip-26-0-169-247]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/06/2024 09:43:56 [WARNING|DP=0|PP=49|TP=0|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[2024-07-06 09:43:57,987] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 470112 closing signal SIGTERM
[2024-07-06 09:43:57,987] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 470113 closing signal SIGTERM
[2024-07-06 09:43:57,988] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 470114 closing signal SIGTERM
[2024-07-06 09:43:57,989] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 470115 closing signal SIGTERM
[2024-07-06 09:43:57,989] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 470116 closing signal SIGTERM
[2024-07-06 09:43:57,989] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 470117 closing signal SIGTERM
[2024-07-06 09:43:57,989] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 470118 closing signal SIGTERM
[2024-07-06 09:44:00,011] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -6) local_rank: 0 (pid: 470111) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-06_09:43:57
  host      : ip-26-0-169-139.ec2.internal
  rank      : 16 (local_rank: 0)
  exitcode  : -6 (pid: 470111)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 470111
============================================================
srun: error: ip-26-0-169-139: task 3: Exited with exit code 1
[default1]:[rank33]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[default1]:[rank33]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down.
[default1]:[rank33]:[E ProcessGroupNCCL.cpp:1182] [Rank 33] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3
[default1]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.
[default1]:Last error:
[default1]:socketProgress: Connection closed by remote peer ip-26-0-169-139.ec2.internal<34196>
[default1]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first):
[default1]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fb75c73cd87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default1]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector<std::shared_ptr<c10d::NCCLComm>, std::allocator<std::shared_ptr<c10d::NCCLComm> > > const&) + 0x2f3 (0x7fb75d8e3fa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7fb75d8e427b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7fb75d8e7c1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7fb75d8e8839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:frame #5: <unknown function> + 0xd3e95 (0x7fb7a75ece95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6)
[default1]:frame #6: <unknown function> + 0x8609 (0x7fb7ac6f4609 in /lib/x86_64-linux-gnu/libpthread.so.0)
[default1]:frame #7: clone + 0x43 (0x7fb7ac4bf353 in /lib/x86_64-linux-gnu/libc.so.6)
[default1]:
[default1]:terminate called after throwing an instance of 'c10::DistBackendError'
[default1]:  what():  [Rank 33] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3
[default1]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.
[default1]:Last error:
[default1]:socketProgress: Connection closed by remote peer ip-26-0-169-139.ec2.internal<34196>
[default1]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first):
[default1]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fb75c73cd87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default1]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector<std::shared_ptr<c10d::NCCLComm>, std::allocator<std::shared_ptr<c10d::NCCLComm> > > const&) + 0x2f3 (0x7fb75d8e3fa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7fb75d8e427b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7fb75d8e7c1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7fb75d8e8839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:frame #5: <unknown function> + 0xd3e95 (0x7fb7a75ece95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6)
[default1]:frame #6: <unknown function> + 0x8609 (0x7fb7ac6f4609 in /lib/x86_64-linux-gnu/libpthread.so.0)
[default1]:frame #7: clone + 0x43 (0x7fb7ac4bf353 in /lib/x86_64-linux-gnu/libc.so.6)
[default1]:
[default1]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1186 (most recent call first):
[default1]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fb75c73cd87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default1]:frame #1: <unknown function> + 0xdf6b11 (0x7fb75d63eb11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:frame #2: <unknown function> + 0xd3e95 (0x7fb7a75ece95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6)
[default1]:frame #3: <unknown function> + 0x8609 (0x7fb7ac6f4609 in /lib/x86_64-linux-gnu/libpthread.so.0)
[default1]:frame #4: clone + 0x43 (0x7fb7ac4bf353 in /lib/x86_64-linux-gnu/libc.so.6)
[default1]:
[2024-07-06 09:44:02,993] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 57123 closing signal SIGTERM
[2024-07-06 09:44:02,993] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 57125 closing signal SIGTERM
[2024-07-06 09:44:02,993] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 57126 closing signal SIGTERM
[2024-07-06 09:44:02,995] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 57127 closing signal SIGTERM
[2024-07-06 09:44:02,995] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 57128 closing signal SIGTERM
[2024-07-06 09:44:02,996] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 57129 closing signal SIGTERM
[2024-07-06 09:44:02,996] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 57130 closing signal SIGTERM
[2024-07-06 09:44:04,912] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -6) local_rank: 1 (pid: 57124) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-06_09:44:02
  host      : ip-26-0-169-239.ec2.internal
  rank      : 33 (local_rank: 1)
  exitcode  : -6 (pid: 57124)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 57124
============================================================
srun: error: ip-26-0-169-239: task 5: Exited with exit code 1
[default0]:07/06/2024 09:44:05 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Resuming training from stage Training Stage, it has trained for 0 samples and has 19 remaining train steps
[default0]:07/06/2024 09:44:05 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:  Memory usage: 1382.63MiB. Peak allocated 1382.63MiB. Peak reserved: 1402.00MiB
[default0]:[rank48]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[default0]:[rank48]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down.
[default0]:[rank48]:[E ProcessGroupNCCL.cpp:1182] [Rank 48] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3
[default0]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.
[default0]:Last error:
[default0]:socketProgress: Connection closed by remote peer ip-26-0-169-239.ec2.internal<36542>
[default0]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first):
[default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f86d9d86d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector<std::shared_ptr<c10d::NCCLComm>, std::allocator<std::shared_ptr<c10d::NCCLComm> > > const&) + 0x2f3 (0x7f86daf2dfa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7f86daf2e27b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7f86daf31c1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f86daf32839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #5: <unknown function> + 0xd3e95 (0x7f8724c36e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6)
[default0]:frame #6: <unknown function> + 0x8609 (0x7f8729d3e609 in /lib/x86_64-linux-gnu/libpthread.so.0)
[default0]:frame #7: clone + 0x43 (0x7f8729b09353 in /lib/x86_64-linux-gnu/libc.so.6)
[default0]:
[default0]:terminate called after throwing an instance of 'c10::DistBackendError'
[default0]:  what():  [Rank 48] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3
[default0]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.
[default0]:Last error:
[default0]:socketProgress: Connection closed by remote peer ip-26-0-169-239.ec2.internal<36542>
[default0]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first):
[default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f86d9d86d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector<std::shared_ptr<c10d::NCCLComm>, std::allocator<std::shared_ptr<c10d::NCCLComm> > > const&) + 0x2f3 (0x7f86daf2dfa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7f86daf2e27b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7f86daf31c1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f86daf32839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #5: <unknown function> + 0xd3e95 (0x7f8724c36e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6)
[default0]:frame #6: <unknown function> + 0x8609 (0x7f8729d3e609 in /lib/x86_64-linux-gnu/libpthread.so.0)
[default0]:frame #7: clone + 0x43 (0x7f8729b09353 in /lib/x86_64-linux-gnu/libc.so.6)
[default0]:
[default0]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1186 (most recent call first):
[default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f86d9d86d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:frame #1: <unknown function> + 0xdf6b11 (0x7f86dac88b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #2: <unknown function> + 0xd3e95 (0x7f8724c36e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6)
[default0]:frame #3: <unknown function> + 0x8609 (0x7f8729d3e609 in /lib/x86_64-linux-gnu/libpthread.so.0)
[default0]:frame #4: clone + 0x43 (0x7f8729b09353 in /lib/x86_64-linux-gnu/libc.so.6)
[default0]:
[default0]:[rank0]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[default0]:[rank0]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down.
[default0]:[rank0]:[E ProcessGroupNCCL.cpp:1182] [Rank 0] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3
[default0]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.
[default0]:Last error:
[default0]:socketProgress: Connection closed by remote peer ip-26-0-169-239.ec2.internal<42826>
[default0]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first):
[default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f74890d1d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector<std::shared_ptr<c10d::NCCLComm>, std::allocator<std::shared_ptr<c10d::NCCLComm> > > const&) + 0x2f3 (0x7f748a278fa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7f748a27927b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7f748a27cc1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f748a27d839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #5: <unknown function> + 0xd3e95 (0x7f74d3f81e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6)
[default0]:frame #6: <unknown function> + 0x8609 (0x7f74d9089609 in /lib/x86_64-linux-gnu/libpthread.so.0)
[default0]:frame #7: clone + 0x43 (0x7f74d8e54353 in /lib/x86_64-linux-gnu/libc.so.6)
[default0]:
[default0]:terminate called after throwing an instance of 'c10::DistBackendError'
[default0]:  what():  [Rank 0] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3
[default0]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.
[default0]:Last error:
[default0]:socketProgress: Connection closed by remote peer ip-26-0-169-239.ec2.internal<42826>
[default0]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first):
[default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f74890d1d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector<std::shared_ptr<c10d::NCCLComm>, std::allocator<std::shared_ptr<c10d::NCCLComm> > > const&) + 0x2f3 (0x7f748a278fa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7f748a27927b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7f748a27cc1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f748a27d839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #5: <unknown function> + 0xd3e95 (0x7f74d3f81e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6)
[default0]:frame #6: <unknown function> + 0x8609 (0x7f74d9089609 in /lib/x86_64-linux-gnu/libpthread.so.0)
[default0]:frame #7: clone + 0x43 (0x7f74d8e54353 in /lib/x86_64-linux-gnu/libc.so.6)
[default0]:
[default0]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1186 (most recent call first):
[default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f74890d1d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:frame #1: <unknown function> + 0xdf6b11 (0x7f7489fd3b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #2: <unknown function> + 0xd3e95 (0x7f74d3f81e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6)
[default0]:frame #3: <unknown function> + 0x8609 (0x7f74d9089609 in /lib/x86_64-linux-gnu/libpthread.so.0)
[default0]:frame #4: clone + 0x43 (0x7f74d8e54353 in /lib/x86_64-linux-gnu/libc.so.6)
[default0]:
[default0]:Traceback (most recent call last):
[default0]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:    trainer.train(dataloader)
[default0]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train
[default0]:    outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default0]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step
[default0]:    outputs = self.pipeline_engine.train_batch_iter(
[default0]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter
[default0]:    output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:    output = model(**micro_batch)
[default0]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default0]:    return self._call_impl(*args, **kwargs)
[default0]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default0]:    return forward_call(*args, **kwargs)
[default0]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward
[default0]:    sharded_logits = self.model(
[default0]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default0]:    return self._call_impl(*args, **kwargs)
[default0]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default0]:    return forward_call(*args, **kwargs)
[default0]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default0]:    return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default0]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default0]:    hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default0]:    return self._call_impl(*args, **kwargs)
[default0]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default0]:    return forward_call(*args, **kwargs)
[default0]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default0]:    new_kwargs[name] = recv_from_pipeline_state_buffer(
[default0]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default0]:    pipeline_state.run_communication()
[default0]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default0]:    recv_activation_tensor = recv_activation()
[default0]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default0]:    return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default0]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p[default6]:Traceback (most recent call last):
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:    trainer.train(dataloader)
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train
[default6]:    outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step
[default6]:    outputs = self.pipeline_engine.train_batch_iter(
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter
[default6]:    output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:    output = 2p.py", line 353, in recv_tensors
model(**micro_batch)
[default6]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default6]:    return self._call_impl(*args, **kwargs)
[default0]:    buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default0]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default0]:    meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default0]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default0]:    dist.recv(
[default0]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
[default0]:    return func(*args, **kwargs)
[default0]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv
[default0]:    pg.recv([tensor], group_src_rank, tag).wait()
[default0]:torch.distributed.DistBackendError: [8] is setting up NCCL communicator and retrieving ncclUnique[default6]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default6]:    return forward_call(*args, **kwargs)
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward
[default6]:    sharded_logits = self.model(
[default6]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default6]:    return self._call_impl(*args, **kwargs)
[default6]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default6]:    return forward_call(*args, **kwargs)
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:    return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
Id from [0] via c10d key-value store by key '7:8', but store->get('7:8') got error: Connection reset by peer
[default0]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first):
[default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fc7b0a9bd87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:    hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default6]:    return self._call_impl(*args, **kwargs)
[default6]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default6]:    return forward_call(*args, **kwargs)
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default6]:    new_kwargs[name] = recv_from_pipeline_state_buffer(
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_[default0]:frame #1: <unknown function> + 0x589518e (0x7fc7e8a5518e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x360 (0x7fc7e8a4f9a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fc7e8a4fce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fc7e8a50b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
buffer
[default6]:    pipeline_state.run_communication()
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default6]:    recv_activation_tensor = recv_activation()
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default6]:    return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default6]:    buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default6]:    meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default0]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc7e8a05f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc7e8a05f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc7e8a05f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default0]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc7e8a05f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:    dist.recv(
[default0]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fc7b1c43c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector<c10::Device, std::allocator<c10::Device> > const&, c10d::OpType, int, bool) + 0x22b (0x7fc7b1c4ac5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x550 (0x7fc7b1c6db60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
[default0]:frame #12: <unknown function> + 0x5838439 (0x7fc7e89f8439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:frame #13: <unknown function> + 0x5843330 (0x7fc7e8a03330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:    return func(*args, **kwargs)
[default0]:frame #14: <unknown function> + 0x58433c5 (0x7fc7e8a033c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:frame #15: <unknown function> + 0x4e893cc (0x7fc7e80493cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:frame #16: <unknown function> + 0x1a08a88 (0x7fc7e4bc8a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:Traceback (most recent call last):
[default0]:frame #17: <unknown function> + 0x5849a84 (0x7fc7e8a09a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv
[default0]:frame #18: <unknown function> + 0x584ed35 (0x7fc7e8a0ed35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:frame #19: <unknown function> + 0xc97eee (0x7fc7fb2c0eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:frame #20: <unknown function> + 0x413ea4 (0x7fc7faa3cea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:    pg.recv([tensor], group_src_rank, tag).wait()
[default0]:frame #21: <unknown function> + 0x1445a6 (0x556b82f4e5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #22: _PyObject_MakeTpCall + 0x26b (0x556b82f47a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:torch.distributed.DistBackendError: [6] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '5:6', but store->get('5:6') got error: Connection reset by peer
[default5]:    trainer.train(dataloader)
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train
[default0]:frame #23: <unknown function> + 0x150866 (0x556b82f5a866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x556b82f43142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #25: _PyFunction_Vectorcall + 0x6c (0x556b82f4ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #26: PyObject_Call + 0xbc (0x556b82f5af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x556b82f412b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #28: _PyFunction_Vectorcall + 0x6c (0x556b82f4ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x556b82f3f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #30: <unknown function> + 0x150[default6]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first):
[default6]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fcb8c292d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default5]:    outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:frame #1: <unknown function> + 0x589518e (0x7fcbc424c18e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step
582 (0x556b82f5a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x360 (0x7fcbc42469a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:    outputs = self.pipeline_engine.train_batch_iter(
[default6]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fcbc4246ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter
[default6]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fcbc4247b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x556b82f3f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #32: <unknown function> + 0x150582 (0x556b82f5a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x556b82f3f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #34: <unknown function> + 0x150582 (0x556b82f5a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x556b82f3f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:    output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x556b82f46f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #37: _PyObject_Call_Prepend + 0x69 (0x556b82f58c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #38: <unknown function> + 0x211239 (0x556b8301b239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fcbc41fcf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:frame #39: _PyObject_MakeTpCall + 0x26b (0x556b82f47a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x556b82f433e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #41: _PyFunction_Vectorcall + 0x6c (0x556b82f4ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x556b82f3ec5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:frame #43: _PyFunction_Vectorcall + 0x6c (0x556b82f4ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x556b82f3f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #45: <unknown function> + 0x150582 (0x556b82f5a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #46: PyObject_Call + 0xbc (0x556b82f5af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:    output = model(**micro_batch)
[default0]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x556b82f412b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #48: <unknown function> + 0x150582 (0x556b82f5a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #49: PyObject_Call + 0xbc (0x556b82f5af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fcbc41fcf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fcbc41fcf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fcbc41fcf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x556b82f412b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #51: _PyFunction_Vectorcall + 0x6c (0x556b82f4ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fcb8d43ac69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x556b82f47007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #53: _PyObject_Call_Prepend + 0x69 (0x556b82f58c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #54: <unknown function> + 0x211239 (0x556b8301b239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #55: PyObject_Call + 0x207 (0x556b82f5b067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default0]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x556b82f412b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #57: <unknown function> + 0x150582 (0x556b82f5a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x556b82f3f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector<c10::Device, std::allocator<c10::Device> > const&, c10d::OpType, int, bool) + 0x22b (0x7fcb8d441c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #59: <unknown function> + 0x150582 (0x556b82f5a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #60: PyObject_Call + 0xbc (0x556b82f5af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:    return self._call_impl(*args, **kwargs)
[default0]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x556b82f412b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #62: <unknown function> + 0x150582 (0x556b82f5a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:frame #63: PyObject_Call + 0xbc (0x556b82f5af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:. This may indicate a possible application crash on rank 0 or a network set up issue.
[default6]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x550 (0x7fcb8d464b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:Traceback (most recent call last):
[default3]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default6]:frame #12: <unknown function> + 0x5838439 (0x7fcbc41ef439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:    trainer.train(dataloader)
[default5]:    return forward_call(*args, **kwargs)
[default3]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train
[default3]:    outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step
[default3]:    outputs = self.pipeline_engine.train_batch_iter(
[default3]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter
[default3]:    output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default3]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default3]:    output = model(**micro_batch)
[default3]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default6]:frame #13: <unknown function> + 0x5843330 (0x7fcbc41fa330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:    return self._call_impl(*args, **kwargs)
[default3]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default3]:    return forward_call(*args, **kwargs)
[default3]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward
[default3]:    sharded_logits = self.model(
[default3]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default3]:    return self._call_impl(*args, **kwargs)
[default6]:frame #14: <unknown function> + 0x58433c5 (0x7fcbc41fa3c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default3]:    return forward_call(*args, **kwargs)
[default5]:    sharded_logits = self.model(
[default3]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:    return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default3]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:frame #15: <unknown function> + 0x4e893cc (0x7fcbc38403cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:    hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default3]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default3]:    return self._call_impl(*args, **kwargs)
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default5]:    return self._call_impl(*args, **kwargs)
[default6]:frame #16: <unknown function> + 0x1a08a88 (0x7fcbc03bfa88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default3]:    return forward_call(*args, **kwargs)
[default3]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default3]:    new_kwargs[name] = recv_from_pipeline_state_buffer(
[default3]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default6]:frame #17: <unknown function> + 0x5849a84 (0x7fcbc4200a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:    pipeline_state.run_communication()
[default3]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default3]:    recv_activation_tensor = recv_activation()
[default3]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default5]:    return forward_call(*args, **kwargs)
[default3]:    return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default3]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default6]:frame #18: <unknown function> + 0x584ed35 (0x7fcbc4205d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:    buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default3]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default3]:    meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default3]:    dist.recv(
[default3]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
[default6]:frame #19: <unknown function> + 0xc97eee (0x7fcbd6ab7eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:frame #20: <unknown function> + 0x413ea4 (0x7fcbd6233ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default3]:    return func(*args, **kwargs)
[default3]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv
[default5]:    return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default3]:    pg.recv([tensor], group_src_rank, tag).wait()
[default3]:torch.distributed.DistBackendError: [11] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '10:11', but store->get('10:11') got error: Connection reset by peer
[default6]:frame #21: <unknown function> + 0x1445a6 (0x5608e106a5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first):
[default3]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f77be777d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default3]:frame #1: <unknown function> + 0x589518e (0x7f77f673118e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x360 (0x7f77f672b9a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f77f672bce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #22: _PyObject_MakeTpCall + 0x26b (0x5608e1063a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f77f672cb11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f77f66e1f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f77f66e1f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f77f66e1f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f77f66e1f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:frame #9: c10d::ProcessGro[default5]:    hidden_encoder_states = encoder_block(**hidden_encoder_states)
upNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f77bf91fc69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector<c10::Device, std::allocator<c10::Device> > const&, c10d::OpType, int, bool) + 0x22b (0x7f77bf926c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:frame #23: <unknown function> + 0x150866 (0x5608e1076866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x550 (0x7f77bf949b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:frame #12: <unknown function> + 0x5838439 (0x7f77f66d4439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:frame #13: <unknown function> + 0x5843330 (0x7f77f66df330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:frame #14: <unknown function> + 0x58433c5 (0x7f77f66df3c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:frame #15: <unknown function> + 0x4e893cc (0x7f77f5d253cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default3]:frame #16: <unknown function> + 0x1a08a88 (0x7f77f28a4a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:frame #17: <unknown function> + 0x5849a84 (0x7f77f66e5a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5608e105f142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #18: <unknown function> + 0x584ed35 (0x7f77f66ead35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:frame #19: <unknown function> + 0xc97eee (0x7f7808f9ceee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default3]:frame #20: <unknown function> + 0x413ea4 (0x7f7808718ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:    return self._call_impl(*args, **kwargs)
[default3]:frame #21: <unknown function> + 0x1445a6 (0x5602d0f685a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #22: _PyObject_MakeTpCall + 0x26b (0x5602d0f61a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #25: _PyFunction_Vectorcall + 0x6c (0x5608e106aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default3]:frame #23: <unknown function> + 0x150866 (0x5602d0f74866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5602d0f5d142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #25: _PyFunction_Vectorcall + 0x6c (0x5602d0f68a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #26: PyObject_Call + 0xbc (0x5602d0f74f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5602d0f5b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #28: _PyFunction_Vectorcall + 0x6c (0x5602d0f68a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #26: PyObject_Call + 0xbc (0x5608e1076f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5602d0f598fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #30: <unknown function> + 0x150582 (0x5602d0f74582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:    return forward_call(*args, **kwargs)
[default3]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5602d0f598fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #32: <unknown function> + 0x150582 (0x5602d0f74582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5602d0f598fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5608e105d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default6]:frame #28: _PyFunction_Vectorcall + 0x6c (0x5608e106aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #34: <unknown function> + 0x150582 (0x5602d0f74582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5602d0f598fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5602d0f60f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #37: _PyObject_Call_Prepend + 0x69 (0x5602d0f72c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:    new_kwargs[name] = recv_from_pipeline_state_buffer(
[default3]:frame #38: <unknown function> + 0x211239 (0x5602d1035239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #39: _PyObject_MakeTpCall + 0x26b (0x5602d0f61a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5602d0f5d3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5608e105b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #41: _PyFunction_Vectorcall + 0x6c (0x5602d0f68a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5602d0f58c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default5]:    pipeline_state.run_communication()
[default3]:frame #43: _PyFunction_Vectorcall + 0x6c (0x5602d0f68a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5602d0f598fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #45: <unknown function> + 0x150582 (0x5602d0f74582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #30: <unknown function> + 0x150582 (0x5608e1076582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #46: PyObject_Call + 0xbc (0x5602d0f74f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5602d0f5b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #48: <unknown function> + 0x150582 (0x5602d0f74582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #49: PyObject_Call + 0xbc (0x5602d0f74f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5602d0f5b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #51: _PyFunction_Vectorcall + 0x6c (0x5602d0f68a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5602d0f61007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #53: _PyObject_Call_Prepend + 0x69 (0[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
x5602d0f72c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #54: <unknown function> + 0x211239 (0x5602d1035239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #55: PyObject_Call + 0x207 (0x5602d0f75067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5608e105b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5602d0f5b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #57: <unknown function> + 0x150582 (0x5602d0f74582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5602d0f598fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:    recv_activation_tensor = recv_activation()
[default3]:frame #59: <unknown function> + 0x150582 (0x5602d0f74582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #60: PyObject_Call + 0xbc (0x5602d0f74f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5602d0f5b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #32: <unknown function> + 0x150582 (0x5608e1076582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #62: <unknown function> + 0x150582 (0x5602d0f74582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:frame #63: PyObject_Call + 0xbc (0x5602d0f74f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:. This may indicate a possible application crash on rank 0 or a network set up issue.
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default5]:Traceback (most recent call last):
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:    trainer.train(dataloader)
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train
[default5]:    outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step
[default5]:    outputs = self.pipeline_engine.train_batch_iter(
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter
[default5]:    output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:    output = [default6]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5608e105b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
model(**micro_batch)
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default5]:    return self._call_impl(*args, **kwargs)
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default5]:    return forward_call(*args, **kwargs)
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward
[default5]:    return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default5]:    sharded_logits = self.model(
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default5]:    return self._call_impl(*args, **kwargs)
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default5]:    return forward_call(*args, **kwargs)
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default5]:    return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:    hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/[default6]:frame #34: <unknown function> + 0x150582 (0x5608e1076582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
modules/module.py", line 1511, in _wrapped_call_impl
[default5]:    return self._call_impl(*args, **kwargs)
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default5]:    return forward_call(*args, **kwargs)
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default5]:    new_kwargs[name] = recv_from_pipeline_state_buffer(
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default5]:    pipeline_state.run_communication()
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default5]:    recv_activation_tensor = recv_activation()
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_clust[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
er/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default5]:    return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default5]:    buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default5]:    meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default5]:    dist.recv(
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
[default5]:    return func(*args, **kwargs)
[default[default6]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5608e105b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv
[default5]:    buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default5]:    pg.recv([tensor], group_src_rank, tag).wait()
[default5]:torch.distributed.DistBackendError: [13] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '12:13', but store->get('12:13') got error: Connection reset by peer
[default5]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first):
[default5]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f55f16c9d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default5]:frame #1: <unknown function> + 0x589518e (0x7f562968318e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x360 (0x7f562967d9a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/l[default6]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5608e1062f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
ibtorch_cpu.so)
[default5]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f562967dce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f562967eb11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5629633f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5629633f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5629633f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:frame #8: c[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5629633f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f55f2871c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector<c10::Device, std::allocator<c10::Device> > const&, c10d::OpType, int, bool) + 0x22b (0x7f55f2878c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x550 (0x7f55f289bb60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:frame #12: <unknown function> + 0x5[default6]:frame #37: _PyObject_Call_Prepend + 0x69 (0x5608e1074c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
838439 (0x7f5629626439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:frame #13: <unknown function> + 0x5843330 (0x7f5629631330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:frame #14: <unknown function> + 0x58433c5 (0x7f56296313c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:frame #15: <unknown function> + 0x4e893cc (0x7f5628c773cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:    meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default5]:frame #16: <unknown function> + 0x1a08a88 (0x7f56257f6a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:frame #17: <unknown function> + 0x5849a84 (0x7f5629637a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #38: <unknown function> + 0x211239 (0x5608e1137239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #18: <unknown function> + 0x584ed35 (0x7f562963cd35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:frame #19: <unknown function> + 0xc97eee (0x7f563beeeeee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default5]:frame #20: <unknown function> + 0x413ea4 (0x7f563b66aea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:frame #21: <unknown function> + 0x1445a6 (0x55ffdeb8e5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #22: _PyObject_MakeTpCall + 0x26b (0x55ffdeb87a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #39: _PyObject_MakeTpCall + 0x26b (0x5608e1063a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #23: <unknown function> + 0x150866 (0x55ffdeb9a866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55ffdeb83142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #25: _PyFunction_Vectorcall + 0x6c (0x55ffdeb8ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:    dist.recv(
[default5]:frame #26: PyObject_Call + 0xbc (0x55ffdeb9af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55ffdeb812b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5608e105f3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #28: _PyFunction_Vectorcall + 0x6c (0x55ffdeb8ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55ffdeb7f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #30: <unknown function> + 0x150582 (0x55ffdeb9a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
[default5]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55ffdeb7f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #32: <unknown function> + 0x150582 (0x55ffdeb9a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55ffdeb7f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #34: <unknown function> + 0x150582 (0x55ffdeb9a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55ffdeb7f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55ffdeb86f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #37: _PyObject_Call_Prepend + 0x69 (0x55ffdeb98c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #38: <unknown fu[default6]:frame #41: _PyFunction_Vectorcall + 0x6c (0x5608e106aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5608e105ac5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
nction> + 0x211239 (0x55ffdec5b239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #39: _PyObject_MakeTpCall + 0x26b (0x55ffdeb87a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55ffdeb833e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #41: _PyFunction_Vectorcall + 0x6c (0x55ffdeb8ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55ffdeb7ec5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #43: _PyFunction_Vectorcall + 0x6c (0x55ffdeb8ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #43: _PyFunction_Vectorcall + 0x6c (0x5608e106aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55ffdeb7f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #45: <unknown function> + 0x150582 (0x55ffdeb9a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #46: PyObject_Call + 0xbc (0x55ffdeb9af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55ffdeb812b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5608e105b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #45: <unknown function> + 0x150582 (0x5608e1076582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #48: <unknown function> + 0x150582 (0x55ffdeb9a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #49: PyObject_Call + 0xbc (0x55ffdeb9af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55ffdeb812b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #51: _PyFunction_Vectorcall + 0x6c (0x55ffdeb8ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55ffdeb87007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #53: _PyObject_Call_Prepend + 0x69 (0x55ffdeb98c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #54: <unknown function> + 0x211239 (0x55ffdec5b239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #55: PyObject_Call + 0x207 (0x55[default6]:frame #46: PyObject_Call + 0xbc (0x5608e1076f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5608e105d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
ffdeb9b067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55ffdeb812b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:    return func(*args, **kwargs)
[default5]:frame #57: <unknown function> + 0x150582 (0x55ffdeb9a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55ffdeb7f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #59: <unknown function> + 0x150582 (0x55ffdeb9a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv
[default5]:frame #60: PyObject_Call + 0xbc (0x55ffdeb9af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55ffdeb812b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #62: <unknown function> + 0x150582 (0x55ffdeb9a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #48: <unknown function> + 0x150582 (0x5608e1076582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #63: PyObject_Call + 0xbc (0x55ffdeb9af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:. This may indicate a possible application crash on rank 0 or a network set up issue.
[default5]:    pg.recv([tensor], group_src_rank, tag).wait()
[default2]:Traceback (most recent call last):
[default2]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:torch.distributed.DistBackendError: [5] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '4:5', but store->get('4:5') got error: Connection reset by peer
[default2]:    trainer.train(dataloader)
[default2]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train
[default5]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first):
[default2]:    outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step
[default2]:    outputs = self.pipeline_engine.train_batch_iter(
[default2]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter
[default2]:    output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:    output = model(**micro_batch)
[default2]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default2]:    return self._call_impl(*args, **kwargs)
[default2]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/py[default5]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f36f04bed87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
thon3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default2]:    return forward_call(*args, **kwargs)
[default5]:frame #1: <unknown function> + 0x589518e (0x7f372847818e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward
[default2]:    sharded_logits = self.model(
[default2]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default2]:    return self._call_impl(*args, **kwargs)
[default5]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x360 (0x7f37284729a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #49: PyObject_Call + 0xbc (0x5608e1076f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5608e105d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default2]:    return forward_call(*args, **kwargs)
[default2]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:frame #51: _PyFunction_Vectorcall + 0x6c (0x5608e106aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5608e1063007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:    return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default2]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default2]:    hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default2]:    return self._call_impl(*args, **kwargs)
[default6]:frame #53: _PyObject_Call_Prepend + 0x69 (0x5608e1074c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:Traceback (most recent call last):
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default2]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default2]:    return forward_call(*args, **kwargs)
[default2]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default5]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f3728472ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:    new_kwargs[name] = recv_from_pipeline_state_buffer(
[default2]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default4]:    trainer.train(dataloader)
[default2]:    pipeline_state.run_communication()
[default2]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default2]:    recv_activation_tensor = recv_activation()
[default2]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default2]:    return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train
[default2]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default2]:    buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default2]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default2]:    meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default2]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default2]:    dist.recv(
[default2]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
[default2]:    return func(*args, **kwargs)
[default5]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f3728473b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv
[default2]:    pg.recv([tensor], group_src_rank, tag).wait()
[default2]:torch.distributed.DistBackendError: [10] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '9:10', but store->get('9:10') got error: Connection reset by peer
[default4]:    outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first):
[default2]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7ff36e586d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default2]:frame #1: <unknown function> + 0x589518e (0x7ff3a654018e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3728428f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x360 (0x7ff3a653a9a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7ff3a653ace2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step
[default2]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7ff3a653bb11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff3a64f0f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff3a64f0f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3728428f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #54: <unknown function> + 0x211239 (0x5608e1137239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:    outputs = self.pipeline_engine.train_batch_iter(
[default2]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff3a64f0f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff3a64f0f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7ff36f72ec69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector<c10::Device, std::allocator<c10::Device> > const&, c10d::OpType, int, bool) + 0x22b (0x7ff36f735c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std[default6]:frame #55: PyObject_Call + 0x207 (0x5608e1077067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
::allocator<at::Tensor> >&, int, int) + 0x550 (0x7ff36f758b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:frame #12: <unknown function> + 0x5838439 (0x7ff3a64e3439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:frame #13: <unknown function> + 0x5843330 (0x7ff3a64ee330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:frame #14: <unknown function> + 0x58433c5 (0x7ff3a64ee3c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:frame #15: <unknown function> + 0x4e893cc (0x7ff3a5b343cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:frame #16: <unknown function> + 0x1a08a88 (0x7ff3a26b3a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/py[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter
thon3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5608e105d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #17: <unknown function> + 0x5849a84 (0x7ff3a64f4a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3728428f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:frame #18: <unknown function> + 0x584ed35 (0x7ff3a64f9d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:    output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:frame #19: <unknown function> + 0xc97eee (0x7ff3b8dabeee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default2]:frame #20: <unknown function> + 0x413ea4 (0x7ff3b8527ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default2]:frame #21: <unknown function> + 0x1445a6 (0x563fd7ac85a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #22: _PyObject_MakeTpCall + 0x26b (0x563fd7ac1a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #23: <unknown function> + 0x150866 (0x563fd7ad4866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3728428f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x563fd7abd142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #25: _PyFunction_Vectorcall + 0x6c (0x563fd7ac8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #26: PyObject_Call + 0xbc (0x563fd7ad4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x563fd7abb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #28: _PyFunction_Vectorcall + 0x6c (0x563fd7ac8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x563fd7ab98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #30: <unknown function> + 0x150582 (0x563fd7ad4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:    output = model(**micro_batch)
[default6]:frame #57: <unknown function> + 0x150582 (0x5608e1076582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5608e105b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x563fd7ab98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #32: <unknown function> + 0x150582 (0x563fd7ad4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x563fd7ab98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #34: <unknown function> + 0x150582 (0x563fd7ad4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default5]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f36f1666c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x563fd7ab98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x563fd7ac0f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #37: _PyObject_Call_Prepend + 0x69 (0x563fd7ad2c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #38: <unknown function> + 0x211239 (0x563fd7b95239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #59: <unknown function> + 0x150582 (0x5608e1076582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #39: _PyObject_MakeTpCall + 0x26b (0x563fd7ac1a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x563fd7abd3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #41: _PyFunction_Vectorcall + 0x6c (0x563fd7ac8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x563fd7ab8c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:    return self._call_impl(*args, **kwargs)
[default5]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector<c10::Device, std::allocator<c10::Device> > const&, c10d::OpType, int, bool) + 0x22b (0x7f36f166dc5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:frame #43: _PyFunction_Vectorcall + 0x6c (0x563fd7ac8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x563fd7ab98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #45: <unknown function> + 0x150582 (0x563fd7ad4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default2]:frame #46: PyObject_Call + 0xbc (0x563fd7ad4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x563fd7abb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #48: <unknown function> + 0x150582 (0x563fd7ad4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #49: PyObject_Call + 0xbc (0x563fd7ad4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x550 (0x7f36f1690b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x563fd7abb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #51: _PyFunction_Vectorcall + 0x6c (0x563fd7ac8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x563fd7ac1007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #53: _PyObject_Call_Prepend + 0x69 (0x563fd7ad2c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #54: <unknown function> + 0x211239 (0x563fd7b95239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #55: PyObject_Call + 0x207 (0x563fd7ad5067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x563fd7abb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #57: <unknown function> + 0[default4]:    return forward_call(*args, **kwargs)
x150582 (0x563fd7ad4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x563fd7ab98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #59: <unknown function> + 0x150582 (0x563fd7ad4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #60: PyObject_Call + 0xbc (0x563fd7ad4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x563fd7abb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #62: <unknown function> + 0x150582 (0x563fd7ad4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:frame #63: PyObject_Call + 0xbc (0x563fd7ad4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:. This may indicate a possible application crash on rank 0 or a network set up issue.
[default5]:frame #12: <unknown function> + 0x5838439 (0x7f372841b439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:Traceback (most recent call last):
[default1]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default1]:    trainer.train(dataloader)
[default1]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train
[default6]:frame #60: PyObject_Call + 0xbc (0x5608e1076f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:Traceback (most recent call last):
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:    trainer.train(dataloader)
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train
[default1]:    outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default1]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward
[default4]:    outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step
[default6]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5608e105d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:    outputs = self.pipeline_engine.train_batch_iter(
[default5]:frame #13: <unknown function> + 0x5843330 (0x7f3728426330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:frame #14: <unknown function> + 0x58433c5 (0x7f37284263c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:    outputs = self.pipeline_engine.train_batch_iter(
[default6]:frame #62: <unknown function> + 0x150582 (0x5608e1076582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:    sharded_logits = self.model(
[default5]:frame #15: <unknown function> + 0x4e893cc (0x7f3727a6c3cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #63: PyObject_Call + 0xbc (0x5608e1076f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter
[default1]:    output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:frame #16: <unknown function> + 0x1a08a88 (0x7f37245eba88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:frame #17: <unknown function> + 0x5849a84 (0x7f372842ca84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter
[default4]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default1]:    output = model(**micro_batch)
[default5]:frame #18: <unknown function> + 0x584ed35 (0x7f3728431d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:frame #19: <unknown function> + 0xc97eee (0x7f373ace3eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:    output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:frame #20: <unknown function> + 0x413ea4 (0x7f373a45fea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:. This may indicate a possible application crash on rank 0 or a network set up issue.
[default1]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default4]:    return self._call_impl(*args, **kwargs)
[default1]:    return self._call_impl(*args, **kwargs)
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default4]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default4]:    return forward_call(*args, **kwargs)
[default4]:    output = model(**micro_batch)
[default5]:frame #21: <unknown function> + 0x1445a6 (0x55edcedcd5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default5]:frame #22: _PyObject_MakeTpCall + 0x26b (0x55edcedc6a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #23: <unknown function> + 0x150866 (0x55edcedd9866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default1]:    return forward_call(*args, **kwargs)
[default5]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55edcedc2142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #25: _PyFunction_Vectorcall + 0x6c (0x55edcedcda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward
[default5]:frame #26: PyObject_Call + 0xbc (0x55edcedd9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:    return self._call_impl(*args, **kwargs)
[default4]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default5]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55edcedc02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #28: _PyFunction_Vectorcall + 0x6c (0x55edcedcda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55edcedbe8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:    sharded_logits = self.model(
[default5]:frame #30: <unknown function> + 0x150582 (0x55edcedd9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55edcedbe8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:    return forward_call(*args, **kwargs)
[default5]:frame #32: <unknown function> + 0x150582 (0x55edcedd9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55edcedbe8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward
[default4]:    return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:    sharded_logits = self.model(
[default5]:frame #34: <unknown function> + 0x150582 (0x55edcedd9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default1]:    return self._call_impl(*args, **kwargs)
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:    return self._call_impl(*args, **kwargs)
[default4]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default4]:    hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:    return forward_call(*args, **kwargs)
[default5]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55edcedbe8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default4]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default1]:    return forward_call(*args, **kwargs)
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default4]:    return self._call_impl(*args, **kwargs)
[default4]:    return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default1]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default5]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55edcedc5f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #37: _PyObject_Call_Prepend + 0x69 (0x55edcedd7c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:    return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:frame #38: <unknown function> + 0x211239 (0x55edcee9a239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #39: _PyObject_MakeTpCall + 0x26b (0x55edcedc6a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:    return forward_call(*args, **kwargs)
[default1]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55edcedc23e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #41: _PyFunction_Vectorcall + 0x6c (0x55edcedcda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:    hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:    hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55edcedbdc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default5]:frame #43: _PyFunction_Vectorcall + 0x6c (0x55edcedcda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55edcedbe8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:    return self._call_impl(*args, **kwargs)
[default1]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default1]:    return self._call_impl(*args, **kwargs)
[default1]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default1]:    return forward_call(*args, **kwargs)
[default4]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default4]:    return forward_call(*args, **kwargs)
[default5]:frame #45: <unknown function> + 0x150582 (0x55edcedd9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default4]:    new_kwargs[name] = recv_from_pipeline_state_buffer(
[default5]:frame #46: PyObject_Call + 0xbc (0x55edcedd9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55edcedc02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #48: <unknown function> + 0x150582 (0x55edcedd9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #49: PyObject_Call + 0xbc (0x55edcedd9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default1]:    new_kwargs[name] = recv_from_pipeline_state_buffer(
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default5]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55edcedc02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:    pipeline_state.run_communication()
[default4]:    new_kwargs[name] = recv_from_pipeline_state_buffer(
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default1]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default4]:    recv_activation_tensor = recv_activation()
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default4]:    pipeline_state.run_communication()
[default4]:    return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default4]:    recv_activation_tensor = recv_activation()
[default1]:    pipeline_state.run_communication()
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default1]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default4]:    return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default1]:    recv_activation_tensor = recv_activation()
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default5]:frame #51: _PyFunction_Vectorcall + 0x6c (0x55edcedcda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55edcedc6007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default1]:    return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default4]:    buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default1]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default4]:    buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default4]:    meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default1]:    buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default1]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default5]:frame #53: _PyObject_Call_Prepend + 0x69 (0x55edcedd7c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default5]:frame #54: <unknown function> + 0x211239 (0x55edcee9a239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:    dist.recv(
[default4]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
[default4]:    meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default4]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default5]:frame #55: PyObject_Call + 0x207 (0x55edcedda067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55edcedc02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #57: <unknown function> + 0x150582 (0x55edcedd9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:    dist.recv(
[default5]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55edcedbe8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #59: <unknown function> + 0x150582 (0x55edcedd9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:    meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default1]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default5]:frame #60: PyObject_Call + 0xbc (0x55edcedd9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55edcedc02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
[default4]:    return func(*args, **kwargs)
[default4]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv
[default4]:    return func(*args, **kwargs)
[default4]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv
[default4]:    pg.recv([tensor], group_src_rank, tag).wait()
[default4]:    pg.recv([tensor], group_src_rank, tag).wait()
[default5]:frame #62: <unknown function> + 0x150582 (0x55edcedd9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:torch.distributed.DistBackendError: [12] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '11:12', but store->get('11:12') got error: Connection reset by peer
[default5]:frame #63: PyObject_Call + 0xbc (0x55edcedd9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first):
[default5]:. This may indicate a possible application crash on rank 0 or a network set up issue.
[default4]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f6e13f71d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default4]:torch.distributed.DistBackendError: [4] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '3:4', but store->get('3:4') got error: Connection reset by peer
[default4]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first):
[default4]:frame #1: <unknown function> + 0x589518e (0x7f6e4bf2b18e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fd964436d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default4]:frame #1: <unknown function> + 0x589518e (0x7fd99c3f018e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x360 (0x7f6e4bf259a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x360 (0x7fd99c3ea9a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fd99c3eace2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f6e4bf25ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:    dist.recv(
[default4]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fd99c3ebb11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd99c3a0f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
[default4]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd99c3a0f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd99c3a0f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd99c3a0f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f6e4bf26b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fd9655dec69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6e4bedbf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6e4bedbf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector<c10::Device, std::allocator<c10::Device> > const&, c10d::OpType, int, bool) + 0x22b (0x7fd9655e5c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x550 (0x7fd965608b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:frame #12: <unknown function> + 0x5838439 (0x7fd99c393439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:    return func(*args, **kwargs)
[default4]:frame #13: <unknown function> + 0x5843330 (0x7fd99c39e330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #14: <unknown function> + 0x58433c5 (0x7fd99c39e3c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #15: <unknown function> + 0x4e893cc (0x7fd99b9e43cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6e4bedbf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6e4bedbf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #16: <unknown function> + 0x1a08a88 (0x7fd998563a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #17: <unknown function> + 0x5849a84 (0x7fd99c3a4a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv
[default4]:frame #18: <unknown function> + 0x584ed35 (0x7fd99c3a9d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #19: <unknown function> + 0xc97eee (0x7fd9aec5beee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f6e15119c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:frame #20: <unknown function> + 0x413ea4 (0x7fd9ae3d7ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:frame #21: <unknown function> + 0x1445a6 (0x55eed16945a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector<c10::Device, std::allocator<c10::Device> > const&, c10d::OpType, int, bool) + 0x22b (0x7f6e15120c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:frame #22: _PyObject_MakeTpCall + 0x26b (0x55eed168da6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #23: <unknown function> + 0x150866 (0x55eed16a0866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55eed1689142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x550 (0x7f6e15143b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:frame #12: <unknown function> + 0x5838439 (0x7f6e4bece439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #25: _PyFunction_Vectorcall + 0x6c (0x55eed1694a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #26: PyObject_Call + 0xbc (0x55eed16a0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55eed16872b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #13: <unknown function> + 0x5843330 (0x7f6e4bed9330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #28: _PyFunction_Vectorcall + 0x6c (0x55eed1694a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55eed16858fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #14: <unknown function> + 0x58433c5 (0x7f6e4bed93c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:    pg.recv([tensor], group_src_rank, tag).wait()
[default4]:frame #30: <unknown function> + 0x150582 (0x55eed16a0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55eed16858fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #32: <unknown function> + 0x150582 (0x55eed16a0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #15: <unknown function> + 0x4e893cc (0x7f6e4b51f3cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55eed16858fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #34: <unknown function> + 0x150582 (0x55eed16a0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55eed16858fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:torch.distributed.DistBackendError: [9] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '8:9', but store->get('8:9') got error: Connection reset by peer
[default4]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55eed168cf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #37: _PyObject_Call_Prepend + 0x69 (0x55eed169ec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #38: <unknown function> + 0x211239 (0x55eed1761239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first):
[default4]:frame #16: <unknown function> + 0x1a08a88 (0x7f6e4809ea88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #17: <unknown function> + 0x5849a84 (0x7f6e4bedfa84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #39: _PyObject_MakeTpCall + 0x26b (0x55eed168da6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55eed16893e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #41: _PyFunction_Vectorcall + 0x6c (0x55eed1694a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f39a5ff0d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default4]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55eed1684c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #43: _PyFunction_Vectorcall + 0x6c (0x55eed1694a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55eed16858fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #1: <unknown function> + 0x589518e (0x7f39ddfaa18e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x360 (0x7f39ddfa49a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #45: <unknown function> + 0x150582 (0x55eed16a0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #46: PyObject_Call + 0xbc (0x55eed16a0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55eed16872b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f39ddfa4ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f39ddfa5b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #48: <unknown function> + 0x150582 (0x55eed16a0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #49: PyObject_Call + 0xbc (0x55eed16a0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55eed16872b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f39ddf5af81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #51: _PyFunction_Vectorcall + 0x6c (0x55eed1694a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55eed168d007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #53: _PyObject_Call_Prepend + 0x69 (0x55eed169ec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #18: <unknown function> + 0x584ed35 (0x7f6e4bee4d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #54: <unknown function> + 0x211239 (0x55eed1761239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #55: PyObject_Call + 0x207 (0x55eed16a1067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55eed16872b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f39ddf5af81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #57: <unknown function> + 0x150582 (0x55eed16a0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55eed16858fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #19: <unknown function> + 0xc97eee (0x7f6e5e796eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:frame #20: <unknown function> + 0x413ea4 (0x7f6e5df12ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:frame #59: <unknown function> + 0x150582 (0x55eed16a0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #60: PyObject_Call + 0xbc (0x55eed16a0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55eed16872b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #21: <unknown function> + 0x1445a6 (0x56335ba015a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #62: <unknown function> + 0x150582 (0x55eed16a0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #63: PyObject_Call + 0xbc (0x55eed16a0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:. This may indicate a possible application crash on rank 0 or a network set up issue.
[default4]:frame #22: _PyObject_MakeTpCall + 0x26b (0x56335b9faa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #23: <unknown function> + 0x150866 (0x56335ba0d866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x56335b9f6142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:Traceback (most recent call last):
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default7]:    trainer.train(dataloader)
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train
[default7]:    outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step
[default7]:    outputs = self.pipeline_engine.train_batch_iter(
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter
[default7]:    output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default7]:    output = [default1]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f39ddf5af81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
model(**micro_batch)
[default7]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default7]:    return self._call_impl(*args, **kwargs)
[default7]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default7]:    return forward_call(*args, **kwargs)
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward
[default4]:frame #25: _PyFunction_Vectorcall + 0x6c (0x56335ba01a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:    sharded_logits = self.model(
[default7]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default7]:    return self._call_impl(*args, **kwargs)
[default7]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default7]:    return forward_call(*args, **kwargs)
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:    return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:    hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default7]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/[default1]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f39ddf5af81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
modules/module.py", line 1511, in _wrapped_call_impl
[default7]:    return self._call_impl(*args, **kwargs)
[default7]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default7]:    return forward_call(*args, **kwargs)
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default7]:    new_kwargs[name] = recv_from_pipeline_state_buffer(
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default7]:    pipeline_state.run_communication()
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default7]:    recv_activation_tensor = recv_activation()
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_clust[default4]:frame #26: PyObject_Call + 0xbc (0x56335ba0df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
er/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default7]:    return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default7]:    buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default1]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f39a7198c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default7]:    meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default7]:    dist.recv(
[default7]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
[default7]:    return func(*args, **kwargs)
[default7]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv
[default7]:    pg.recv([tensor], group_src_rank, tag).wait()
[default7]:torch.distributed.DistBackendError: [7] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '6:7', but store->get('6:7') got error: Connection reset by peer[default4]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x56335b9f42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #28: _PyFunction_Vectorcall + 0x6c (0x56335ba01a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x56335b9f28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)

[default7]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first):
[default7]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fe05cfd1d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default7]:frame #1: <unknown function> + 0x589518e (0x7fe094f8b18e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x360 (0x7fe094f859a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #30: <unknown function> + 0x150582 (0x56335ba0d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x56335b9f28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fe094f85ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fe094f86b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe094f3bf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe094f3bf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe094f3bf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector<c10::Device, std::allocator<c10::Device> > const&, c10d::OpType, int, bool) + 0x22b (0x7f39a719fc5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe094f3bf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fe05e179c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:frame #32: <unknown function> + 0x150582 (0x56335ba0d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector<c10::Device, std::allocator<c10::Device> > const&, c10d::OpType, int, bool) + 0x22b (0x7fe05e180c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x550 (0x7fe05e1a3b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:frame #12: <unknown function> + 0x5838439 (0x7fe094f2e439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #13: <unknown function> + 0x5843330 (0x7fe094f39330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x550 (0x7f39a71c2b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:frame #14: <unknown function> + 0x58433c5 (0x7fe094f393c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #15: <unknown function> + 0x4e893cc (0x7fe09457f3cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #16: <unknown function> + 0x1a08a88 (0x7fe0910fea88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #17: <unknown function> + 0x5849a84 (0x7fe094f3fa84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x56335b9f28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #18: <unknown function> + 0x584ed35 (0x7fe094f44d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #34: <unknown function> + 0x150582 (0x56335ba0d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #19: <unknown function> + 0xc97eee (0x7fe0a77f6eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:frame #20: <unknown function> + 0x413ea4 (0x7fe0a6f72ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:frame #21: <unknown function> + 0x1445a6 (0x557d308e95a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x56335b9f28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x56335b9f9f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #22: _PyObject_MakeTpCall + 0x26b (0x557d308e2a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #23: <unknown function> + 0x150866 (0x557d308f5866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x557d308de142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #37: _PyObject_Call_Prepend + 0x69 (0x56335ba0bc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #25: _PyFunction_Vectorcall + 0x6c (0x557d308e9a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #26: PyObject_Call + 0xbc (0x557d308f5f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x557d308dc2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #28: _PyFunction_Vectorcall + 0x6c (0x557d308e9a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #12: <unknown function> + 0x5838439 (0x7f39ddf4d439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x557d308da8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #30: <unknown function> + 0x150582 (0x557d308f5582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x557d308da8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #13: <unknown function> + 0x5843330 (0x7f39ddf58330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #32: <unknown function> + 0x150582 (0x557d308f5582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x557d308da8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #34: <unknown function> + 0x150582 (0x557d308f5582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x557d308da8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #14: <unknown function> + 0x58433c5 (0x7f39ddf583c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #38: <unknown function> + 0x211239 (0x56335bace239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x557d308e1f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #37: _PyObject_Call_Prepend + 0x69 (0x557d308f3c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #38: <unknown function> + 0x211239 (0x557d309b6239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #39: _PyObject_MakeTpCall + 0x26b (0x56335b9faa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x56335b9f63e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #39: _PyObject_MakeTpCall + 0x26b (0x557d308e2a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x557d308de3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #41: _PyFunction_Vectorcall + 0x6c (0x557d308e9a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #15: <unknown function> + 0x4e893cc (0x7f39dd59e3cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x557d308d9c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #43: _PyFunction_Vectorcall + 0x6c (0x557d308e9a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x557d308da8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #45: <unknown function> + 0x150582 (0x557d308f5582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #16: <unknown function> + 0x1a08a88 (0x7f39da11da88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #46: PyObject_Call + 0xbc (0x557d308f5f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x557d308dc2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #48: <unknown function> + 0x150582 (0x557d308f5582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #49: PyObject_Call + 0xbc (0x557d308f5f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #17: <unknown function> + 0x5849a84 (0x7f39ddf5ea84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:frame #41: _PyFunction_Vectorcall + 0x6c (0x56335ba01a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x557d308dc2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #51: _PyFunction_Vectorcall + 0x6c (0x557d308e9a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x557d308e2007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x56335b9f1c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #43: _PyFunction_Vectorcall + 0x6c (0x56335ba01a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #53: _PyObject_Call_Prepend + 0x69 (0x557d308f3c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #54: <unknown function> + 0x211239 (0x557d309b6239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #55: PyObject_Call + 0x207 (0x557d308f6067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x557d308dc2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x56335b9f28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #57: <unknown function> + 0x150582 (0x557d308f5582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x557d308da8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #59: <unknown function> + 0x150582 (0x557d308f5582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #60: PyObject_Call + 0xbc (0x557d308f5f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #18: <unknown function> + 0x584ed35 (0x7f39ddf63d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x557d308dc2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #62: <unknown function> + 0x150582 (0x557d308f5582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #63: PyObject_Call + 0xbc (0x557d308f5f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #45: <unknown function> + 0x150582 (0x56335ba0d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:. This may indicate a possible application crash on rank 0 or a network set up issue.
[default4]:frame #46: PyObject_Call + 0xbc (0x56335ba0df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #19: <unknown function> + 0xc97eee (0x7f39f0815eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x56335b9f42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #48: <unknown function> + 0x150582 (0x56335ba0d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #20: <unknown function> + 0x413ea4 (0x7f39eff91ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:frame #49: PyObject_Call + 0xbc (0x56335ba0df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #21: <unknown function> + 0x1445a6 (0x55877839b5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x56335b9f42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #22: _PyObject_MakeTpCall + 0x26b (0x558778394a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #23: <unknown function> + 0x150866 (0x5587783a7866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x558778390142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #51: _PyFunction_Vectorcall + 0x6c (0x56335ba01a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x56335b9fa007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #25: _PyFunction_Vectorcall + 0x6c (0x55877839ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #53: _PyObject_Call_Prepend + 0x69 (0x56335ba0bc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #26: PyObject_Call + 0xbc (0x5587783a7f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55877838e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #54: <unknown function> + 0x211239 (0x56335bace239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #28: _PyFunction_Vectorcall + 0x6c (0x55877839ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55877838c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #30: <unknown function> + 0x150582 (0x5587783a7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55877838c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #55: PyObject_Call + 0x207 (0x56335ba0e067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x56335b9f42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #32: <unknown function> + 0x150582 (0x5587783a7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #57: <unknown function> + 0x150582 (0x56335ba0d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55877838c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x56335b9f28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #59: <unknown function> + 0x150582 (0x56335ba0d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #60: PyObject_Call + 0xbc (0x56335ba0df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x56335b9f42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #62: <unknown function> + 0x150582 (0x56335ba0d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:frame #63: PyObject_Call + 0xbc (0x56335ba0df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #34: <unknown function> + 0x150582 (0x5587783a7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:. This may indicate a possible application crash on rank 0 or a network set up issue.
[default1]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55877838c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x558778393f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #37: _PyObject_Call_Prepend + 0x69 (0x5587783a5c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #38: <unknown function> + 0x211239 (0x558778468239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #39: _PyObject_MakeTpCall + 0x26b (0x558778394a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5587783903e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #41: _PyFunction_Vectorcall + 0x6c (0x55877839ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55877838bc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #43: _PyFunction_Vectorcall + 0x6c (0x55877839ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55877838c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #45: <unknown function> + 0x150582 (0x5587783a7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #46: PyObject_Call + 0xbc (0x5587783a7f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55877838e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #48: <unknown function> + 0x150582 (0x5587783a7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #49: PyObject_Call + 0xbc (0x5587783a7f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55877838e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #51: _PyFunction_Vectorcall + 0x6c (0x55877839ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x558778394007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #53: _PyObject_Call_Prepend + 0x69 (0x5587783a5c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #54: <unknown function> + 0x211239 (0x558778468239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #55: PyObject_Call + 0x207 (0x5587783a8067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55877838e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #57: <unknown function> + 0x150582 (0x5587783a7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55877838c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #59: <unknown function> + 0x150582 (0x5587783a7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #60: PyObject_Call + 0xbc (0x5587783a7f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55877838e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #62: <unknown function> + 0x150582 (0x5587783a7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:frame #63: PyObject_Call + 0xbc (0x5587783a7f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:. This may indicate a possible application crash on rank 0 or a network set up issue.
[default6]:Traceback (most recent call last):
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:    trainer.train(dataloader)
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train
[default6]:    outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step
[default6]:    outputs = self.pipeline_engine.train_batch_iter(
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter
[default6]:    output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:    output = model(**micro_batch)
[default6]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default6]:    return self._call_impl(*args, **kwargs)
[default6]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default6]:    return forward_call(*args, **kwargs)
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward
[default6]:    sharded_logits = self.model(
[default6]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default6]:    return self._call_impl(*args, **kwargs)
[default6]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default6]:    return forward_call(*args, **kwargs)
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:    return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:    hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default6]:    return self._call_impl(*args, **kwargs)
[default6]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default6]:    return forward_call(*args, **kwargs)
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default6]:    new_kwargs[name] = recv_from_pipeline_state_buffer(
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default6]:    pipeline_state.run_communication()
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default6]:    recv_activation_tensor = recv_activation()
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default6]:    return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default6]:    buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default6]:    meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default6]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default6]:    dist.recv(
[default6]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
[default6]:    return func(*args, **kwargs)
[default6]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv
[default6]:    pg.recv([tensor], group_src_rank, tag).wait()
[default6]:torch.distributed.DistBackendError: [14] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '13:14', but store->get('13:14') got error: Connection reset by peer
[default6]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first):
[default6]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f68df159d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default6]:frame #1: <unknown function> + 0x589518e (0x7f691711318e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x360 (0x7f691710d9a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f691710dce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f691710eb11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f69170c3f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f69170c3f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f69170c3f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f69170c3f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f68e0301c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector<c10::Device, std::allocator<c10::Device> > const&, c10d::OpType, int, bool) + 0x22b (0x7f68e0308c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x550 (0x7f68e032bb60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:frame #12: <unknown function> + 0x5838439 (0x7f69170b6439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #13: <unknown function> + 0x5843330 (0x7f69170c1330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #14: <unknown function> + 0x58433c5 (0x7f69170c13c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #15: <unknown function> + 0x4e893cc (0x7f69167073cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #16: <unknown function> + 0x1a08a88 (0x7f6913286a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #17: <unknown function> + 0x5849a84 (0x7f69170c7a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #18: <unknown function> + 0x584ed35 (0x7f69170ccd35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:frame #19: <unknown function> + 0xc97eee (0x7f692997eeee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:frame #20: <unknown function> + 0x413ea4 (0x7f69290faea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:frame #21: <unknown function> + 0x1445a6 (0x55bb607675a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #22: _PyObject_MakeTpCall + 0x26b (0x55bb60760a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #23: <unknown function> + 0x150866 (0x55bb60773866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55bb6075c142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #25: _PyFunction_Vectorcall + 0x6c (0x55bb60767a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #26: PyObject_Call + 0xbc (0x55bb60773f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55bb6075a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #28: _PyFunction_Vectorcall + 0x6c (0x55bb60767a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55bb607588fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #30: <unknown function> + 0x150582 (0x55bb60773582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55bb607588fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #32: <unknown function> + 0x150582 (0x55bb60773582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55bb607588fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #34: <unknown function> + 0x150582 (0x55bb60773582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55bb607588fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55bb6075ff50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #37: _PyObject_Call_Prepend + 0x69 (0x55bb60771c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #38: <unknown function> + 0x211239 (0x55bb60834239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #39: _PyObject_MakeTpCall + 0x26b (0x55bb60760a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55bb6075c3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #41: _PyFunction_Vectorcall + 0x6c (0x55bb60767a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55bb60757c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #43: _PyFunction_Vectorcall + 0x6c (0x55bb60767a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55bb607588fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #45: <unknown function> + 0x150582 (0x55bb60773582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #46: PyObject_Call + 0xbc (0x55bb60773f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55bb6075a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #48: <unknown function> + 0x150582 (0x55bb60773582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #49: PyObject_Call + 0xbc (0x55bb60773f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55bb6075a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #51: _PyFunction_Vectorcall + 0x6c (0x55bb60767a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55bb60760007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #53: _PyObject_Call_Prepend + 0x69 (0x55bb60771c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #54: <unknown function> + 0x211239 (0x55bb60834239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #55: PyObject_Call + 0x207 (0x55bb60774067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55bb6075a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #57: <unknown function> + 0x150582 (0x55bb60773582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55bb607588fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #59: <unknown function> + 0x150582 (0x55bb60773582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #60: PyObject_Call + 0xbc (0x55bb60773f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55bb6075a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #62: <unknown function> + 0x150582 (0x55bb60773582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:frame #63: PyObject_Call + 0xbc (0x55bb60773f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:. This may indicate a possible application crash on rank 0 or a network set up issue.
[default7]:Traceback (most recent call last):
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default7]:    trainer.train(dataloader)
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train
[default7]:    outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step
[default7]:    outputs = self.pipeline_engine.train_batch_iter(
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter
[default7]:    output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default7]:    output = model(**micro_batch)
[default7]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default7]:    return self._call_impl(*args, **kwargs)
[default7]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default7]:    return forward_call(*args, **kwargs)
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward
[default7]:    sharded_logits = self.model(
[default7]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default7]:    return self._call_impl(*args, **kwargs)
[default7]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default7]:    return forward_call(*args, **kwargs)
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:    return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:    hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default7]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
[default7]:    return self._call_impl(*args, **kwargs)
[default7]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
[default7]:    return forward_call(*args, **kwargs)
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default7]:    new_kwargs[name] = recv_from_pipeline_state_buffer(
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default7]:    pipeline_state.run_communication()
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default7]:    recv_activation_tensor = recv_activation()
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default7]:    return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default7]:    buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default7]:    meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default7]:  File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default7]:    dist.recv(
[default7]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
[default7]:    return func(*args, **kwargs)
[default7]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv
[default7]:    pg.recv([tensor], group_src_rank, tag).wait()
[default7]:torch.distributed.DistBackendError: [15] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '14:15', but store->get('14:15') got error: Connection reset by peer
[default7]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first):
[default7]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f234a721d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default7]:frame #1: <unknown function> + 0x589518e (0x7f23826db18e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x360 (0x7f23826d59a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f23826d5ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f23826d6b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f238268bf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f238268bf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f238268bf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f238268bf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f234b8c9c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector<c10::Device, std::allocator<c10::Device> > const&, c10d::OpType, int, bool) + 0x22b (0x7f234b8d0c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x550 (0x7f234b8f3b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:frame #12: <unknown function> + 0x5838439 (0x7f238267e439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #13: <unknown function> + 0x5843330 (0x7f2382689330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #14: <unknown function> + 0x58433c5 (0x7f23826893c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #15: <unknown function> + 0x4e893cc (0x7f2381ccf3cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #16: <unknown function> + 0x1a08a88 (0x7f237e84ea88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #17: <unknown function> + 0x5849a84 (0x7f238268fa84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #18: <unknown function> + 0x584ed35 (0x7f2382694d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:frame #19: <unknown function> + 0xc97eee (0x7f2394f46eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:frame #20: <unknown function> + 0x413ea4 (0x7f23946c2ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:frame #21: <unknown function> + 0x1445a6 (0x564d0c2385a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #22: _PyObject_MakeTpCall + 0x26b (0x564d0c231a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #23: <unknown function> + 0x150866 (0x564d0c244866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x564d0c22d142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #25: _PyFunction_Vectorcall + 0x6c (0x564d0c238a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #26: PyObject_Call + 0xbc (0x564d0c244f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x564d0c22b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #28: _PyFunction_Vectorcall + 0x6c (0x564d0c238a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x564d0c2298fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #30: <unknown function> + 0x150582 (0x564d0c244582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x564d0c2298fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #32: <unknown function> + 0x150582 (0x564d0c244582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x564d0c2298fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #34: <unknown function> + 0x150582 (0x564d0c244582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x564d0c2298fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x564d0c230f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #37: _PyObject_Call_Prepend + 0x69 (0x564d0c242c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #38: <unknown function> + 0x211239 (0x564d0c305239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #39: _PyObject_MakeTpCall + 0x26b (0x564d0c231a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x564d0c22d3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #41: _PyFunction_Vectorcall + 0x6c (0x564d0c238a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x564d0c228c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #43: _PyFunction_Vectorcall + 0x6c (0x564d0c238a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x564d0c2298fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #45: <unknown function> + 0x150582 (0x564d0c244582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #46: PyObject_Call + 0xbc (0x564d0c244f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x564d0c22b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #48: <unknown function> + 0x150582 (0x564d0c244582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #49: PyObject_Call + 0xbc (0x564d0c244f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x564d0c22b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #51: _PyFunction_Vectorcall + 0x6c (0x564d0c238a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x564d0c231007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #53: _PyObject_Call_Prepend + 0x69 (0x564d0c242c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #54: <unknown function> + 0x211239 (0x564d0c305239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #55: PyObject_Call + 0x207 (0x564d0c245067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x564d0c22b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #57: <unknown function> + 0x150582 (0x564d0c244582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x564d0c2298fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #59: <unknown function> + 0x150582 (0x564d0c244582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #60: PyObject_Call + 0xbc (0x564d0c244f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x564d0c22b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #62: <unknown function> + 0x150582 (0x564d0c244582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:frame #63: PyObject_Call + 0xbc (0x564d0c244f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:. This may indicate a possible application crash on rank 0 or a network set up issue.
[2024-07-06 09:44:13,000] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 73821 closing signal SIGTERM
[2024-07-06 09:44:13,001] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 73822 closing signal SIGTERM
[2024-07-06 09:44:13,001] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 73823 closing signal SIGTERM
[2024-07-06 09:44:13,001] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 73824 closing signal SIGTERM
[2024-07-06 09:44:13,001] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 73827 closing signal SIGTERM
[2024-07-06 09:44:13,008] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 72592 closing signal SIGTERM
[2024-07-06 09:44:13,010] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 637196 closing signal SIGTERM
[2024-07-06 09:44:13,010] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 637197 closing signal SIGTERM
[2024-07-06 09:44:13,008] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 72593 closing signal SIGTERM
[2024-07-06 09:44:13,010] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 637198 closing signal SIGTERM
[2024-07-06 09:44:13,009] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 72594 closing signal SIGTERM
[2024-07-06 09:44:13,009] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 72595 closing signal SIGTERM
[2024-07-06 09:44:13,010] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 72596 closing signal SIGTERM
[2024-07-06 09:44:13,011] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 72597 closing signal SIGTERM
[2024-07-06 09:44:13,012] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 72598 closing signal SIGTERM
[2024-07-06 09:44:14,626] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 4 (pid: 73825) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
[2024-07-06 09:44:14,632] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -6) local_rank: 0 (pid: 637195) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-07-06_09:44:13
  host      : ip-26-0-169-132.ec2.internal
  rank      : 13 (local_rank: 5)
  exitcode  : 1 (pid: 73826)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-07-06_09:44:13
  host      : ip-26-0-169-132.ec2.internal
  rank      : 15 (local_rank: 7)
  exitcode  : 1 (pid: 73828)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-06_09:44:13
  host      : ip-26-0-169-132.ec2.internal
  rank      : 12 (local_rank: 4)
  exitcode  : 1 (pid: 73825)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-07-06_09:44:13
  host      : ip-26-0-160-192.ec2.internal
  rank      : 4 (local_rank: 4)
  exitcode  : 1 (pid: 637199)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-07-06_09:44:13
  host      : ip-26-0-160-192.ec2.internal
  rank      : 5 (local_rank: 5)
  exitcode  : 1 (pid: 637200)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2024-07-06_09:44:13
  host      : ip-26-0-160-192.ec2.internal
  rank      : 6 (local_rank: 6)
  exitcode  : 1 (pid: 637201)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
  time      : 2024-07-06_09:44:13
  host      : ip-26-0-160-192.ec2.internal
  rank      : 7 (local_rank: 7)
  exitcode  : 1 (pid: 637202)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-06_09:44:13
  host      : ip-26-0-160-192.ec2.internal
  rank      : 0 (local_rank: 0)
  exitcode  : -6 (pid: 637195)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 637195
============================================================
srun: error: ip-26-0-169-132: task 2: Exited with exit code 1
srun: error: ip-26-0-160-192: task 0: Exited with exit code 1
[default0]:[rank56]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[default0]:[rank56]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down.
[default0]:[rank56]:[E ProcessGroupNCCL.cpp:1182] [Rank 56] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3
[default0]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.
[default0]:Last error:
[default0]:socketProgress: Connection closed by remote peer ip-26-0-169-86.ec2.internal<33950>
[default0]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first):
[default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f6a31211d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector<std::shared_ptr<c10d::NCCLComm>, std::allocator<std::shared_ptr<c10d::NCCLComm> > > const&) + 0x2f3 (0x7f6a323b8fa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7f6a323b927b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7f6a323bcc1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f6a323bd839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #5: <unknown function> + 0xd3e95 (0x7f6a7c0c1e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6)
[default0]:frame #6: <unknown function> + 0x8609 (0x7f6a811c9609 in /lib/x86_64-linux-gnu/libpthread.so.0)
[default0]:frame #7: clone + 0x43 (0x7f6a80f94353 in /lib/x86_64-linux-gnu/libc.so.6)
[default0]:
[default0]:terminate called after throwing an instance of 'c10::DistBackendError'
[default0]:  what():  [Rank 56] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3
[default0]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.
[default0]:Last error:
[default0]:socketProgress: Connection closed by remote peer ip-26-0-169-86.ec2.internal<33950>
[default0]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first):
[default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f6a31211d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector<std::shared_ptr<c10d::NCCLComm>, std::allocator<std::shared_ptr<c10d::NCCLComm> > > const&) + 0x2f3 (0x7f6a323b8fa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7f6a323b927b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7f6a323bcc1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python[2024-07-06 09:44:15,231] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -6) local_rank: 0 (pid: 72591) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f6a323bd839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #5: <unknown function> + 0xd3e95 (0x7f6a7c0c1e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6)
[default0]:frame #6: <unknown function> + 0x8609 (0x7f6a811c9609 in /lib/x86_64-linux-gnu/libpthread.so.0)
[default0]:frame #7: clone + 0x43 (0x7f6a80f94353 in /lib/x86_64-linux-gnu/libc.so.6)
[default0]:
[default0]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1186 (most recent call first):
[default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f6a31211d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:frame #1: <unknown function> + 0xdf6b11 (0x7f6a32113b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:frame #2: <unknown function> + 0xd3e95 (0x7f6a7c0c1e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6)
[default0]:frame #3: <unknown function> + 0x8609 (0x7f6a811c9609 in /lib/x86_64-linux-gnu/libpthread.so.0)
[default0]:frame #4: clone + 0x43 (0x7f6a80f94353 in /lib/x86_64-linux-gnu/libc.so.6)
[default0]:
[2024-07-06 09:44:15,277] torch.distributed.elastic.rendezvous.dynamic_rendezvous: [WARNING] The node 'ip-26-0-169-86.ec2.internal_72522_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-06_09:44:13
  host      : ip-26-0-169-86.ec2.internal
  rank      : 48 (local_rank: 0)
  exitcode  : -6 (pid: 72591)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 72591
============================================================
srun: error: ip-26-0-169-86: task 1: Exited with exit code 1
[2024-07-06 09:44:17,815] torch.distributed.elastic.rendezvous.dynamic_rendezvous: [WARNING] The node 'ip-26-0-169-247.ec2.internal_541030_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
[2024-07-06 09:44:17,969] torch.distributed.elastic.rendezvous.dynamic_rendezvous: [WARNING] The node 'ip-26-0-170-31.ec2.internal_48404_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
[2024-07-06 09:44:18,003] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 48476 closing signal SIGTERM
[2024-07-06 09:44:18,004] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 48477 closing signal SIGTERM
[2024-07-06 09:44:18,004] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 48478 closing signal SIGTERM
[2024-07-06 09:44:18,004] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 48479 closing signal SIGTERM
[2024-07-06 09:44:18,005] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 48480 closing signal SIGTERM
[2024-07-06 09:44:18,005] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 48481 closing signal SIGTERM
[2024-07-06 09:44:18,005] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 48482 closing signal SIGTERM
[2024-07-06 09:44:18,014] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 541101 closing signal SIGTERM
[2024-07-06 09:44:18,014] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 541102 closing signal SIGTERM
[2024-07-06 09:44:18,015] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 541103 closing signal SIGTERM
[2024-07-06 09:44:18,015] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 541104 closing signal SIGTERM
[2024-07-06 09:44:18,016] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 541105 closing signal SIGTERM
[2024-07-06 09:44:18,016] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 541106 closing signal SIGTERM
[2024-07-06 09:44:18,017] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 541107 closing signal SIGTERM
[2024-07-06 09:44:18,017] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 541108 closing signal SIGTERM
[2024-07-06 09:44:19,836] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -6) local_rank: 0 (pid: 48475) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
[2024-07-06 09:44:19,876] torch.distributed.elastic.rendezvous.dynamic_rendezvous: [WARNING] The node 'ip-26-0-170-31.ec2.internal_48404_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-06_09:44:18
  host      : ip-26-0-170-31.ec2.internal
  rank      : 56 (local_rank: 0)
  exitcode  : -6 (pid: 48475)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 48475
============================================================
srun: error: ip-26-0-170-31: task 7: Exited with exit code 1
[2024-07-06 09:44:20,639] torch.distributed.elastic.rendezvous.dynamic_rendezvous: [WARNING] The node 'ip-26-0-169-247.ec2.internal_541030_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 113, in _call_store
    return getattr(self._store, store_op)(*args, **kwargs)
torch.distributed.DistNetworkError: Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 259, in launch_agent
    result = agent.run()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper
    result = f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 727, in run
    result = self._invoke_run(role)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 900, in _invoke_run
    num_nodes_waiting = rdzv_handler.num_nodes_waiting()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1083, in num_nodes_waiting
    self._state_holder.sync()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 409, in sync
    get_response = self._backend.get_state()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 73, in get_state
    base64_state: bytes = self._call_store("get", self._key)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 115, in _call_store
    raise RendezvousConnectionError(
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.
srun: error: ip-26-0-169-247: task 6: Exited with exit code 1
Consider using `hf_transfer` for faster uploads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details.