========================
START TIME: Wed Jul  3 23:09:08 UTC 2024
python3 version = Python 3.10.14
========================
The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.
Token is valid (permission: write).
Your token has been saved to /admin/home/ferdinand_mom/.cache/huggingface/token
Login successful
Already on 'bench_cluster'
M	examples/config_tiny_llama.py
M	examples/config_tiny_llama.yaml
M	examples/train_tiny_llama.sh
M	src/nanotron/models/llama.py
M	src/nanotron/trainer.py
Your branch is up to date with 'origin/bench_cluster'.
Job status: RUNNING
W0703 23:09:14.260000 140032670197568 torch/distributed/run.py:757] 
W0703 23:09:14.260000 140032670197568 torch/distributed/run.py:757] *****************************************
W0703 23:09:14.260000 140032670197568 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0703 23:09:14.260000 140032670197568 torch/distributed/run.py:757] *****************************************
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Config:
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Config(general=GeneralArgs(project='bench_cluster',
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                            run='%date_%jobid',
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                            seed=42,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                            step=None,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                            consumed_train_samples=None,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                            benchmark_csv_path=None,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                            ignore_sanity_checks=True),
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        parallelism=ParallelismArgs(dp=1,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    pp=8,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    tp=1,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    pp_engine=<nanotron.parallel.pipeline_parallel.engine.OneForwardOneBackwardPipelineEngine object at 0x7f312b8a0880>,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    tp_mode=<TensorParallelLinearMode.REDUCE_SCATTER: 2>,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    tp_linear_async_communication=False,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    expert_parallel_size=1),
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        model=ModelArgs(model_config=LlamaConfig(bos_token_id=1,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 eos_token_id=2,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 hidden_act='silu',
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 hidden_size=2048,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 initializer_range=0.02,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 intermediate_size=4096,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 is_llama_config=True,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 max_position_embeddings=4096,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 num_attention_heads=32,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 num_hidden_layers=24,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 num_key_value_heads=32,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 pad_token_id=None,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 pretraining_tp=1,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 rms_norm_eps=1e-05,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 rope_scaling=None,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 rope_theta=10000.0,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 tie_word_embeddings=True,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 use_cache=True,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 vocab_size=50257),
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                        init_method=RandomInit(std=0.025),
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                        dtype=torch.bfloat16,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                        make_vocab_size_divisible_by=1,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                        ddp_bucket_cap_mb=25),
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        tokenizer=TokenizerArgs(tokenizer_name_or_path='openai-community/gpt2',
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                tokenizer_revision=None,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                tokenizer_max_length=None),
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        checkpoints=CheckpointsArgs(checkpoints_path=Path('/dev/null'),
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    checkpoint_interval=100000,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    save_initial_state=False,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    resume_checkpoint_path=None,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    checkpoints_path_is_shared_file_system=False),
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        logging=LoggingArgs(log_level='info',
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                            log_level_replica='info',
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                            iteration_step_info_interval=1),
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        tokens=TokensArgs(sequence_length=4096,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                          train_steps=20,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                          micro_batch_size=16,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                          batch_accumulation_per_replica=64,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                          val_check_interval=-1,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                          limit_val_batches=0,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                          limit_test_batches=0),
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        optimizer=OptimizerArgs(optimizer_factory=AdamWOptimizerArgs(adam_eps=1e-08,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                     adam_beta1=0.9,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                     adam_beta2=0.95,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                     torch_adam_is_fused=True,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                     name='adamW'),
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                zero_stage=1,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                weight_decay=0.01,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                clip_grad=1.0,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                accumulate_grad_in_fp32=True,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                learning_rate_scheduler=LRSchedulerArgs(learning_rate=0.0001,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                        lr_warmup_steps=1,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                        lr_warmup_style='linear',
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                        lr_decay_style='linear',
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                        lr_decay_steps=19,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                        lr_decay_starting_step=None,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                        min_decay_lr=1e-05)),
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        data_stages=[DatasetStageArgs(name='Training Stage',
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                      start_training_step=1,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                      data=DataArgs(dataset=PretrainDatasetsArgs(hf_dataset_or_datasets='roneneldan/TinyStories',
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                                 hf_dataset_splits='train',
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                                 hf_dataset_config_name=None,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                                 dataset_processing_num_proc_per_process=64,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                                 dataset_overwrite_cache=False,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                                 text_column_name='text'),
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                    seed=42,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                    num_loading_workers=0))],
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        profiler=ProfilerArgs(profiler_export_path=Path('/fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/8_GPUS/dp-1_tp-1_pp-8_mbz-16')),
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        lighteval=None)
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Model Config:
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: LlamaConfig(bos_token_id=1,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             eos_token_id=2,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             hidden_act='silu',
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             hidden_size=2048,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             initializer_range=0.02,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             intermediate_size=4096,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             is_llama_config=True,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             max_position_embeddings=4096,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             num_attention_heads=32,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             num_hidden_layers=24,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             num_key_value_heads=32,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             pad_token_id=None,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             pretraining_tp=1,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             rms_norm_eps=1e-05,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             rope_scaling=None,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             rope_theta=10000.0,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             tie_word_embeddings=True,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             use_cache=True,
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             vocab_size=50257)
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Building model..
[default0]:07/03/2024 23:09:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Setting PP block ranks...
[default4]:07/03/2024 23:09:50 [INFO|DP=0|PP=4|TP=0|ip-26-0-160-225]: Local number of parameters: 126M (240.02MiB)
[default4]:07/03/2024 23:09:50 [INFO|DP=0|PP=4|TP=0|ip-26-0-160-225]: [After model building] Memory usage: 243.03MiB. Peak allocated: 245.06MiB Peak reserved: 262.00MiB
[default4]:07/03/2024 23:09:50 [INFO|DP=0|PP=4|TP=0|ip-26-0-160-225]: No checkpoint path provided.
[default6]:07/03/2024 23:09:50 [INFO|DP=0|PP=6|TP=0|ip-26-0-160-225]: Local number of parameters: 168M (320.03MiB)
[default6]:07/03/2024 23:09:50 [INFO|DP=0|PP=6|TP=0|ip-26-0-160-225]: [After model building] Memory usage: 324.04MiB. Peak allocated: 326.07MiB Peak reserved: 336.00MiB
[default6]:07/03/2024 23:09:50 [INFO|DP=0|PP=6|TP=0|ip-26-0-160-225]: No checkpoint path provided.
[default2]:07/03/2024 23:09:50 [INFO|DP=0|PP=2|TP=0|ip-26-0-160-225]: Local number of parameters: 126M (240.02MiB)
[default2]:07/03/2024 23:09:50 [INFO|DP=0|PP=2|TP=0|ip-26-0-160-225]: [After model building] Memory usage: 243.03MiB. Peak allocated: 245.06MiB Peak reserved: 262.00MiB
[default2]:07/03/2024 23:09:50 [INFO|DP=0|PP=2|TP=0|ip-26-0-160-225]: No checkpoint path provided.
[default5]:07/03/2024 23:09:50 [INFO|DP=0|PP=5|TP=0|ip-26-0-160-225]: Local number of parameters: 126M (240.02MiB)
[default5]:07/03/2024 23:09:50 [INFO|DP=0|PP=5|TP=0|ip-26-0-160-225]: [After model building] Memory usage: 243.03MiB. Peak allocated: 245.06MiB Peak reserved: 262.00MiB
[default5]:07/03/2024 23:09:50 [INFO|DP=0|PP=5|TP=0|ip-26-0-160-225]: No checkpoint path provided.
[default1]:07/03/2024 23:09:50 [INFO|DP=0|PP=1|TP=0|ip-26-0-160-225]: Local number of parameters: 126M (240.02MiB)
[default1]:07/03/2024 23:09:50 [INFO|DP=0|PP=1|TP=0|ip-26-0-160-225]: [After model building] Memory usage: 243.03MiB. Peak allocated: 245.06MiB Peak reserved: 262.00MiB
[default1]:07/03/2024 23:09:50 [INFO|DP=0|PP=1|TP=0|ip-26-0-160-225]: No checkpoint path provided.
[default0]:07/03/2024 23:09:50 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Total number of parameters: 1.21G (2312.82MiB)
[default0]:07/03/2024 23:09:50 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Local number of parameters: 271M (516.35MiB)
[default0]:07/03/2024 23:09:50 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [After model building] Memory usage: 520.36MiB. Peak allocated: 522.39MiB Peak reserved: 534.00MiB
[default0]:07/03/2024 23:09:50 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: No checkpoint path provided.
[default0]:07/03/2024 23:09:50 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Parametrizing model parameters using StandardParametrizator
[default3]:07/03/2024 23:09:50 [INFO|DP=0|PP=3|TP=0|ip-26-0-160-225]: Local number of parameters: 168M (320.03MiB)
[default3]:07/03/2024 23:09:50 [INFO|DP=0|PP=3|TP=0|ip-26-0-160-225]: [After model building] Memory usage: 324.04MiB. Peak allocated: 326.07MiB Peak reserved: 336.00MiB
[default3]:07/03/2024 23:09:50 [INFO|DP=0|PP=3|TP=0|ip-26-0-160-225]: No checkpoint path provided.
[default7]:07/03/2024 23:09:50 [INFO|DP=0|PP=7|TP=0|ip-26-0-160-225]: Local number of parameters: 103M (196.32MiB)
[default7]:07/03/2024 23:09:50 [INFO|DP=0|PP=7|TP=0|ip-26-0-160-225]: [After model building] Memory usage: 196.33MiB. Peak allocated: 196.33MiB Peak reserved: 200.00MiB
[default7]:07/03/2024 23:09:50 [INFO|DP=0|PP=7|TP=0|ip-26-0-160-225]: No checkpoint path provided.
[default0]:07/03/2024 23:09:50 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [Optimizer Building] Using LearningRateForSP as learning rate
[default0]:07/03/2024 23:09:50 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [ZeRO sharding] Size of optimizer params per rank:
[default0]:07/03/2024 23:09:50 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [ZeRO sharding] DP Rank 0 has 271M out of 271M (100.00%) params' optimizer states
[default0]:07/03/2024 23:09:51 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [Training Plan] Stage Training Stage has 19 remaining training steps and has consumed 0 samples
[default0]:07/03/2024 23:09:51 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Using `datasets` library
[default0]:07/03/2024 23:09:51 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Loading tokenizer from openai-community/gpt2 and transformers/hf_hub versions ('4.41.2', '0.23.4')
[default0]:07/03/2024 23:09:52 [WARNING|DP=0|PP=0|TP=0|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/03/2024 23:09:54 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [Training Plan] There are 1 training stages 
[default0]:07/03/2024 23:09:54 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [Stage Training Stage] start from step 1 
[default0]:07/03/2024 23:09:54 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: 
[default0]:07/03/2024 23:09:54 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [Start training] datetime: 2024-07-03 23:09:54.875055 | mbs: 16 | grad_accum: 64 | global_batch_size: 1024 | sequence_length: 4096 | train_steps: 20 | start_iteration_step: 0 | consumed_train_samples: 0
[default0]:07/03/2024 23:09:54 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Resuming training from stage Training Stage, it has trained for 0 samples and has 19 remaining train steps
[default0]:07/03/2024 23:09:54 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:  Memory usage: 2585.75MiB. Peak allocated 2585.75MiB. Peak reserved: 2602.00MiB
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/03/2024 23:09:55 [WARNING|DP=0|PP=5|TP=0|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/03/2024 23:09:55 [WARNING|DP=0|PP=6|TP=0|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/03/2024 23:09:55 [WARNING|DP=0|PP=7|TP=0|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/03/2024 23:09:55 [WARNING|DP=0|PP=2|TP=0|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/03/2024 23:09:55 [WARNING|DP=0|PP=3|TP=0|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/03/2024 23:10:05 [WARNING|DP=0|PP=6|TP=0|ip-26-0-160-225]: Using the latest cached version of the dataset since roneneldan/TinyStories couldn't be found on the Hugging Face Hub
[default6]:07/03/2024 23:10:05 [WARNING|DP=0|PP=6|TP=0|ip-26-0-160-225]: Found the latest cached dataset configuration 'default' at /admin/home/ferdinand_mom/.cache/roneneldan___tiny_stories/default/0.0.0/691b0d9bd48ade766778c940011ca1c549f6359b (last modified on Mon Jun 24 07:59:52 2024).
[default6]:Using the latest cached version of the dataset since roneneldan/TinyStories couldn't be found on the Hugging Face Hub
[default6]:Found the latest cached dataset configuration 'default' at /admin/home/ferdinand_mom/.cache/roneneldan___tiny_stories/default/0.0.0/691b0d9bd48ade766778c940011ca1c549f6359b (last modified on Mon Jun 24 07:59:52 2024).
[default4]:07/03/2024 23:10:19 [WARNING|DP=0|PP=4|TP=0|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/03/2024 23:10:37 [WARNING|DP=0|PP=1|TP=0|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:[rank0]: Traceback (most recent call last):
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:[rank0]:     trainer.train(dataloader)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default0]:[rank0]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank0]:     outputs = self.pipeline_engine.train_batch_iter(
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default0]:[rank0]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:[rank0]:     output = model(**micro_batch)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:     return self._call_impl(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank0]:     return forward_call(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default0]:[rank0]:     sharded_logits = self.model(
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:     return self._call_impl(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank0]:     return forward_call(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default0]:[rank0]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default0]:[rank0]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:     return self._call_impl(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank0]:     return forward_call(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default0]:[rank0]:     output = self.pp_block(**new_kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:     return self._call_impl(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank0]:     return forward_call(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default0]:[rank0]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:     return self._call_impl(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank0]:     return forward_call(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 389, in forward
[default0]:[rank0]:     .contiguous()
[default0]:[rank0]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 768.00 MiB. GPU 
[default5]:[rank5]: Traceback (most recent call last):
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank5]:     trainer.train(dataloader)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank5]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank5]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default5]:[rank5]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank5]:     output = model(**micro_batch)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank5]:     sharded_logits = self.model(
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default5]:[rank5]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:[rank5]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default5]:[rank5]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default5]:[rank5]:     pipeline_state.run_communication()
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default5]:[rank5]:     recv_activation_tensor = recv_activation()
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default5]:[rank5]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default5]:[rank5]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default5]:[rank5]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default5]:[rank5]:     dist.recv(
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default5]:[rank5]:     return func(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default5]:[rank5]:     pg.recv([tensor], group_src_rank, tag).wait()
[default5]:[rank5]: torch.distributed.DistBackendError: [5] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '4:5', but store->get('4:5') got error: Connection reset by peer
[default5]:[rank5]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default5]:[rank5]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f22de669897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default5]:[rank5]: frame #1: <unknown function> + 0x5b3a23e (0x7f231818623e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank5]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f2318180c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank5]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f2318180f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank5]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f2318181fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank5]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2318136371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank5]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2318136371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank5]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2318136371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank5]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2318136371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank5]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f22df943189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank5]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f22df94a610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank5]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f22df969978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank5]: frame #12: <unknown function> + 0x5adc309 (0x7f2318128309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank5]: frame #13: <unknown function> + 0x5ae6f10 (0x7f2318132f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank5]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f2318132fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank5]: frame #15: <unknown function> + 0x5124446 (0x7f2317770446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank5]: frame #16: <unknown function> + 0x1acf4b8 (0x7f231411b4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank5]: frame #17: <unknown function> + 0x5aee004 (0x7f231813a004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank5]: frame #18: <unknown function> + 0x5af36b5 (0x7f231813f6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank5]: frame #19: <unknown function> + 0xd2631e (0x7f232ad2931e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:[rank5]: frame #20: <unknown function> + 0x47def4 (0x7f232a480ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:[rank5]: frame #21: <unknown function> + 0x1445a6 (0x55cf8973b5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55cf89734a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #23: <unknown function> + 0x150866 (0x55cf89747866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55cf89730142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55cf8973ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #26: PyObject_Call + 0xbc (0x55cf89747f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55cf8972e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55cf8973ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55cf8972c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #30: <unknown function> + 0x150582 (0x55cf89747582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55cf8972c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #32: <unknown function> + 0x150582 (0x55cf89747582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55cf8972c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #34: <unknown function> + 0x150582 (0x55cf89747582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55cf8972c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55cf89733f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55cf89745c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #38: <unknown function> + 0x211239 (0x55cf89808239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55cf89734a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55cf897303e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55cf8973ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55cf8972bc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55cf8973ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: Traceback (most recent call last):
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank5]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55cf8972c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]:     trainer.train(dataloader)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default6]:[rank6]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank6]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default5]:[rank5]: frame #45: <unknown function> + 0x150582 (0x55cf89747582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank5]: frame #46: PyObject_Call + 0xbc (0x55cf89747f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]:     output = model(**micro_batch)
[default5]:[rank5]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55cf8972e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank6]:     return self._call_impl(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank6]:     return forward_call(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank5]: frame #48: <unknown function> + 0x150582 (0x55cf89747582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #49: PyObject_Call + 0xbc (0x55cf89747f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]:     sharded_logits = self.model(
[default5]:[rank5]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55cf8972e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55cf8973ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank6]:     return self._call_impl(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank6]:     return forward_call(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default5]:[rank5]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55cf89734007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank5]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55cf89745c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #54: <unknown function> + 0x211239 (0x55cf89808239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #55: PyObject_Call + 0x207 (0x55cf89748067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55cf8972e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #57: <unknown function> + 0x150582 (0x55cf89747582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank6]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank6]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55cf8972c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #59: <unknown function> + 0x150582 (0x55cf89747582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #60: PyObject_Call + 0xbc (0x55cf89747f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55cf8972e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #62: <unknown function> + 0x150582 (0x55cf89747582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: frame #63: PyObject_Call + 0xbc (0x55cf89747f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank5]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank6]:     return forward_call(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default6]:[rank6]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default6]:[rank6]:     pipeline_state.run_communication()
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default6]:[rank6]:     recv_activation_tensor = recv_activation()
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default6]:[rank6]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default6]:[rank6]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default6]:[rank6]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default6]:[rank6]:     dist.recv(
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default6]:[rank6]:     return func(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default6]:[rank6]:     pg.recv([tensor], group_src_rank, tag).wait()
[default6]:[rank6]: torch.distributed.DistBackendError: [6] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '5:6', but store->get('5:6') got error: Connection reset by peer
[default6]:[rank6]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default6]:[rank6]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f641363f897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default6]:[rank6]: frame #1: <unknown function> + 0x5b3a23e (0x7f644d15c23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank6]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f644d156c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank6]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f644d156f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank6]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f644d157fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank6]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f644d10c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank6]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f644d10c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank6]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f644d10c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank6]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f644d10c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank6]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f6414919189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank6]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f6414920610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank6]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f641493f978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank6]: frame #12: <unknown function> + 0x5adc309 (0x7f644d0fe309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank6]: frame #13: <unknown function> + 0x5ae6f10 (0x7f644d108f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank6]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f644d108fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank6]: frame #15: <unknown function> + 0x5124446 (0x7f644c746446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank6]: frame #16: <unknown function> + 0x1acf4b8 (0x7f64490f14b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank6]: frame #17: <unknown function> + 0x5aee004 (0x7f644d110004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank6]: frame #18: <unknown function> + 0x5af36b5 (0x7f644d1156b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank6]: frame #19: <unknown function> + 0xd2631e (0x7f645fcff31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank6]: frame #20: <unknown function> + 0x47def4 (0x7f645f456ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank6]: frame #21: <unknown function> + 0x1445a6 (0x55b70a1955a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55b70a18ea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #23: <unknown function> + 0x150866 (0x55b70a1a1866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55b70a18a142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55b70a195a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #26: PyObject_Call + 0xbc (0x55b70a1a1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55b70a1882b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55b70a195a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55b70a1868fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #30: <unknown function> + 0x150582 (0x55b70a1a1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55b70a1868fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #32: <unknown function> + 0x150582 (0x55b70a1a1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55b70a1868fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #34: <unknown function> + 0x150582 (0x55b70a1a1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55b70a1868fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55b70a18df50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55b70a19fc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #38: <unknown function> + 0x211239 (0x55b70a262239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55b70a18ea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55b70a18a3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55b70a195a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55b70a185c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55b70a195a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55b70a1868fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #45: <unknown function> + 0x150582 (0x55b70a1a1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #46: PyObject_Call + 0xbc (0x55b70a1a1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55b70a1882b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #48: <unknown function> + 0x150582 (0x55b70a1a1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #49: PyObject_Call + 0xbc (0x55b70a1a1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55b70a1882b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55b70a195a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55b70a18e007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55b70a19fc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #54: <unknown function> + 0x211239 (0x55b70a262239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #55: PyObject_Call + 0x207 (0x55b70a1a2067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55b70a1882b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #57: <unknown function> + 0x150582 (0x55b70a1a1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55b70a1868fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #59: <unknown function> + 0x150582 (0x55b70a1a1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #60: PyObject_Call + 0xbc (0x55b70a1a1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55b70a1882b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #62: <unknown function> + 0x150582 (0x55b70a1a1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: frame #63: PyObject_Call + 0xbc (0x55b70a1a1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank6]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default7]:[rank7]: Traceback (most recent call last):
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default7]:[rank7]:     trainer.train(dataloader)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default7]:[rank7]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default7]:[rank7]:     outputs = self.pipeline_engine.train_batch_iter(
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default7]:[rank7]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default7]:[rank7]:     output = model(**micro_batch)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank7]:     return self._call_impl(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank7]:     return forward_call(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default7]:[rank7]:     sharded_logits = self.model(
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank7]:     return self._call_impl(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank7]:     return forward_call(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank7]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 782, in forward_with_hidden_states
[default7]:[rank7]:     hidden_states = self.final_layer_norm(input=hidden_encoder_states["hidden_states"])["hidden_states"]
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank7]:     return self._call_impl(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank7]:     return forward_call(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default7]:[rank7]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default7]:[rank7]:     pipeline_state.run_communication()
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default7]:[rank7]:     recv_activation_tensor = recv_activation()
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default7]:[rank7]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default7]:[rank7]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default7]:[rank7]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default7]:[rank7]:     dist.recv(
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default7]:[rank7]:     return func(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default7]:[rank7]:     pg.recv([tensor], group_src_rank, tag).wait()
[default7]:[rank7]: torch.distributed.DistBackendError: [7] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '6:7', but store->get('6:7') got error: Connection reset by peer
[default7]:[rank7]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default7]:[rank7]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f333cc8d897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default7]:[rank7]: frame #1: <unknown function> + 0x5b3a23e (0x7f33767aa23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank7]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f33767a4c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank7]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f33767a4f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank7]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f33767a5fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank7]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f337675a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank7]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f337675a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank7]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f337675a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank7]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f337675a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank7]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f333df67189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank7]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f333df6e610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank7]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f333df8d978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank7]: frame #12: <unknown function> + 0x5adc309 (0x7f337674c309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank7]: frame #13: <unknown function> + 0x5ae6f10 (0x7f3376756f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank7]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f3376756fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank7]: frame #15: <unknown function> + 0x5124446 (0x7f3375d94446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank7]: frame #16: <unknown function> + 0x1acf4b8 (0x7f337273f4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank7]: frame #17: <unknown function> + 0x5aee004 (0x7f337675e004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank7]: frame #18: <unknown function> + 0x5af36b5 (0x7f33767636b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank7]: frame #19: <unknown function> + 0xd2631e (0x7f338934d31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank7]: frame #20: <unknown function> + 0x47def4 (0x7f3388aa4ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank7]: frame #21: <unknown function> + 0x1445a6 (0x56516430e5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #22: _PyObject_MakeTpCall + 0x26b (0x565164307a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #23: <unknown function> + 0x150866 (0x56516431a866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x565164303142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #25: _PyFunction_Vectorcall + 0x6c (0x56516430ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #26: PyObject_Call + 0xbc (0x56516431af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5651643012b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #28: _PyFunction_Vectorcall + 0x6c (0x56516430ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5651642ff8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #30: <unknown function> + 0x150582 (0x56516431a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5651642ff8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #32: <unknown function> + 0x150582 (0x56516431a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5651642ff8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #34: <unknown function> + 0x150582 (0x56516431a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5651642ff8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x565164306f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #37: _PyObject_Call_Prepend + 0x69 (0x565164318c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #38: <unknown function> + 0x211239 (0x5651643db239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #39: _PyObject_MakeTpCall + 0x26b (0x565164307a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5651643033e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #41: _PyFunction_Vectorcall + 0x6c (0x56516430ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5651642fec5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #43: _PyFunction_Vectorcall + 0x6c (0x56516430ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5651642ff8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #45: <unknown function> + 0x150582 (0x56516431a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #46: PyObject_Call + 0xbc (0x56516431af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5651643012b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #48: <unknown function> + 0x150582 (0x56516431a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #49: PyObject_Call + 0xbc (0x56516431af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5651643012b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #51: _PyFunction_Vectorcall + 0x6c (0x56516430ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x565164307007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #53: _PyObject_Call_Prepend + 0x69 (0x565164318c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #54: <unknown function> + 0x211239 (0x5651643db239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #55: _PyObject_MakeTpCall + 0x26b (0x565164307a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #56: _PyEval_EvalFrameDefault + 0x5723 (0x565164303c53 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #57: <unknown function> + 0x150582 (0x56516431a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5651642ff8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #59: <unknown function> + 0x150582 (0x56516431a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #60: PyObject_Call + 0xbc (0x56516431af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5651643012b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #62: <unknown function> + 0x150582 (0x56516431a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: frame #63: PyObject_Call + 0xbc (0x56516431af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank7]: . This may indicate a possible application crash on rank 0 or a network set up issue.
W0703 23:10:49.532000 140032670197568 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 350538 closing signal SIGTERM
W0703 23:10:49.533000 140032670197568 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 350539 closing signal SIGTERM
W0703 23:10:49.533000 140032670197568 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 350540 closing signal SIGTERM
W0703 23:10:49.533000 140032670197568 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 350541 closing signal SIGTERM
E0703 23:10:50.966000 140032670197568 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 350537) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-07-03_23:10:49
  host      : ip-26-0-160-225.ec2.internal
  rank      : 5 (local_rank: 5)
  exitcode  : 1 (pid: 350542)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-07-03_23:10:49
  host      : ip-26-0-160-225.ec2.internal
  rank      : 6 (local_rank: 6)
  exitcode  : 1 (pid: 350543)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2024-07-03_23:10:49
  host      : ip-26-0-160-225.ec2.internal
  rank      : 7 (local_rank: 7)
  exitcode  : 1 (pid: 350544)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-03_23:10:49
  host      : ip-26-0-160-225.ec2.internal
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 350537)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
srun: error: ip-26-0-160-225: task 0: Exited with exit code 1
Consider using `hf_transfer` for faster uploads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details.