| ======================== | |
| START TIME: Tue Jul 2 14:11:48 UTC 2024 | |
| python3 version = Python 3.10.14 | |
| ======================== | |
| The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well. | |
| Token is valid (permission: write). | |
| Your token has been saved to /admin/home/ferdinand_mom/.cache/huggingface/token | |
| Login successful | |
| Already on 'bench_cluster' | |
| M examples/config_tiny_llama.py | |
| M examples/config_tiny_llama.yaml | |
| M examples/train_tiny_llama.sh | |
| M src/nanotron/models/llama.py | |
| M src/nanotron/trainer.py | |
| Your branch is up to date with 'origin/bench_cluster'. | |
| Job status: RUNNING | |
| W0702 14:11:51.290000 140655077304128 torch/distributed/run.py:757] | |
| W0702 14:11:51.290000 140655077304128 torch/distributed/run.py:757] ***************************************** | |
| W0702 14:11:51.290000 140655077304128 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. | |
| W0702 14:11:51.290000 140655077304128 torch/distributed/run.py:757] ***************************************** | |
| W0702 14:11:51.290000 140614650271552 torch/distributed/run.py:757] | |
| W0702 14:11:51.290000 140614650271552 torch/distributed/run.py:757] ***************************************** | |
| W0702 14:11:51.290000 140614650271552 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. | |
| W0702 14:11:51.290000 140614650271552 torch/distributed/run.py:757] ***************************************** | |
| [default0]:07/02/2024 14:12:09 [WARNING|DP=0|PP=0|TP=0|ip-26-0-161-178]: [Vocab Size Padding] Padded vocab (size: 50257) with 7 dummy tokens (new size: 50264) | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: Config: | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: Config(general=GeneralArgs(project='bench_cluster', | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: run='%date_%jobid', | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: seed=42, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: step=None, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: consumed_train_samples=None, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: benchmark_csv_path=None, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: ignore_sanity_checks=True), | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: parallelism=ParallelismArgs(dp=1, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: pp=2, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: tp=8, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: pp_engine=<nanotron.parallel.pipeline_parallel.engine.OneForwardOneBackwardPipelineEngine object at 0x7f1996cf06a0>, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: tp_mode=<TensorParallelLinearMode.REDUCE_SCATTER: 2>, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: tp_linear_async_communication=False, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: expert_parallel_size=1), | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: model=ModelArgs(model_config=LlamaConfig(bos_token_id=1, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: eos_token_id=2, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: hidden_act='silu', | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: hidden_size=2048, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: initializer_range=0.02, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: intermediate_size=4096, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: is_llama_config=True, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: max_position_embeddings=4096, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: num_attention_heads=32, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: num_hidden_layers=24, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: num_key_value_heads=32, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: pad_token_id=None, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: pretraining_tp=1, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: rms_norm_eps=1e-05, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: rope_scaling=None, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: rope_theta=10000.0, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: tie_word_embeddings=True, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: use_cache=True, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: vocab_size=50264), | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: init_method=RandomInit(std=0.025), | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: dtype=torch.bfloat16, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: make_vocab_size_divisible_by=1, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: ddp_bucket_cap_mb=25), | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: tokenizer=TokenizerArgs(tokenizer_name_or_path='openai-community/gpt2', | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: tokenizer_revision=None, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: tokenizer_max_length=None), | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: checkpoints=CheckpointsArgs(checkpoints_path=Path('/dev/null'), | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: checkpoint_interval=100000, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: save_initial_state=False, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: resume_checkpoint_path=None, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: checkpoints_path_is_shared_file_system=False), | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: logging=LoggingArgs(log_level='info', | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: log_level_replica='info', | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: iteration_step_info_interval=1), | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: tokens=TokensArgs(sequence_length=4096, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: train_steps=20, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: micro_batch_size=1024, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: batch_accumulation_per_replica=1, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: val_check_interval=-1, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: limit_val_batches=0, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: limit_test_batches=0), | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: optimizer=OptimizerArgs(optimizer_factory=AdamWOptimizerArgs(adam_eps=1e-08, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: adam_beta1=0.9, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: adam_beta2=0.95, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: torch_adam_is_fused=True, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: name='adamW'), | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: zero_stage=1, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: weight_decay=0.01, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: clip_grad=1.0, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: accumulate_grad_in_fp32=True, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: learning_rate_scheduler=LRSchedulerArgs(learning_rate=0.0001, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: lr_warmup_steps=1, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: lr_warmup_style='linear', | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: lr_decay_style='linear', | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: lr_decay_steps=19, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: lr_decay_starting_step=None, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: min_decay_lr=1e-05)), | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: data_stages=[DatasetStageArgs(name='Training Stage', | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: start_training_step=1, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: data=DataArgs(dataset=PretrainDatasetsArgs(hf_dataset_or_datasets='roneneldan/TinyStories', | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: hf_dataset_splits='train', | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: hf_dataset_config_name=None, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: dataset_processing_num_proc_per_process=64, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: dataset_overwrite_cache=False, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: text_column_name='text'), | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: seed=42, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: num_loading_workers=32))], | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: profiler=ProfilerArgs(profiler_export_path=Path('/fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/16_GPUS/dp-1_tp-8_pp-2_mbz-1024')), | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: lighteval=None) | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: Model Config: | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: LlamaConfig(bos_token_id=1, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: eos_token_id=2, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: hidden_act='silu', | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: hidden_size=2048, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: initializer_range=0.02, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: intermediate_size=4096, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: is_llama_config=True, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: max_position_embeddings=4096, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: num_attention_heads=32, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: num_hidden_layers=24, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: num_key_value_heads=32, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: pad_token_id=None, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: pretraining_tp=1, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: rms_norm_eps=1e-05, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: rope_scaling=None, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: rope_theta=10000.0, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: tie_word_embeddings=True, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: use_cache=True, | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: vocab_size=50264) | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: Building model.. | |
| [default0]:07/02/2024 14:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: Setting PP block ranks... | |
| [default3]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=3|ip-26-0-163-147]: Local number of parameters: 65.3M (124.62MiB) | |
| [default6]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=6|ip-26-0-163-147]: Local number of parameters: 65.3M (124.62MiB) | |
| [default6]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=6|ip-26-0-163-147]: [After model building] Memory usage: 135.64MiB. Peak allocated: 137.67MiB Peak reserved: 150.00MiB | |
| [default6]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=6|ip-26-0-163-147]: No checkpoint path provided. | |
| [default3]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=3|ip-26-0-163-147]: [After model building] Memory usage: 135.64MiB. Peak allocated: 137.67MiB Peak reserved: 150.00MiB | |
| [default3]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=3|ip-26-0-163-147]: No checkpoint path provided. | |
| [default0]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=0|ip-26-0-163-147]: Local number of parameters: 65.3M (124.62MiB) | |
| [default0]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=0|ip-26-0-163-147]: [After model building] Memory usage: 135.64MiB. Peak allocated: 137.67MiB Peak reserved: 150.00MiB | |
| [default0]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=0|ip-26-0-163-147]: No checkpoint path provided. | |
| [default1]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=1|ip-26-0-163-147]: Local number of parameters: 65.3M (124.62MiB) | |
| [default1]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=1|ip-26-0-163-147]: [After model building] Memory usage: 135.64MiB. Peak allocated: 137.67MiB Peak reserved: 150.00MiB | |
| [default1]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=1|ip-26-0-163-147]: No checkpoint path provided. | |
| [default5]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=5|ip-26-0-163-147]: Local number of parameters: 65.3M (124.62MiB) | |
| [default5]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=5|ip-26-0-163-147]: [After model building] Memory usage: 135.64MiB. Peak allocated: 137.67MiB Peak reserved: 150.00MiB | |
| [default5]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=5|ip-26-0-163-147]: No checkpoint path provided. | |
| [default4]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=4|ip-26-0-163-147]: Local number of parameters: 65.3M (124.62MiB) | |
| [default4]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=4|ip-26-0-163-147]: [After model building] Memory usage: 135.64MiB. Peak allocated: 137.67MiB Peak reserved: 150.00MiB | |
| [default0]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: Total number of parameters: 1.21G (2314.22MiB) | |
| [default0]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: Local number of parameters: 86.3M (164.65MiB) | |
| [default0]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: [After model building] Memory usage: 179.67MiB. Peak allocated: 181.70MiB Peak reserved: 198.00MiB | |
| [default0]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: No checkpoint path provided. | |
| [default0]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: Parametrizing model parameters using StandardParametrizator | |
| [default6]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=6|ip-26-0-161-178]: Local number of parameters: 86.3M (164.65MiB) | |
| [default6]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=6|ip-26-0-161-178]: [After model building] Memory usage: 179.67MiB. Peak allocated: 181.70MiB Peak reserved: 198.00MiB | |
| [default6]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=6|ip-26-0-161-178]: No checkpoint path provided. | |
| [default4]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=4|ip-26-0-163-147]: No checkpoint path provided. | |
| [default7]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=7|ip-26-0-163-147]: Local number of parameters: 65.3M (124.62MiB) | |
| [default7]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=7|ip-26-0-163-147]: [After model building] Memory usage: 135.64MiB. Peak allocated: 137.67MiB Peak reserved: 150.00MiB | |
| [default7]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=7|ip-26-0-163-147]: No checkpoint path provided. | |
| [default2]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=2|ip-26-0-163-147]: Local number of parameters: 65.3M (124.62MiB) | |
| [default2]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=2|ip-26-0-163-147]: [After model building] Memory usage: 135.64MiB. Peak allocated: 137.67MiB Peak reserved: 150.00MiB | |
| [default2]:07/02/2024 14:12:25 [INFO|DP=0|PP=1|TP=2|ip-26-0-163-147]: No checkpoint path provided. | |
| [default3]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=3|ip-26-0-161-178]: Local number of parameters: 86.3M (164.65MiB) | |
| [default3]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=3|ip-26-0-161-178]: [After model building] Memory usage: 179.67MiB. Peak allocated: 181.70MiB Peak reserved: 198.00MiB | |
| [default3]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=3|ip-26-0-161-178]: No checkpoint path provided. | |
| [default2]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=2|ip-26-0-161-178]: Local number of parameters: 86.3M (164.65MiB) | |
| [default2]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=2|ip-26-0-161-178]: [After model building] Memory usage: 179.67MiB. Peak allocated: 181.70MiB Peak reserved: 198.00MiB | |
| [default2]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=2|ip-26-0-161-178]: No checkpoint path provided. | |
| [default1]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=1|ip-26-0-161-178]: Local number of parameters: 86.3M (164.65MiB) | |
| [default1]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=1|ip-26-0-161-178]: [After model building] Memory usage: 179.67MiB. Peak allocated: 181.70MiB Peak reserved: 198.00MiB | |
| [default1]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=1|ip-26-0-161-178]: No checkpoint path provided. | |
| [default4]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=4|ip-26-0-161-178]: Local number of parameters: 86.3M (164.65MiB) | |
| [default4]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=4|ip-26-0-161-178]: [After model building] Memory usage: 179.67MiB. Peak allocated: 181.70MiB Peak reserved: 198.00MiB | |
| [default4]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=4|ip-26-0-161-178]: No checkpoint path provided. | |
| [default7]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=7|ip-26-0-161-178]: Local number of parameters: 86.3M (164.65MiB) | |
| [default5]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=5|ip-26-0-161-178]: Local number of parameters: 86.3M (164.65MiB) | |
| [default7]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=7|ip-26-0-161-178]: [After model building] Memory usage: 179.67MiB. Peak allocated: 181.70MiB Peak reserved: 198.00MiB | |
| [default7]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=7|ip-26-0-161-178]: No checkpoint path provided. | |
| [default5]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=5|ip-26-0-161-178]: [After model building] Memory usage: 179.67MiB. Peak allocated: 181.70MiB Peak reserved: 198.00MiB | |
| [default5]:07/02/2024 14:12:25 [INFO|DP=0|PP=0|TP=5|ip-26-0-161-178]: No checkpoint path provided. | |
| [default0]:07/02/2024 14:12:26 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: [Optimizer Building] Using LearningRateForSP as learning rate | |
| [default0]:07/02/2024 14:12:26 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: [ZeRO sharding] Size of optimizer params per rank: | |
| [default0]:07/02/2024 14:12:26 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: [ZeRO sharding] DP Rank 0 has 86.3M out of 86.3M (100.00%) params' optimizer states | |
| [default0]:07/02/2024 14:12:27 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: [Training Plan] Stage Training Stage has 19 remaining training steps and has consumed 0 samples | |
| [default0]:07/02/2024 14:12:27 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: Using `datasets` library | |
| [default0]:07/02/2024 14:12:27 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: Loading tokenizer from openai-community/gpt2 and transformers/hf_hub versions ('4.41.2', '0.23.4') | |
| [default0]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:07/02/2024 14:12:27 [WARNING|DP=0|PP=0|TP=0|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:07/02/2024 14:12:28 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: [Training Plan] There are 1 training stages | |
| [default0]:07/02/2024 14:12:28 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: [Stage Training Stage] start from step 1 | |
| [default0]:07/02/2024 14:12:28 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: | |
| [default0]:07/02/2024 14:12:28 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: [Start training] datetime: 2024-07-02 14:12:28.670725 | mbs: 1024 | grad_accum: 1 | global_batch_size: 1024 | sequence_length: 4096 | train_steps: 20 | start_iteration_step: 0 | consumed_train_samples: 0 | |
| [default0]:07/02/2024 14:12:28 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: Resuming training from stage Training Stage, it has trained for 0 samples and has 19 remaining train steps | |
| [default0]:07/02/2024 14:12:28 [INFO|DP=0|PP=0|TP=0|ip-26-0-161-178]: Memory usage: 839.67MiB. Peak allocated 839.67MiB. Peak reserved: 858.00MiB | |
| [default0]:07/02/2024 14:12:28 [WARNING|DP=0|PP=1|TP=0|ip-26-0-163-147]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:07/02/2024 14:12:28 [WARNING|DP=0|PP=1|TP=6|ip-26-0-163-147]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:07/02/2024 14:12:28 [WARNING|DP=0|PP=1|TP=1|ip-26-0-163-147]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:07/02/2024 14:12:28 [WARNING|DP=0|PP=1|TP=2|ip-26-0-163-147]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:07/02/2024 14:12:28 [WARNING|DP=0|PP=0|TP=3|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:07/02/2024 14:12:28 [WARNING|DP=0|PP=0|TP=2|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:07/02/2024 14:12:28 [WARNING|DP=0|PP=0|TP=7|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:07/02/2024 14:12:28 [WARNING|DP=0|PP=0|TP=5|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:07/02/2024 14:12:28 [WARNING|DP=0|PP=1|TP=3|ip-26-0-163-147]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:07/02/2024 14:12:28 [WARNING|DP=0|PP=1|TP=5|ip-26-0-163-147]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:07/02/2024 14:12:28 [WARNING|DP=0|PP=1|TP=4|ip-26-0-163-147]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:07/02/2024 14:12:28 [WARNING|DP=0|PP=1|TP=7|ip-26-0-163-147]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:07/02/2024 14:12:28 [WARNING|DP=0|PP=0|TP=6|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:07/02/2024 14:12:28 [WARNING|DP=0|PP=0|TP=1|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:07/02/2024 14:12:28 [WARNING|DP=0|PP=0|TP=4|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:[rank7]: Traceback (most recent call last): | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default7]:[rank7]: trainer.train(dataloader) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default7]:[rank7]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default7]:[rank7]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default7]:[rank7]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default7]:[rank7]: output = model(**micro_batch) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default7]:[rank7]: sharded_logits = self.model( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default7]:[rank7]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default7]:[rank7]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward | |
| [default7]:[rank7]: output = self.pp_block(**new_kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward | |
| [default7]:[rank7]: hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"] | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward | |
| [default7]:[rank7]: hidden_states = self.down_proj(self.split_silu_mul(merged_states)) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 128, in forward | |
| [default7]:[rank7]: return self.act(gate_states) * up_states | |
| [default7]:[rank7]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.00 GiB. GPU has a total capacity of 79.33 GiB of which 1.60 GiB is free. Including non-PyTorch memory, this process has 77.71 GiB memory in use. Of the allocated memory 66.99 GiB is allocated by PyTorch, and 1.94 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) | |
| [default5]:[rank5]: Traceback (most recent call last): | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default5]:[rank5]: trainer.train(dataloader) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default5]:[rank5]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default5]:[rank5]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default5]:[rank5]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default5]:[rank5]: output = model(**micro_batch) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default5]:[rank5]: sharded_logits = self.model( | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default5]:[rank5]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default5]:[rank5]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward | |
| [default5]:[rank5]: output = self.pp_block(**new_kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward | |
| [default5]:[rank5]: hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"] | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward | |
| [default5]:[rank5]: hidden_states = self.down_proj(self.split_silu_mul(merged_states)) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 128, in forward | |
| [default5]:[rank5]: return self.act(gate_states) * up_states | |
| [default5]:[rank5]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.00 GiB. GPU has a total capacity of 79.33 GiB of which 1.32 GiB is free. Including non-PyTorch memory, this process has 77.99 GiB memory in use. Of the allocated memory 66.99 GiB is allocated by PyTorch, and 1.94 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) | |
| [default4]:[rank4]: Traceback (most recent call last): | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default4]:[rank4]: trainer.train(dataloader) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default4]:[rank4]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default4]:[rank4]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default4]:[rank4]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default4]:[rank4]: output = model(**micro_batch) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default4]:[rank4]: sharded_logits = self.model( | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default4]:[rank4]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default4]:[rank4]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward | |
| [default4]:[rank4]: output = self.pp_block(**new_kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward | |
| [default4]:[rank4]: hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"] | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward | |
| [default4]:[rank4]: hidden_states = self.down_proj(self.split_silu_mul(merged_states)) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 128, in forward | |
| [default4]:[rank4]: return self.act(gate_states) * up_states | |
| [default4]:[rank4]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.00 GiB. GPU has a total capacity of 79.33 GiB of which 1.32 GiB is free. Including non-PyTorch memory, this process has 77.99 GiB memory in use. Of the allocated memory 66.99 GiB is allocated by PyTorch, and 1.94 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) | |
| [default2]:[rank2]: Traceback (most recent call last): | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default2]:[rank2]: trainer.train(dataloader) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default2]:[rank2]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default2]:[rank2]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default2]:[rank2]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default2]:[rank2]: output = model(**micro_batch) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank2]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank2]: return forward_call(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default2]:[rank2]: sharded_logits = self.model( | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank2]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank2]: return forward_call(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default2]:[rank2]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default2]:[rank2]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank2]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank2]: return forward_call(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward | |
| [default2]:[rank2]: output = self.pp_block(**new_kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank2]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank2]: return forward_call(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward | |
| [default2]:[rank2]: hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"] | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank2]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank2]: return forward_call(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward | |
| [default2]:[rank2]: hidden_states = self.down_proj(self.split_silu_mul(merged_states)) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank2]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank2]: return forward_call(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 128, in forward | |
| [default2]:[rank2]: return self.act(gate_states) * up_states | |
| [default2]:[rank2]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.00 GiB. GPU has a total capacity of 79.33 GiB of which 1.48 GiB is free. Including non-PyTorch memory, this process has 77.83 GiB memory in use. Of the allocated memory 66.99 GiB is allocated by PyTorch, and 1.94 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) | |
| [default6]:[rank6]: Traceback (most recent call last): | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default6]:[rank6]: trainer.train(dataloader) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default6]:[rank6]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default6]:[rank6]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default6]:[rank6]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default6]:[rank6]: output = model(**micro_batch) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default6]:[rank6]: sharded_logits = self.model( | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default6]:[rank6]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default6]:[rank6]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward | |
| [default6]:[rank6]: output = self.pp_block(**new_kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward | |
| [default6]:[rank6]: hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"] | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward | |
| [default6]:[rank6]: hidden_states = self.down_proj(self.split_silu_mul(merged_states)) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 128, in forward | |
| [default6]:[rank6]: return self.act(gate_states) * up_states | |
| [default6]:[rank6]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.00 GiB. GPU has a total capacity of 79.33 GiB of which 1.32 GiB is free. Including non-PyTorch memory, this process has 77.99 GiB memory in use. Of the allocated memory 66.99 GiB is allocated by PyTorch, and 1.94 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) | |
| [default1]:[rank1]: Traceback (most recent call last): | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default1]:[rank1]: trainer.train(dataloader) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default1]:[rank1]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default1]:[rank1]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default1]:[rank1]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default1]:[rank1]: output = model(**micro_batch) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default1]:[rank1]: sharded_logits = self.model( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default1]:[rank1]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default1]:[rank1]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward | |
| [default1]:[rank1]: output = self.pp_block(**new_kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward | |
| [default1]:[rank1]: hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"] | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward | |
| [default1]:[rank1]: hidden_states = self.down_proj(self.split_silu_mul(merged_states)) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 128, in forward | |
| [default1]:[rank1]: return self.act(gate_states) * up_states | |
| [default1]:[rank1]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.00 GiB. GPU has a total capacity of 79.33 GiB of which 1.58 GiB is free. Including non-PyTorch memory, this process has 77.72 GiB memory in use. Of the allocated memory 66.99 GiB is allocated by PyTorch, and 1.94 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) | |
| [default0]:[rank0]: Traceback (most recent call last): | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default0]:[rank0]: trainer.train(dataloader) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default0]:[rank0]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default0]:[rank0]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default0]:[rank0]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default0]:[rank0]: output = model(**micro_batch) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default0]:[rank0]: sharded_logits = self.model( | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default0]:[rank0]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default0]:[rank0]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward | |
| [default0]:[rank0]: output = self.pp_block(**new_kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward | |
| [default0]:[rank0]: hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"] | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward | |
| [default0]:[rank0]: hidden_states = self.down_proj(self.split_silu_mul(merged_states)) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 128, in forward | |
| [default0]:[rank0]: return self.act(gate_states) * up_states | |
| [default0]:[rank0]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.00 GiB. GPU | |
| [default2]:Exception in thread Thread-2 (_pin_memory_loop): | |
| [default2]:Traceback (most recent call last): | |
| [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/threading.py", line 1016, in _bootstrap_inner | |
| [default2]: self.run() | |
| [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/threading.py", line 953, in run | |
| [default2]: self._target(*self._args, **self._kwargs) | |
| [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 54, in _pin_memory_loop | |
| [default2]: do_one_step() | |
| [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 31, in do_one_step | |
| [default2]: r = in_queue.get(timeout=MP_STATUS_CHECK_INTERVAL) | |
| [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/queues.py", line 122, in get | |
| [default2]: return _ForkingPickler.loads(res) | |
| [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/multiprocessing/reductions.py", line 495, in rebuild_storage_fd | |
| [default2]: fd = df.detach() | |
| [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/resource_sharer.py", line 57, in detach | |
| [default2]: with _resource_sharer.get_connection(self._id) as conn: | |
| [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/resource_sharer.py", line 86, in get_connection | |
| [default2]: c = Client(address, authkey=process.current_process().authkey) | |
| [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 502, in Client | |
| [default2]: c = SocketClient(address) | |
| [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 630, in SocketClient | |
| [default2]: s.connect(address) | |
| [default2]:ConnectionRefusedError: [Errno 111] Connection refused | |
| [default6]:Exception in thread Thread-2 (_pin_memory_loop): | |
| [default6]:Traceback (most recent call last): | |
| [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/threading.py", line 1016, in _bootstrap_inner | |
| [default6]: self.run() | |
| [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/threading.py", line 953, in run | |
| [default6]: self._target(*self._args, **self._kwargs) | |
| [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 54, in _pin_memory_loop | |
| [default6]: do_one_step() | |
| [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 31, in do_one_step | |
| [default6]: r = in_queue.get(timeout=MP_STATUS_CHECK_INTERVAL) | |
| [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/queues.py", line 122, in get | |
| [default6]: return _ForkingPickler.loads(res) | |
| [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/multiprocessing/reductions.py", line 495, in rebuild_storage_fd | |
| [default6]: fd = df.detach() | |
| [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/resource_sharer.py", line 57, in detach | |
| [default6]: with _resource_sharer.get_connection(self._id) as conn: | |
| [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/resource_sharer.py", line 86, in get_connection | |
| [default6]: c = Client(address, authkey=process.current_process().authkey) | |
| [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 508, in Client | |
| [default6]: answer_challenge(c, authkey) | |
| [default3]:[rank3]: Traceback (most recent call last): | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default3]:[rank3]: trainer.train(dataloader) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default3]:[rank3]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default3]:[rank3]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default3]:[rank3]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default3]:[rank3]: output = model(**micro_batch) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank3]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank3]: return forward_call(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default3]:[rank3]: sharded_logits = self.model( | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank3]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank3]: return forward_call(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default3]:[rank3]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default3]:[rank3]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank3]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank3]: return forward_call(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward | |
| [default3]:[rank3]: output = self.pp_block(**new_kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank3]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank3]: return forward_call(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward | |
| [default3]:[rank3]: hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"] | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank3]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank3]: return forward_call(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward | |
| [default3]:[rank3]: hidden_states = self.down_proj(self.split_silu_mul(merged_states)) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank3]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank3]: return forward_call(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 128, in forward | |
| [default3]:[rank3]: return self.act(gate_states) * up_states | |
| [default3]:[rank3]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.00 GiB. GPU has a total capacity of 79.33 GiB of which 1.45 GiB is free. Including non-PyTorch memory, this process has 77.86 GiB memory in use. Of the allocated memory 66.99 GiB is allocated by PyTorch, and 1.94 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) | |
| [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 752, in answer_challenge | |
| [default6]: message = connection.recv_bytes(256) # reject large message | |
| [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes | |
| [default6]: buf = self._recv_bytes(maxlength) | |
| [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes | |
| [default6]: buf = self._recv(4) | |
| [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 379, in _recv | |
| [default6]: chunk = read(handle, remaining) | |
| [default6]:ConnectionResetError: [Errno 104] Connection reset by peer | |
| [default0]:Exception in thread Thread-2 (_pin_memory_loop): | |
| [default0]:Traceback (most recent call last): | |
| [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/threading.py", line 1016, in _bootstrap_inner | |
| [default0]: self.run() | |
| [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/threading.py", line 953, in run | |
| [default0]: self._target(*self._args, **self._kwargs) | |
| [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 54, in _pin_memory_loop | |
| [default0]: do_one_step() | |
| [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 31, in do_one_step | |
| [default0]: r = in_queue.get(timeout=MP_STATUS_CHECK_INTERVAL) | |
| [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/queues.py", line 122, in get | |
| [default0]: return _ForkingPickler.loads(res) | |
| [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/multiprocessing/reductions.py", line 495, in rebuild_storage_fd | |
| [default0]: fd = df.detach() | |
| [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/resource_sharer.py", line 57, in detach | |
| [default0]: with _resource_sharer.get_connection(self._id) as conn: | |
| [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/resource_sharer.py", line 86, in get_connection | |
| [default0]: c = Client(address, authkey=process.current_process().authkey) | |
| [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 508, in Client | |
| [default0]: answer_challenge(c, authkey) | |
| [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 752, in answer_challenge | |
| [default0]: message = connection.recv_bytes(256) # reject large message | |
| [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes | |
| [default0]: buf = self._recv_bytes(maxlength) | |
| [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes | |
| [default0]: buf = self._recv(4) | |
| [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 379, in _recv | |
| [default0]: chunk = read(handle, remaining) | |
| [default0]:ConnectionResetError: [Errno 104] Connection reset by peer | |
| [default1]:Exception in thread Thread-2 (_pin_memory_loop): | |
| [default1]:Traceback (most recent call last): | |
| [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/threading.py", line 1016, in _bootstrap_inner | |
| [default1]: self.run() | |
| [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/threading.py", line 953, in run | |
| [default1]: self._target(*self._args, **self._kwargs) | |
| [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 54, in _pin_memory_loop | |
| [default1]: do_one_step() | |
| [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 31, in do_one_step | |
| [default1]: r = in_queue.get(timeout=MP_STATUS_CHECK_INTERVAL) | |
| [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/queues.py", line 122, in get | |
| [default1]: return _ForkingPickler.loads(res) | |
| [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/multiprocessing/reductions.py", line 495, in rebuild_storage_fd | |
| [default1]: fd = df.detach() | |
| [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/resource_sharer.py", line 57, in detach | |
| [default1]: with _resource_sharer.get_connection(self._id) as conn: | |
| [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/resource_sharer.py", line 86, in get_connection | |
| [default1]: c = Client(address, authkey=process.current_process().authkey) | |
| [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 508, in Client | |
| [default1]: answer_challenge(c, authkey) | |
| [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 752, in answer_challenge | |
| [default1]: message = connection.recv_bytes(256) # reject large message | |
| [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes | |
| [default1]: buf = self._recv_bytes(maxlength) | |
| [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes | |
| [default1]: buf = self._recv(4) | |
| [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 379, in _recv | |
| [default1]: chunk = read(handle, remaining) | |
| [default1]:ConnectionResetError: [Errno 104] Connection reset by peer | |
| [default3]:Exception in thread Thread-2 (_pin_memory_loop): | |
| [default3]:Traceback (most recent call last): | |
| [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/threading.py", line 1016, in _bootstrap_inner | |
| [default3]: self.run() | |
| [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/threading.py", line 953, in run | |
| [default3]: self._target(*self._args, **self._kwargs) | |
| [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 54, in _pin_memory_loop | |
| [default3]: do_one_step() | |
| [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 31, in do_one_step | |
| [default3]: r = in_queue.get(timeout=MP_STATUS_CHECK_INTERVAL) | |
| [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/queues.py", line 122, in get | |
| [default3]: return _ForkingPickler.loads(res) | |
| [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/multiprocessing/reductions.py", line 495, in rebuild_storage_fd | |
| [default3]: fd = df.detach() | |
| [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/resource_sharer.py", line 57, in detach | |
| [default3]: with _resource_sharer.get_connection(self._id) as conn: | |
| [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/resource_sharer.py", line 86, in get_connection | |
| [default3]: c = Client(address, authkey=process.current_process().authkey) | |
| [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 508, in Client | |
| [default3]: answer_challenge(c, authkey) | |
| [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 752, in answer_challenge | |
| [default3]: message = connection.recv_bytes(256) # reject large message | |
| [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes | |
| [default3]: buf = self._recv_bytes(maxlength) | |
| [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes | |
| [default3]: buf = self._recv(4) | |
| [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 379, in _recv | |
| [default1]:[rank9]: Traceback (most recent call last): | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default1]:[rank9]: trainer.train(dataloader) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default1]:[rank9]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default1]:[rank9]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default1]:[rank9]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default1]:[rank9]: output = model(**micro_batch) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank9]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank9]: return forward_call(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default1]:[rank9]: sharded_logits = self.model( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank9]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank9]: return forward_call(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default1]:[rank9]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default1]:[rank9]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank9]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank9]: return forward_call(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default1]:[rank9]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default1]:[rank9]: pipeline_state.run_communication() | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default1]:[rank9]: recv_activation_tensor = recv_activation() | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default1]:[rank9]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default1]:[rank9]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default1]:[rank9]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default1]:[rank9]: dist.recv( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default1]:[rank9]: return func(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default1]:[rank9]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default5]:[rank13]: Traceback (most recent call last): | |
| [default1]:[rank9]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default1]:[rank9]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default5]:[rank13]: trainer.train(dataloader) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default5]:[rank13]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default5]:[rank13]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default5]:[rank13]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default5]:[rank13]: output = model(**micro_batch) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank13]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank13]: return forward_call(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default5]:[rank13]: sharded_logits = self.model( | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank13]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank13]: return forward_call(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default5]:[rank13]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default5]:[rank13]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank13]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank13]: return forward_call(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default5]:[rank13]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default5]:[rank13]: pipeline_state.run_communication() | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default5]:[rank13]: recv_activation_tensor = recv_activation() | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default5]:[rank13]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default5]:[rank13]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default5]:[rank13]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default5]:[rank13]: dist.recv( | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default6]:[rank14]: Traceback (most recent call last): | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default6]:[rank14]: trainer.train(dataloader) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default1]:[rank9]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fa645a5d897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default1]:[rank9]: frame #1: <unknown function> + 0x5b3a23e (0x7fa67f57a23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default6]:[rank14]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default6]:[rank14]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default6]:[rank14]: output = model(**micro_batch) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank14]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank9]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fa67f574c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fa67f574f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: Traceback (most recent call last): | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default3]:[rank11]: trainer.train(dataloader) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default3]:[rank11]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default1]:[rank9]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fa67f575fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default3]:[rank11]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default3]:[rank11]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default1]:[rank9]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa67f52a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: return func(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default5]:[rank13]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default5]:[rank13]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default0]:[rank8]: Traceback (most recent call last): | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default3]:[rank11]: output = model(**micro_batch) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank11]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank13]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default5]:[rank13]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f4856489897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default0]:[rank8]: trainer.train(dataloader) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank11]: return forward_call(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default3]:[rank11]: sharded_logits = self.model( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank14]: return forward_call(*args, **kwargs) | |
| [default5]:[rank13]: frame #1: <unknown function> + 0x5b3a23e (0x7f488ffa623e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f488ffa0c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default5]:[rank13]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f488ffa0f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f488ffa1fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f488ff56371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f488ff56371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f488ff56371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f488ff56371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa67f52a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa67f52a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa67f52a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fa646d37189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default1]:[rank9]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fa646d3e610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default1]:[rank9]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fa646d5d978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default1]:[rank9]: frame #12: <unknown function> + 0x5adc309 (0x7fa67f51c309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #13: <unknown function> + 0x5ae6f10 (0x7fa67f526f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fa67f526fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #15: <unknown function> + 0x5124446 (0x7fa67eb64446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #16: <unknown function> + 0x1acf4b8 (0x7fa67b50f4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #17: <unknown function> + 0x5aee004 (0x7fa67f52e004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #18: <unknown function> + 0x5af36b5 (0x7fa67f5336b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #19: <unknown function> + 0xd2631e (0x7fa69211d31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default1]:[rank9]: frame #20: <unknown function> + 0x47def4 (0x7fa691874ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default1]:[rank9]: frame #21: <unknown function> + 0x1445a6 (0x563e6636b5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #22: _PyObject_MakeTpCall + 0x26b (0x563e66364a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #23: <unknown function> + 0x150866 (0x563e66377866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x563e66360142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #25: _PyFunction_Vectorcall + 0x6c (0x563e6636ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #26: PyObject_Call + 0xbc (0x563e66377f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x563e6635e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #28: _PyFunction_Vectorcall + 0x6c (0x563e6636ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x563e6635c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #30: <unknown function> + 0x150582 (0x563e66377582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x563e6635c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #32: <unknown function> + 0x150582 (0x563e66377582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x563e6635c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #34: <unknown function> + 0x150582 (0x563e66377582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x563e6635c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x563e66363f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #37: _PyObject_Call_Prepend + 0x69 (0x563e66375c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #38: <unknown function> + 0x211239 (0x563e66438239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #39: _PyObject_MakeTpCall + 0x26b (0x563e66364a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x563e663603e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #41: _PyFunction_Vectorcall + 0x6c (0x563e6636ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x563e6635bc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #43: _PyFunction_Vectorcall + 0x6c (0x563e6636ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x563e6635c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #45: <unknown function> + 0x150582 (0x563e66377582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #46: PyObject_Call + 0xbc (0x563e66377f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x563e6635e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #48: <unknown function> + 0x150582 (0x563e66377582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #49: PyObject_Call + 0xbc (0x563e66377f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x563e6635e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #51: _PyFunction_Vectorcall + 0x6c (0x563e6636ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x563e66364007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #53: _PyObject_Call_Prepend + 0x69 (0x563e66375c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #54: <unknown function> + 0x211239 (0x563e66438239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #55: PyObject_Call + 0x207 (0x563e66378067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x563e6635e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #57: <unknown function> + 0x150582 (0x563e66377582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x563e6635c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #59: <unknown function> + 0x150582 (0x563e66377582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #60: PyObject_Call + 0xbc (0x563e66377f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x563e6635e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #62: <unknown function> + 0x150582 (0x563e66377582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #63: PyObject_Call + 0xbc (0x563e66377f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default0]:[rank8]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default0]:[rank8]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default0]:[rank8]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default0]:[rank8]: output = model(**micro_batch) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank8]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank8]: return forward_call(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default0]:[rank8]: sharded_logits = self.model( | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank8]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank8]: return forward_call(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default0]:[rank8]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default0]:[rank8]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank8]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank14]: sharded_logits = self.model( | |
| [default0]:[rank8]: return forward_call(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default5]:[rank13]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f4857763189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:[rank13]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f485776a610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank14]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank8]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default0]:[rank8]: pipeline_state.run_communication() | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank14]: return forward_call(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default6]:[rank14]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default6]:[rank14]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank14]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank14]: return forward_call(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default6]:[rank14]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default6]:[rank14]: pipeline_state.run_communication() | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default6]:[rank14]: recv_activation_tensor = recv_activation() | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default6]:[rank14]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default6]:[rank14]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default6]:[rank14]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default6]:[rank14]: dist.recv( | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default6]:[rank14]: return func(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default6]:[rank14]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default6]:[rank14]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default6]:[rank14]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default6]:[rank14]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fb28bda2897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default6]:[rank14]: frame #1: <unknown function> + 0x5b3a23e (0x7fb2c58bf23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fb2c58b9c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fb2c58b9f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fb2c58bafd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb2c586f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb2c586f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb2c586f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb2c586f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fb28d07c189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank14]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fb28d083610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank14]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fb28d0a2978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank14]: frame #12: <unknown function> + 0x5adc309 (0x7fb2c5861309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #13: <unknown function> + 0x5ae6f10 (0x7fb2c586bf10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fb2c586bfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #15: <unknown function> + 0x5124446 (0x7fb2c4ea9446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #16: <unknown function> + 0x1acf4b8 (0x7fb2c18544b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #17: <unknown function> + 0x5aee004 (0x7fb2c5873004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #18: <unknown function> + 0x5af36b5 (0x7fb2c58786b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #19: <unknown function> + 0xd2631e (0x7fb2d846231e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default6]:[rank14]: frame #20: <unknown function> + 0x47def4 (0x7fb2d7bb9ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default6]:[rank14]: frame #21: <unknown function> + 0x1445a6 (0x55e3feca25a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55e3fec9ba6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #23: <unknown function> + 0x150866 (0x55e3fecae866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55e3fec97142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55e3feca2a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #26: PyObject_Call + 0xbc (0x55e3fecaef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55e3fec952b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55e3feca2a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55e3fec938fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #30: <unknown function> + 0x150582 (0x55e3fecae582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55e3fec938fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #32: <unknown function> + 0x150582 (0x55e3fecae582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55e3fec938fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #34: <unknown function> + 0x150582 (0x55e3fecae582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55e3fec938fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55e3fec9af50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55e3fecacc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #38: <unknown function> + 0x211239 (0x55e3fed6f239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55e3fec9ba6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default0]:[rank8]: recv_activation_tensor = recv_activation() | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank11]: return forward_call(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default3]:[rank11]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default0]:[rank8]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default5]:[rank13]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f4857789978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:[rank13]: frame #12: <unknown function> + 0x5adc309 (0x7f488ff48309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default0]:[rank8]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default5]:[rank13]: frame #13: <unknown function> + 0x5ae6f10 (0x7f488ff52f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default6]:[rank14]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55e3fec973e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55e3feca2a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55e3fec92c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55e3feca2a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55e3fec938fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #45: <unknown function> + 0x150582 (0x55e3fecae582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #46: PyObject_Call + 0xbc (0x55e3fecaef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55e3fec952b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #48: <unknown function> + 0x150582 (0x55e3fecae582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #49: PyObject_Call + 0xbc (0x55e3fecaef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55e3fec952b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55e3feca2a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55e3fec9b007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55e3fecacc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #54: <unknown function> + 0x211239 (0x55e3fed6f239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #55: PyObject_Call + 0x207 (0x55e3fecaf067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55e3fec952b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #57: <unknown function> + 0x150582 (0x55e3fecae582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55e3fec938fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #59: <unknown function> + 0x150582 (0x55e3fecae582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #60: PyObject_Call + 0xbc (0x55e3fecaef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55e3fec952b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #62: <unknown function> + 0x150582 (0x55e3fecae582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #63: PyObject_Call + 0xbc (0x55e3fecaef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default3]:[rank11]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank11]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank11]: return forward_call(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default0]:[rank8]: dist.recv( | |
| [default5]:[rank13]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f488ff52fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #15: <unknown function> + 0x5124446 (0x7f488f590446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #16: <unknown function> + 0x1acf4b8 (0x7f488bf3b4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #17: <unknown function> + 0x5aee004 (0x7f488ff5a004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #18: <unknown function> + 0x5af36b5 (0x7f488ff5f6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #19: <unknown function> + 0xd2631e (0x7f48a2b4931e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default5]:[rank13]: frame #20: <unknown function> + 0x47def4 (0x7f48a22a0ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default0]:[rank8]: return func(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default3]:[rank11]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default0]:[rank8]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default5]:[rank13]: frame #21: <unknown function> + 0x1445a6 (0x5634edc2a5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #22: _PyObject_MakeTpCall + 0x26b (0x5634edc23a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default5]:[rank13]: frame #23: <unknown function> + 0x150866 (0x5634edc36866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5634edc1f142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #25: _PyFunction_Vectorcall + 0x6c (0x5634edc2aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #26: PyObject_Call + 0xbc (0x5634edc36f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5634edc1d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #28: _PyFunction_Vectorcall + 0x6c (0x5634edc2aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5634edc1b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #30: <unknown function> + 0x150582 (0x5634edc36582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5634edc1b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #32: <unknown function> + 0x150582 (0x5634edc36582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5634edc1b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #34: <unknown function> + 0x150582 (0x5634edc36582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5634edc1b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5634edc22f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #37: _PyObject_Call_Prepend + 0x69 (0x5634edc34c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #38: <unknown function> + 0x211239 (0x5634edcf7239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #39: _PyObject_MakeTpCall + 0x26b (0x5634edc23a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5634edc1f3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #41: _PyFunction_Vectorcall + 0x6c (0x5634edc2aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5634edc1ac5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #43: _PyFunction_Vectorcall + 0x6c (0x5634edc2aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5634edc1b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #45: <unknown function> + 0x150582 (0x5634edc36582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #46: PyObject_Call + 0xbc (0x5634edc36f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5634edc1d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #48: <unknown function> + 0x150582 (0x5634edc36582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #49: PyObject_Call + 0xbc (0x5634edc36f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5634edc1d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #51: _PyFunction_Vectorcall + 0x6c (0x5634edc2aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5634edc23007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #53: _PyObject_Call_Prepend + 0x69 (0x5634edc34c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #54: <unknown function> + 0x211239 (0x5634edcf7239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #55: PyObject_Call + 0x207 (0x5634edc37067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5634edc1d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #57: <unknown function> + 0x150582 (0x5634edc36582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5634edc1b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #59: <unknown function> + 0x150582 (0x5634edc36582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #60: PyObject_Call + 0xbc (0x5634edc36f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5634edc1d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #62: <unknown function> + 0x150582 (0x5634edc36582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #63: PyObject_Call + 0xbc (0x5634edc36f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default3]:[rank11]: pipeline_state.run_communication() | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default3]:[rank11]: recv_activation_tensor = recv_activation() | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default3]:[rank11]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default3]:[rank11]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default3]:[rank11]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default3]:[rank11]: dist.recv( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default3]:[rank11]: return func(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default3]:[rank11]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default3]:[rank11]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default3]:[rank11]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default3]:[rank11]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f70fc241897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default3]:[rank11]: frame #1: <unknown function> + 0x5b3a23e (0x7f7135d5e23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f7135d58c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f7135d58f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f7135d59fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7135d0e371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7135d0e371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7135d0e371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7135d0e371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f70fd51b189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:[rank11]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f70fd522610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:[rank11]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f70fd541978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:[rank11]: frame #12: <unknown function> + 0x5adc309 (0x7f7135d00309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #13: <unknown function> + 0x5ae6f10 (0x7f7135d0af10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f7135d0afa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #15: <unknown function> + 0x5124446 (0x7f7135348446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #16: <unknown function> + 0x1acf4b8 (0x7f7131cf34b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #17: <unknown function> + 0x5aee004 (0x7f7135d12004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #18: <unknown function> + 0x5af36b5 (0x7f7135d176b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #19: <unknown function> + 0xd2631e (0x7f714890131e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default3]:[rank11]: frame #20: <unknown function> + 0x47def4 (0x7f7148058ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default3]:[rank11]: frame #21: <unknown function> + 0x1445a6 (0x56263b70f5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #22: _PyObject_MakeTpCall + 0x26b (0x56263b708a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #23: <unknown function> + 0x150866 (0x56263b71b866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x56263b704142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #25: _PyFunction_Vectorcall + 0x6c (0x56263b70fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #26: PyObject_Call + 0xbc (0x56263b71bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x56263b7022b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #28: _PyFunction_Vectorcall + 0x6c (0x56263b70fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x56263b7008fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #30: <unknown function> + 0x150582 (0x56263b71b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x56263b7008fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #32: <unknown function> + 0x150582 (0x56263b71b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x56263b7008fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #34: <unknown function> + 0x150582 (0x56263b71b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x56263b7008fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x56263b707f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #37: _PyObject_Call_Prepend + 0x69 (0x56263b719c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #38: <unknown function> + 0x211239 (0x56263b7dc239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #39: _PyObject_MakeTpCall + 0x26b (0x56263b708a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x56263b7043e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default0]:[rank8]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f8e60840897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default3]:[rank11]: frame #41: _PyFunction_Vectorcall + 0x6c (0x56263b70fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x56263b6ffc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #43: _PyFunction_Vectorcall + 0x6c (0x56263b70fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x56263b7008fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #1: <unknown function> + 0x5b3a23e (0x7f8e9a35d23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #45: <unknown function> + 0x150582 (0x56263b71b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #46: PyObject_Call + 0xbc (0x56263b71bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x56263b7022b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #48: <unknown function> + 0x150582 (0x56263b71b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f8e9a357c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f8e9a357f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f8e9a358fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #49: PyObject_Call + 0xbc (0x56263b71bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x56263b7022b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #51: _PyFunction_Vectorcall + 0x6c (0x56263b70fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x56263b708007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #53: _PyObject_Call_Prepend + 0x69 (0x56263b719c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #54: <unknown function> + 0x211239 (0x56263b7dc239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8e9a30d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8e9a30d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8e9a30d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8e9a30d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f8e61b1a189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:[rank11]: frame #55: PyObject_Call + 0x207 (0x56263b71c067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x56263b7022b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #57: <unknown function> + 0x150582 (0x56263b71b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x56263b7008fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #59: <unknown function> + 0x150582 (0x56263b71b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #60: PyObject_Call + 0xbc (0x56263b71bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x56263b7022b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #62: <unknown function> + 0x150582 (0x56263b71b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #63: PyObject_Call + 0xbc (0x56263b71bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default0]:[rank8]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f8e61b21610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default0]:[rank8]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f8e61b40978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default0]:[rank8]: frame #12: <unknown function> + 0x5adc309 (0x7f8e9a2ff309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #13: <unknown function> + 0x5ae6f10 (0x7f8e9a309f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f8e9a309fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #15: <unknown function> + 0x5124446 (0x7f8e99947446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #16: <unknown function> + 0x1acf4b8 (0x7f8e962f24b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #17: <unknown function> + 0x5aee004 (0x7f8e9a311004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #18: <unknown function> + 0x5af36b5 (0x7f8e9a3166b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #19: <unknown function> + 0xd2631e (0x7f8eacf0031e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default0]:[rank8]: frame #20: <unknown function> + 0x47def4 (0x7f8eac657ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default0]:[rank8]: frame #21: <unknown function> + 0x1445a6 (0x560973fa55a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #22: _PyObject_MakeTpCall + 0x26b (0x560973f9ea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #23: <unknown function> + 0x150866 (0x560973fb1866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x560973f9a142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #25: _PyFunction_Vectorcall + 0x6c (0x560973fa5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #26: PyObject_Call + 0xbc (0x560973fb1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x560973f982b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #28: _PyFunction_Vectorcall + 0x6c (0x560973fa5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x560973f968fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #30: <unknown function> + 0x150582 (0x560973fb1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x560973f968fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #32: <unknown function> + 0x150582 (0x560973fb1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x560973f968fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #34: <unknown function> + 0x150582 (0x560973fb1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x560973f968fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x560973f9df50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #37: _PyObject_Call_Prepend + 0x69 (0x560973fafc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #38: <unknown function> + 0x211239 (0x560974072239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #39: _PyObject_MakeTpCall + 0x26b (0x560973f9ea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x560973f9a3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #41: _PyFunction_Vectorcall + 0x6c (0x560973fa5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x560973f95c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #43: _PyFunction_Vectorcall + 0x6c (0x560973fa5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x560973f968fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #45: <unknown function> + 0x150582 (0x560973fb1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #46: PyObject_Call + 0xbc (0x560973fb1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x560973f982b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #48: <unknown function> + 0x150582 (0x560973fb1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #49: PyObject_Call + 0xbc (0x560973fb1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x560973f982b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #51: _PyFunction_Vectorcall + 0x6c (0x560973fa5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x560973f9e007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #53: _PyObject_Call_Prepend + 0x69 (0x560973fafc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #54: <unknown function> + 0x211239 (0x560974072239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #55: PyObject_Call + 0x207 (0x560973fb2067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x560973f982b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #57: <unknown function> + 0x150582 (0x560973fb1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x560973f968fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #59: <unknown function> + 0x150582 (0x560973fb1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #60: PyObject_Call + 0xbc (0x560973fb1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x560973f982b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #62: <unknown function> + 0x150582 (0x560973fb1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #63: PyObject_Call + 0xbc (0x560973fb1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default7]:[rank15]: Traceback (most recent call last): | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default7]:[rank15]: trainer.train(dataloader) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default7]:[rank15]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default7]:[rank15]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default7]:[rank15]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default7]:[rank15]: output = model(**micro_batch) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank15]: return forward_call(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default7]:[rank15]: sharded_logits = self.model( | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank15]: return forward_call(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default7]:[rank15]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default7]:[rank15]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank15]: return forward_call(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default7]:[rank15]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default7]:[rank15]: pipeline_state.run_communication() | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default7]:[rank15]: recv_activation_tensor = recv_activation() | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default7]:[rank15]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default7]:[rank15]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default7]:[rank15]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default7]:[rank15]: dist.recv( | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default7]:[rank15]: return func(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default7]:[rank15]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default7]:[rank15]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default7]:[rank15]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default7]:[rank15]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f449a289897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default7]:[rank15]: frame #1: <unknown function> + 0x5b3a23e (0x7f44d3da623e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f44d3da0c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f44d3da0f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f44d3da1fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f44d3d56371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f44d3d56371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f44d3d56371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f44d3d56371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f449b563189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:[rank15]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f449b56a610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:[rank15]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f449b589978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:[rank15]: frame #12: <unknown function> + 0x5adc309 (0x7f44d3d48309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #13: <unknown function> + 0x5ae6f10 (0x7f44d3d52f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f44d3d52fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #15: <unknown function> + 0x5124446 (0x7f44d3390446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #16: <unknown function> + 0x1acf4b8 (0x7f44cfd3b4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #17: <unknown function> + 0x5aee004 (0x7f44d3d5a004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #18: <unknown function> + 0x5af36b5 (0x7f44d3d5f6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #19: <unknown function> + 0xd2631e (0x7f44e694931e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default7]:[rank15]: frame #20: <unknown function> + 0x47def4 (0x7f44e60a0ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default7]:[rank15]: frame #21: <unknown function> + 0x1445a6 (0x556113ead5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #22: _PyObject_MakeTpCall + 0x26b (0x556113ea6a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #23: <unknown function> + 0x150866 (0x556113eb9866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x556113ea2142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #25: _PyFunction_Vectorcall + 0x6c (0x556113eada2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #26: PyObject_Call + 0xbc (0x556113eb9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x556113ea02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #28: _PyFunction_Vectorcall + 0x6c (0x556113eada2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x556113e9e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #30: <unknown function> + 0x150582 (0x556113eb9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x556113e9e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #32: <unknown function> + 0x150582 (0x556113eb9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x556113e9e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #34: <unknown function> + 0x150582 (0x556113eb9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x556113e9e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x556113ea5f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #37: _PyObject_Call_Prepend + 0x69 (0x556113eb7c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #38: <unknown function> + 0x211239 (0x556113f7a239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #39: _PyObject_MakeTpCall + 0x26b (0x556113ea6a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x556113ea23e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #41: _PyFunction_Vectorcall + 0x6c (0x556113eada2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x556113e9dc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #43: _PyFunction_Vectorcall + 0x6c (0x556113eada2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x556113e9e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #45: <unknown function> + 0x150582 (0x556113eb9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #46: PyObject_Call + 0xbc (0x556113eb9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x556113ea02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #48: <unknown function> + 0x150582 (0x556113eb9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #49: PyObject_Call + 0xbc (0x556113eb9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x556113ea02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #51: _PyFunction_Vectorcall + 0x6c (0x556113eada2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x556113ea6007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #53: _PyObject_Call_Prepend + 0x69 (0x556113eb7c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #54: <unknown function> + 0x211239 (0x556113f7a239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #55: PyObject_Call + 0x207 (0x556113eba067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x556113ea02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #57: <unknown function> + 0x150582 (0x556113eb9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x556113e9e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #59: <unknown function> + 0x150582 (0x556113eb9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #60: PyObject_Call + 0xbc (0x556113eb9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x556113ea02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #62: <unknown function> + 0x150582 (0x556113eb9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #63: PyObject_Call + 0xbc (0x556113eb9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default2]:[rank10]: Traceback (most recent call last): | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default2]:[rank10]: trainer.train(dataloader) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default2]:[rank10]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default2]:[rank10]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default2]:[rank10]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default2]:[rank10]: output = model(**micro_batch) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank10]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank10]: return forward_call(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default2]:[rank10]: sharded_logits = self.model( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank10]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank10]: return forward_call(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default2]:[rank10]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default2]:[rank10]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank10]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank10]: return forward_call(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default2]:[rank10]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default2]:[rank10]: pipeline_state.run_communication() | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default2]:[rank10]: recv_activation_tensor = recv_activation() | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default2]:[rank10]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default2]:[rank10]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default2]:[rank10]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default2]:[rank10]: dist.recv( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default2]:[rank10]: return func(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default2]:[rank10]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default2]:[rank10]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default2]:[rank10]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default2]:[rank10]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f1d1bc06897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default2]:[rank10]: frame #1: <unknown function> + 0x5b3a23e (0x7f1d5572323e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f1d5571dc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f1d5571df82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f1d5571efd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1d556d3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1d556d3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1d556d3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1d556d3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f1d1cee0189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:[rank10]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f1d1cee7610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:[rank10]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f1d1cf06978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:[rank10]: frame #12: <unknown function> + 0x5adc309 (0x7f1d556c5309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #13: <unknown function> + 0x5ae6f10 (0x7f1d556cff10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f1d556cffa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #15: <unknown function> + 0x5124446 (0x7f1d54d0d446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #16: <unknown function> + 0x1acf4b8 (0x7f1d516b84b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #17: <unknown function> + 0x5aee004 (0x7f1d556d7004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #18: <unknown function> + 0x5af36b5 (0x7f1d556dc6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #19: <unknown function> + 0xd2631e (0x7f1d682c631e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default2]:[rank10]: frame #20: <unknown function> + 0x47def4 (0x7f1d67a1def4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default2]:[rank10]: frame #21: <unknown function> + 0x1445a6 (0x55d89157b5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55d891574a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #23: <unknown function> + 0x150866 (0x55d891587866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55d891570142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55d89157ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #26: PyObject_Call + 0xbc (0x55d891587f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55d89156e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55d89157ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55d89156c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #30: <unknown function> + 0x150582 (0x55d891587582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55d89156c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #32: <unknown function> + 0x150582 (0x55d891587582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55d89156c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #34: <unknown function> + 0x150582 (0x55d891587582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55d89156c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55d891573f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55d891585c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #38: <unknown function> + 0x211239 (0x55d891648239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55d891574a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55d8915703e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55d89157ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55d89156bc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55d89157ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55d89156c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #45: <unknown function> + 0x150582 (0x55d891587582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #46: PyObject_Call + 0xbc (0x55d891587f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55d89156e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #48: <unknown function> + 0x150582 (0x55d891587582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #49: PyObject_Call + 0xbc (0x55d891587f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55d89156e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55d89157ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55d891574007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55d891585c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #54: <unknown function> + 0x211239 (0x55d891648239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #55: PyObject_Call + 0x207 (0x55d891588067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55d89156e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #57: <unknown function> + 0x150582 (0x55d891587582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55d89156c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #59: <unknown function> + 0x150582 (0x55d891587582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #60: PyObject_Call + 0xbc (0x55d891587f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55d89156e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #62: <unknown function> + 0x150582 (0x55d891587582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #63: PyObject_Call + 0xbc (0x55d891587f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default4]:[rank12]: Traceback (most recent call last): | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default4]:[rank12]: trainer.train(dataloader) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default4]:[rank12]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default4]:[rank12]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default4]:[rank12]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default4]:[rank12]: output = model(**micro_batch) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank12]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank12]: return forward_call(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default4]:[rank12]: sharded_logits = self.model( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank12]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank12]: return forward_call(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default4]:[rank12]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default4]:[rank12]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank12]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank12]: return forward_call(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default4]:[rank12]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default4]:[rank12]: pipeline_state.run_communication() | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default4]:[rank12]: recv_activation_tensor = recv_activation() | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default4]:[rank12]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default4]:[rank12]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default4]:[rank12]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default4]:[rank12]: dist.recv( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default4]:[rank12]: return func(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default4]:[rank12]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default4]:[rank12]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default4]:[rank12]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default4]:[rank12]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f0b60dc6897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default4]:[rank12]: frame #1: <unknown function> + 0x5b3a23e (0x7f0b9a8e323e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f0b9a8ddc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f0b9a8ddf82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f0b9a8defd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0b9a893371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0b9a893371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0b9a893371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0b9a893371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f0b620a0189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:[rank12]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f0b620a7610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:[rank12]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f0b620c6978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:[rank12]: frame #12: <unknown function> + 0x5adc309 (0x7f0b9a885309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #13: <unknown function> + 0x5ae6f10 (0x7f0b9a88ff10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f0b9a88ffa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #15: <unknown function> + 0x5124446 (0x7f0b99ecd446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #16: <unknown function> + 0x1acf4b8 (0x7f0b968784b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #17: <unknown function> + 0x5aee004 (0x7f0b9a897004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #18: <unknown function> + 0x5af36b5 (0x7f0b9a89c6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #19: <unknown function> + 0xd2631e (0x7f0bad48631e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default4]:[rank12]: frame #20: <unknown function> + 0x47def4 (0x7f0bacbddef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default4]:[rank12]: frame #21: <unknown function> + 0x1445a6 (0x55c4597265a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55c45971fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #23: <unknown function> + 0x150866 (0x55c459732866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55c45971b142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55c459726a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #26: PyObject_Call + 0xbc (0x55c459732f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55c4597192b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55c459726a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55c4597178fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #30: <unknown function> + 0x150582 (0x55c459732582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55c4597178fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #32: <unknown function> + 0x150582 (0x55c459732582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55c4597178fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #34: <unknown function> + 0x150582 (0x55c459732582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55c4597178fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55c45971ef50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55c459730c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #38: <unknown function> + 0x211239 (0x55c4597f3239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55c45971fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55c45971b3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55c459726a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55c459716c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55c459726a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55c4597178fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #45: <unknown function> + 0x150582 (0x55c459732582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #46: PyObject_Call + 0xbc (0x55c459732f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55c4597192b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #48: <unknown function> + 0x150582 (0x55c459732582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #49: PyObject_Call + 0xbc (0x55c459732f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55c4597192b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55c459726a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55c45971f007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55c459730c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #54: <unknown function> + 0x211239 (0x55c4597f3239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #55: PyObject_Call + 0x207 (0x55c459733067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55c4597192b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #57: <unknown function> + 0x150582 (0x55c459732582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55c4597178fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #59: <unknown function> + 0x150582 (0x55c459732582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #60: PyObject_Call + 0xbc (0x55c459732f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55c4597192b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #62: <unknown function> + 0x150582 (0x55c459732582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #63: PyObject_Call + 0xbc (0x55c459732f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| W0702 14:12:47.462000 140655077304128 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 88701 closing signal SIGTERM | |
| W0702 14:12:47.463000 140655077304128 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 88702 closing signal SIGTERM | |
| W0702 14:12:47.463000 140655077304128 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 88703 closing signal SIGTERM | |
| W0702 14:12:47.463000 140655077304128 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 88704 closing signal SIGTERM | |
| E0702 14:12:49.883000 140655077304128 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 88700) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 | |
| Traceback (most recent call last): | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module> | |
| sys.exit(main()) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper | |
| return f(*args, **kwargs) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main | |
| run(args) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run | |
| elastic_launch( | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ | |
| return launch_agent(self._config, self._entrypoint, list(args)) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent | |
| raise ChildFailedError( | |
| torch.distributed.elastic.multiprocessing.errors.ChildFailedError: | |
| ============================================================ | |
| /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED | |
| ------------------------------------------------------------ | |
| Failures: | |
| [1]: | |
| time : 2024-07-02_14:12:47 | |
| host : ip-26-0-161-178.ec2.internal | |
| rank : 5 (local_rank: 5) | |
| exitcode : 1 (pid: 88705) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| [2]: | |
| time : 2024-07-02_14:12:47 | |
| host : ip-26-0-161-178.ec2.internal | |
| rank : 6 (local_rank: 6) | |
| exitcode : 1 (pid: 88706) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| [3]: | |
| time : 2024-07-02_14:12:47 | |
| host : ip-26-0-161-178.ec2.internal | |
| rank : 7 (local_rank: 7) | |
| exitcode : 1 (pid: 88707) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| ------------------------------------------------------------ | |
| Root Cause (first observed failure): | |
| [0]: | |
| time : 2024-07-02_14:12:47 | |
| host : ip-26-0-161-178.ec2.internal | |
| rank : 0 (local_rank: 0) | |
| exitcode : 1 (pid: 88700) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| ============================================================ | |
| srun: error: ip-26-0-161-178: task 0: Exited with exit code 1 | |
| W0702 14:12:52.430000 140608983451392 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-163-147.ec2.internal_48757_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError. | |
| W0702 14:12:52.466000 140614650271552 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 48843 closing signal SIGTERM | |
| W0702 14:12:52.466000 140614650271552 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 48844 closing signal SIGTERM | |
| W0702 14:12:52.466000 140614650271552 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 48845 closing signal SIGTERM | |
| W0702 14:12:52.466000 140614650271552 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 48848 closing signal SIGTERM | |
| E0702 14:12:53.680000 140614650271552 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 48842) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 | |
| W0702 14:12:53.685000 140614650271552 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-163-147.ec2.internal_48757_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError. | |
| W0702 14:12:53.711000 140614650271552 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-163-147.ec2.internal_48757_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError. | |
| W0702 14:12:53.731000 140614650271552 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-163-147.ec2.internal_48757_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError. | |
| Traceback (most recent call last): | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module> | |
| sys.exit(main()) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper | |
| return f(*args, **kwargs) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main | |
| run(args) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run | |
| elastic_launch( | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ | |
| return launch_agent(self._config, self._entrypoint, list(args)) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent | |
| raise ChildFailedError( | |
| torch.distributed.elastic.multiprocessing.errors.ChildFailedError: | |
| ============================================================ | |
| /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED | |
| ------------------------------------------------------------ | |
| Failures: | |
| [1]: | |
| time : 2024-07-02_14:12:52 | |
| host : ip-26-0-163-147.ec2.internal | |
| rank : 12 (local_rank: 4) | |
| exitcode : 1 (pid: 48846) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| [2]: | |
| time : 2024-07-02_14:12:52 | |
| host : ip-26-0-163-147.ec2.internal | |
| rank : 13 (local_rank: 5) | |
| exitcode : 1 (pid: 48847) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| [3]: | |
| time : 2024-07-02_14:12:52 | |
| host : ip-26-0-163-147.ec2.internal | |
| rank : 15 (local_rank: 7) | |
| exitcode : 1 (pid: 48849) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| ------------------------------------------------------------ | |
| Root Cause (first observed failure): | |
| [0]: | |
| time : 2024-07-02_14:12:52 | |
| host : ip-26-0-163-147.ec2.internal | |
| rank : 8 (local_rank: 0) | |
| exitcode : 1 (pid: 48842) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| ============================================================ | |
| srun: error: ip-26-0-163-147: task 1: Exited with exit code 1 | |
| Consider using `hf_transfer` for faster uploads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details. | |