| ======================== | |
| START TIME: Wed Jul 3 23:03:28 UTC 2024 | |
| python3 version = Python 3.10.14 | |
| ======================== | |
| The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well. | |
| Token is valid (permission: write). | |
| Your token has been saved to /admin/home/ferdinand_mom/.cache/huggingface/token | |
| Login successful | |
| Already on 'bench_cluster' | |
| M examples/config_tiny_llama.py | |
| M examples/config_tiny_llama.yaml | |
| M examples/train_tiny_llama.sh | |
| M src/nanotron/models/llama.py | |
| M src/nanotron/trainer.py | |
| Your branch is up to date with 'origin/bench_cluster'. | |
| Job status: RUNNING | |
| W0703 23:03:31.480000 139769530853184 torch/distributed/run.py:757] | |
| W0703 23:03:31.480000 139769530853184 torch/distributed/run.py:757] ***************************************** | |
| W0703 23:03:31.480000 139769530853184 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. | |
| W0703 23:03:31.480000 139769530853184 torch/distributed/run.py:757] ***************************************** | |
| [default0]:07/03/2024 23:03:47 [WARNING|DP=0|PP=0|TP=0|ip-26-0-164-187]: [Vocab Size Padding] Padded vocab (size: 50257) with 1 dummy tokens (new size: 50258) | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Config: | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Config(general=GeneralArgs(project='bench_cluster', | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: run='%date_%jobid', | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: seed=42, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: step=None, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: consumed_train_samples=None, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: benchmark_csv_path=None, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: ignore_sanity_checks=True), | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: parallelism=ParallelismArgs(dp=1, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: pp=4, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: tp=2, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: pp_engine=<nanotron.parallel.pipeline_parallel.engine.OneForwardOneBackwardPipelineEngine object at 0x7f71446c4670>, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: tp_mode=<TensorParallelLinearMode.REDUCE_SCATTER: 2>, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: tp_linear_async_communication=False, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: expert_parallel_size=1), | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: model=ModelArgs(model_config=LlamaConfig(bos_token_id=1, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: eos_token_id=2, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: hidden_act='silu', | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: hidden_size=2048, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: initializer_range=0.02, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: intermediate_size=4096, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: is_llama_config=True, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: max_position_embeddings=4096, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: num_attention_heads=32, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: num_hidden_layers=24, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: num_key_value_heads=32, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: pad_token_id=None, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: pretraining_tp=1, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: rms_norm_eps=1e-05, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: rope_scaling=None, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: rope_theta=10000.0, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: tie_word_embeddings=True, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: use_cache=True, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: vocab_size=50258), | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: init_method=RandomInit(std=0.025), | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: dtype=torch.bfloat16, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: make_vocab_size_divisible_by=1, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: ddp_bucket_cap_mb=25), | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: tokenizer=TokenizerArgs(tokenizer_name_or_path='openai-community/gpt2', | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: tokenizer_revision=None, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: tokenizer_max_length=None), | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: checkpoints=CheckpointsArgs(checkpoints_path=Path('/dev/null'), | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: checkpoint_interval=100000, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: save_initial_state=False, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: resume_checkpoint_path=None, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: checkpoints_path_is_shared_file_system=False), | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: logging=LoggingArgs(log_level='info', | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: log_level_replica='info', | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: iteration_step_info_interval=1), | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: tokens=TokensArgs(sequence_length=4096, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: train_steps=20, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: micro_batch_size=4, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: batch_accumulation_per_replica=256, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: val_check_interval=-1, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: limit_val_batches=0, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: limit_test_batches=0), | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: optimizer=OptimizerArgs(optimizer_factory=AdamWOptimizerArgs(adam_eps=1e-08, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: adam_beta1=0.9, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: adam_beta2=0.95, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: torch_adam_is_fused=True, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: name='adamW'), | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: zero_stage=1, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: weight_decay=0.01, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: clip_grad=1.0, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: accumulate_grad_in_fp32=True, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: learning_rate_scheduler=LRSchedulerArgs(learning_rate=0.0001, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: lr_warmup_steps=1, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: lr_warmup_style='linear', | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: lr_decay_style='linear', | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: lr_decay_steps=19, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: lr_decay_starting_step=None, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: min_decay_lr=1e-05)), | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: data_stages=[DatasetStageArgs(name='Training Stage', | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: start_training_step=1, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: data=DataArgs(dataset=PretrainDatasetsArgs(hf_dataset_or_datasets='roneneldan/TinyStories', | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: hf_dataset_splits='train', | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: hf_dataset_config_name=None, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: dataset_processing_num_proc_per_process=64, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: dataset_overwrite_cache=False, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: text_column_name='text'), | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: seed=42, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: num_loading_workers=0))], | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: profiler=ProfilerArgs(profiler_export_path=Path('/fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/8_GPUS/dp-1_tp-2_pp-4_mbz-4')), | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: lighteval=None) | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Model Config: | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: LlamaConfig(bos_token_id=1, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: eos_token_id=2, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: hidden_act='silu', | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: hidden_size=2048, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: initializer_range=0.02, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: intermediate_size=4096, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: is_llama_config=True, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: max_position_embeddings=4096, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: num_attention_heads=32, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: num_hidden_layers=24, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: num_key_value_heads=32, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: pad_token_id=None, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: pretraining_tp=1, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: rms_norm_eps=1e-05, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: rope_scaling=None, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: rope_theta=10000.0, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: tie_word_embeddings=True, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: use_cache=True, | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: vocab_size=50258) | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Building model.. | |
| [default0]:07/03/2024 23:03:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Setting PP block ranks... | |
| [default1]:07/03/2024 23:04:01 [INFO|DP=0|PP=0|TP=1|ip-26-0-164-187]: Local number of parameters: 198M (378.21MiB) | |
| [default1]:07/03/2024 23:04:01 [INFO|DP=0|PP=0|TP=1|ip-26-0-164-187]: [After model building] Memory usage: 385.23MiB. Peak allocated: 387.26MiB Peak reserved: 402.00MiB | |
| [default6]:07/03/2024 23:04:01 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: Local number of parameters: 135M (258.20MiB) | |
| [default6]:07/03/2024 23:04:01 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: [After model building] Memory usage: 262.21MiB. Peak allocated: 264.24MiB Peak reserved: 280.00MiB | |
| [default6]:07/03/2024 23:04:01 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: No checkpoint path provided. | |
| [default4]:07/03/2024 23:04:01 [INFO|DP=0|PP=2|TP=0|ip-26-0-164-187]: Local number of parameters: 126M (240.05MiB) | |
| [default4]:07/03/2024 23:04:01 [INFO|DP=0|PP=2|TP=0|ip-26-0-164-187]: [After model building] Memory usage: 246.06MiB. Peak allocated: 248.09MiB Peak reserved: 262.00MiB | |
| [default1]:07/03/2024 23:04:01 [INFO|DP=0|PP=0|TP=1|ip-26-0-164-187]: No checkpoint path provided. | |
| [default4]:07/03/2024 23:04:01 [INFO|DP=0|PP=2|TP=0|ip-26-0-164-187]: No checkpoint path provided. | |
| [default0]:07/03/2024 23:04:01 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Total number of parameters: 1.21G (2313.02MiB) | |
| [default0]:07/03/2024 23:04:01 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Local number of parameters: 198M (378.21MiB) | |
| [default0]:07/03/2024 23:04:01 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: [After model building] Memory usage: 385.23MiB. Peak allocated: 387.26MiB Peak reserved: 402.00MiB | |
| [default0]:07/03/2024 23:04:01 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: No checkpoint path provided. | |
| [default0]:07/03/2024 23:04:01 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Parametrizing model parameters using StandardParametrizator | |
| [default3]:07/03/2024 23:04:01 [INFO|DP=0|PP=1|TP=1|ip-26-0-164-187]: Local number of parameters: 147M (280.05MiB) | |
| [default3]:07/03/2024 23:04:01 [INFO|DP=0|PP=1|TP=1|ip-26-0-164-187]: [After model building] Memory usage: 287.07MiB. Peak allocated: 289.10MiB Peak reserved: 302.00MiB | |
| [default3]:07/03/2024 23:04:01 [INFO|DP=0|PP=1|TP=1|ip-26-0-164-187]: No checkpoint path provided. | |
| [default2]:07/03/2024 23:04:01 [INFO|DP=0|PP=1|TP=0|ip-26-0-164-187]: Local number of parameters: 147M (280.05MiB) | |
| [default2]:07/03/2024 23:04:01 [INFO|DP=0|PP=1|TP=0|ip-26-0-164-187]: [After model building] Memory usage: 287.07MiB. Peak allocated: 289.10MiB Peak reserved: 302.00MiB | |
| [default7]:07/03/2024 23:04:01 [INFO|DP=0|PP=3|TP=1|ip-26-0-164-187]: Local number of parameters: 135M (258.20MiB) | |
| [default7]:07/03/2024 23:04:01 [INFO|DP=0|PP=3|TP=1|ip-26-0-164-187]: [After model building] Memory usage: 262.21MiB. Peak allocated: 264.24MiB Peak reserved: 280.00MiB | |
| [default7]:07/03/2024 23:04:01 [INFO|DP=0|PP=3|TP=1|ip-26-0-164-187]: No checkpoint path provided. | |
| [default5]:07/03/2024 23:04:01 [INFO|DP=0|PP=2|TP=1|ip-26-0-164-187]: Local number of parameters: 126M (240.05MiB) | |
| [default5]:07/03/2024 23:04:01 [INFO|DP=0|PP=2|TP=1|ip-26-0-164-187]: [After model building] Memory usage: 246.06MiB. Peak allocated: 248.09MiB Peak reserved: 262.00MiB | |
| [default5]:07/03/2024 23:04:01 [INFO|DP=0|PP=2|TP=1|ip-26-0-164-187]: No checkpoint path provided. | |
| [default2]:07/03/2024 23:04:01 [INFO|DP=0|PP=1|TP=0|ip-26-0-164-187]: No checkpoint path provided. | |
| [default0]:07/03/2024 23:04:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: [Optimizer Building] Using LearningRateForSP as learning rate | |
| [default0]:07/03/2024 23:04:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: [ZeRO sharding] Size of optimizer params per rank: | |
| [default0]:07/03/2024 23:04:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: [ZeRO sharding] DP Rank 0 has 198M out of 198M (100.00%) params' optimizer states | |
| [default0]:07/03/2024 23:04:04 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: [Training Plan] Stage Training Stage has 19 remaining training steps and has consumed 0 samples | |
| [default0]:07/03/2024 23:04:04 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Using `datasets` library | |
| [default0]:07/03/2024 23:04:04 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Loading tokenizer from openai-community/gpt2 and transformers/hf_hub versions ('4.41.2', '0.23.4') | |
| [default0]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:07/03/2024 23:04:04 [WARNING|DP=0|PP=0|TP=0|ip-26-0-164-187]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:07/03/2024 23:04:05 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: [Training Plan] There are 1 training stages | |
| [default0]:07/03/2024 23:04:05 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: [Stage Training Stage] start from step 1 | |
| [default0]:07/03/2024 23:04:05 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: | |
| [default0]:07/03/2024 23:04:05 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: [Start training] datetime: 2024-07-03 23:04:05.183207 | mbs: 4 | grad_accum: 256 | global_batch_size: 1024 | sequence_length: 4096 | train_steps: 20 | start_iteration_step: 0 | consumed_train_samples: 0 | |
| [default0]:07/03/2024 23:04:05 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Resuming training from stage Training Stage, it has trained for 0 samples and has 19 remaining train steps | |
| [default0]:07/03/2024 23:04:05 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 1898.09MiB. Peak allocated 1898.09MiB. Peak reserved: 1918.00MiB | |
| [default6]:07/03/2024 23:04:05 [WARNING|DP=0|PP=3|TP=0|ip-26-0-164-187]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:07/03/2024 23:04:05 [WARNING|DP=0|PP=0|TP=1|ip-26-0-164-187]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:07/03/2024 23:04:05 [WARNING|DP=0|PP=2|TP=1|ip-26-0-164-187]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:07/03/2024 23:04:05 [WARNING|DP=0|PP=1|TP=1|ip-26-0-164-187]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:07/03/2024 23:04:05 [WARNING|DP=0|PP=3|TP=1|ip-26-0-164-187]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:07/03/2024 23:04:05 [WARNING|DP=0|PP=1|TP=0|ip-26-0-164-187]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:07/03/2024 23:04:05 [WARNING|DP=0|PP=2|TP=0|ip-26-0-164-187]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default7]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default6]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default6]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default4]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default4]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default5]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default5]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default3]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default3]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default2]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default2]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default0]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default0]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default1]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default1]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default3]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default3]: warnings.warn( | |
| [default2]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default2]: warnings.warn( | |
| [default5]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default5]: warnings.warn( | |
| [default4]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default4]: warnings.warn( | |
| [default6]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default6]: warnings.warn( | |
| [default7]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default7]: warnings.warn( | |
| [default0]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default0]: warnings.warn( | |
| [default1]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default1]: warnings.warn( | |
| [default0]:07/03/2024 23:04:49 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 1965.73MiB. Peak allocated 20980.10MiB. Peak reserved: 21402.00MiB | |
| [default6]:07/03/2024 23:04:56 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 1 / 20 | consumed_tokens: 4.19M | elapsed_time_per_iteration_ms: 45.6K | tokens_per_sec: 92K | tokens_per_sec_per_gpu: 11.5K | global_batch_size: 1.02K | lm_loss: 11.2 | lr: 0.0001 | model_tflops_per_gpu: 104 | hardware_tflops_per_gpu: 104 | grad_norm: 14.8 | cuda_memory_allocated: 2.51G | cuda_max_memory_reserved: 8.58G | hd_total_memory_tb: 312G | hd_used_memory_tb: 68.3G | hd_free_memory_tb: 244G | |
| [default0]:07/03/2024 23:04:56 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 3478.61MiB. Peak reserved: 21600.00MiB | |
| [default6]:07/03/2024 23:05:20 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 2 / 20 | consumed_tokens: 8.39M | elapsed_time_per_iteration_ms: 23.6K | tokens_per_sec: 178K | tokens_per_sec_per_gpu: 22.2K | global_batch_size: 1.02K | lm_loss: 11.2 | lr: 9.53e-05 | model_tflops_per_gpu: 202 | hardware_tflops_per_gpu: 202 | grad_norm: 14.9 | cuda_memory_allocated: 2.51G | cuda_max_memory_reserved: 10.2G | hd_total_memory_tb: 312G | hd_used_memory_tb: 68.3G | hd_free_memory_tb: 244G | |
| [default0]:07/03/2024 23:05:20 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 22492.98MiB. Peak reserved: 22876.00MiB | |
| [default0]:07/03/2024 23:05:20 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 3478.63MiB. Peak reserved: 22876.00MiB | |
| [default6]:07/03/2024 23:05:43 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 3 / 20 | consumed_tokens: 12.6M | elapsed_time_per_iteration_ms: 23.7K | tokens_per_sec: 177K | tokens_per_sec_per_gpu: 22.1K | global_batch_size: 1.02K | lm_loss: 9.53 | lr: 9.05e-05 | model_tflops_per_gpu: 201 | hardware_tflops_per_gpu: 201 | grad_norm: 35.7 | cuda_memory_allocated: 2.51G | cuda_max_memory_reserved: 10.2G | hd_total_memory_tb: 312G | hd_used_memory_tb: 68.3G | hd_free_memory_tb: 244G | |
| [default0]:STAGE:2024-07-03 23:05:43 27331:27331 ActivityProfilerController.cpp:314] Completed Stage: Warm Up | |
| [default0]:07/03/2024 23:05:43 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 22492.98MiB. Peak reserved: 23018.00MiB | |
| [default0]:07/03/2024 23:05:43 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 3478.63MiB. Peak reserved: 23018.00MiB | |
| [default6]:07/03/2024 23:06:07 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 4 / 20 | consumed_tokens: 16.8M | elapsed_time_per_iteration_ms: 23.6K | tokens_per_sec: 178K | tokens_per_sec_per_gpu: 22.2K | global_batch_size: 1.02K | lm_loss: 12.3 | lr: 8.58e-05 | model_tflops_per_gpu: 201 | hardware_tflops_per_gpu: 201 | grad_norm: 37.4 | cuda_memory_allocated: 2.51G | cuda_max_memory_reserved: 10.2G | hd_total_memory_tb: 312G | hd_used_memory_tb: 68.3G | hd_free_memory_tb: 244G | |
| [default0]:07/03/2024 23:06:07 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 22492.98MiB. Peak reserved: 23018.00MiB | |
| [default0]:07/03/2024 23:06:07 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 3478.63MiB. Peak reserved: 23018.00MiB | |
| [default6]:07/03/2024 23:06:31 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 5 / 20 | consumed_tokens: 21M | elapsed_time_per_iteration_ms: 23.6K | tokens_per_sec: 178K | tokens_per_sec_per_gpu: 22.2K | global_batch_size: 1.02K | lm_loss: 9.94 | lr: 8.11e-05 | model_tflops_per_gpu: 201 | hardware_tflops_per_gpu: 201 | grad_norm: 14.1 | |
| [default0]:07/03/2024 23:06:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 22492.98MiB. Peak reserved: 23018.00MiB | |
| [default6]:07/03/2024 23:06:54 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 6 / 20 | consumed_tokens: 25.2M | elapsed_time_per_iteration_ms: 23.6K | tokens_per_sec: 178K | tokens_per_sec_per_gpu: 22.2K | global_batch_size: 1.02K | lm_loss: 9.43 | lr: 7.63e-05 | model_tflops_per_gpu: 202 | hardware_tflops_per_gpu: 202 | grad_norm: 8.15 | |
| [default0]:STAGE:2024-07-03 23:07:37 27331:27331 ActivityProfilerController.cpp:320] Completed Stage: Collection | |
| [default0]:STAGE:2024-07-03 23:07:42 27331:27331 ActivityProfilerController.cpp:324] Completed Stage: Post Processing | |
| [default0]:07/03/2024 23:13:13 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 22492.98MiB. Peak reserved: 23018.00MiB | |
| [default0]:07/03/2024 23:13:36 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 22492.98MiB. Peak reserved: 23018.00MiB | |
| [default6]:07/03/2024 23:13:36 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 7 / 20 | consumed_tokens: 29.4M | elapsed_time_per_iteration_ms: 402K | tokens_per_sec: 10.4K | tokens_per_sec_per_gpu: 1.3K | global_batch_size: 1.02K | lm_loss: 8.73 | lr: 7.16e-05 | model_tflops_per_gpu: 11.8 | hardware_tflops_per_gpu: 11.8 | grad_norm: 6.04 | |
| [default0]:07/03/2024 23:14:00 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 22492.98MiB. Peak reserved: 23018.00MiB | |
| [default6]:07/03/2024 23:14:00 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 8 / 20 | consumed_tokens: 33.6M | elapsed_time_per_iteration_ms: 23.6K | tokens_per_sec: 178K | tokens_per_sec_per_gpu: 22.3K | global_batch_size: 1.02K | lm_loss: 9.16 | lr: 6.68e-05 | model_tflops_per_gpu: 202 | hardware_tflops_per_gpu: 202 | grad_norm: 27.9 | |
| [default6]:07/03/2024 23:14:23 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 9 / 20 | consumed_tokens: 37.7M | elapsed_time_per_iteration_ms: 23.6K | tokens_per_sec: 178K | tokens_per_sec_per_gpu: 22.3K | global_batch_size: 1.02K | lm_loss: 8.33 | lr: 6.21e-05 | model_tflops_per_gpu: 202 | hardware_tflops_per_gpu: 202 | grad_norm: 9.35 | |
| [default0]:07/03/2024 23:14:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 22492.98MiB. Peak reserved: 23018.00MiB | |
| [default0]:07/03/2024 23:14:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 22492.98MiB. Peak reserved: 23018.00MiB | |
| [default6]:07/03/2024 23:14:47 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 10 / 20 | consumed_tokens: 41.9M | elapsed_time_per_iteration_ms: 23.5K | tokens_per_sec: 178K | tokens_per_sec_per_gpu: 22.3K | global_batch_size: 1.02K | lm_loss: 8.02 | lr: 5.74e-05 | model_tflops_per_gpu: 202 | hardware_tflops_per_gpu: 202 | grad_norm: 5.24 | |
| [default0]:07/03/2024 23:15:10 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 22492.98MiB. Peak reserved: 23018.00MiB | |
| [default6]:07/03/2024 23:15:10 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 11 / 20 | consumed_tokens: 46.1M | elapsed_time_per_iteration_ms: 23.6K | tokens_per_sec: 178K | tokens_per_sec_per_gpu: 22.3K | global_batch_size: 1.02K | lm_loss: 7.85 | lr: 5.26e-05 | model_tflops_per_gpu: 202 | hardware_tflops_per_gpu: 202 | grad_norm: 4.81 | |
| [default0]:07/03/2024 23:15:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 22492.98MiB. Peak reserved: 23018.00MiB | |
| [default6]:07/03/2024 23:15:34 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 12 / 20 | consumed_tokens: 50.3M | elapsed_time_per_iteration_ms: 23.6K | tokens_per_sec: 178K | tokens_per_sec_per_gpu: 22.3K | global_batch_size: 1.02K | lm_loss: 7.68 | lr: 4.79e-05 | model_tflops_per_gpu: 202 | hardware_tflops_per_gpu: 202 | grad_norm: 4.49 | |
| [default0]:07/03/2024 23:15:58 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 22492.98MiB. Peak reserved: 23018.00MiB | |
| [default6]:07/03/2024 23:15:58 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 13 / 20 | consumed_tokens: 54.5M | elapsed_time_per_iteration_ms: 23.6K | tokens_per_sec: 178K | tokens_per_sec_per_gpu: 22.2K | global_batch_size: 1.02K | lm_loss: 7.53 | lr: 4.32e-05 | model_tflops_per_gpu: 202 | hardware_tflops_per_gpu: 202 | grad_norm: 4.16 | |
| [default6]:07/03/2024 23:16:21 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 14 / 20 | consumed_tokens: 58.7M | elapsed_time_per_iteration_ms: 23.6K | tokens_per_sec: 178K | tokens_per_sec_per_gpu: 22.2K | global_batch_size: 1.02K | lm_loss: 7.4 | lr: 3.84e-05 | model_tflops_per_gpu: 202 | hardware_tflops_per_gpu: 202 | grad_norm: 4.08 | |
| [default0]:07/03/2024 23:16:21 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 22492.98MiB. Peak reserved: 23020.00MiB | |
| [default6]:07/03/2024 23:16:45 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 15 / 20 | consumed_tokens: 62.9M | elapsed_time_per_iteration_ms: 23.6K | tokens_per_sec: 178K | tokens_per_sec_per_gpu: 22.3K | global_batch_size: 1.02K | lm_loss: 7.26 | lr: 3.37e-05 | model_tflops_per_gpu: 202 | hardware_tflops_per_gpu: 202 | grad_norm: 3.25 | |
| [default0]:07/03/2024 23:16:45 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 22492.98MiB. Peak reserved: 23020.00MiB | |
| [default6]:07/03/2024 23:17:08 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 16 / 20 | consumed_tokens: 67.1M | elapsed_time_per_iteration_ms: 23.6K | tokens_per_sec: 178K | tokens_per_sec_per_gpu: 22.2K | global_batch_size: 1.02K | lm_loss: 7.17 | lr: 2.89e-05 | model_tflops_per_gpu: 202 | hardware_tflops_per_gpu: 202 | grad_norm: 2.43 | |
| [default0]:07/03/2024 23:17:08 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 22492.98MiB. Peak reserved: 23020.00MiB | |
| [default6]:07/03/2024 23:17:32 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 17 / 20 | consumed_tokens: 71.3M | elapsed_time_per_iteration_ms: 23.8K | tokens_per_sec: 176K | tokens_per_sec_per_gpu: 22K | global_batch_size: 1.02K | lm_loss: 7.1 | lr: 2.42e-05 | model_tflops_per_gpu: 200 | hardware_tflops_per_gpu: 200 | grad_norm: 2.88 | |
| [default0]:07/03/2024 23:17:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 22492.98MiB. Peak reserved: 23020.00MiB | |
| [default0]:07/03/2024 23:17:56 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 22492.98MiB. Peak reserved: 23020.00MiB | |
| [default6]:07/03/2024 23:17:56 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 18 / 20 | consumed_tokens: 75.5M | elapsed_time_per_iteration_ms: 23.6K | tokens_per_sec: 178K | tokens_per_sec_per_gpu: 22.2K | global_batch_size: 1.02K | lm_loss: 7.03 | lr: 1.95e-05 | model_tflops_per_gpu: 202 | hardware_tflops_per_gpu: 202 | grad_norm: 2.75 | |
| [default6]:07/03/2024 23:18:19 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 19 / 20 | consumed_tokens: 79.7M | elapsed_time_per_iteration_ms: 23.6K | tokens_per_sec: 178K | tokens_per_sec_per_gpu: 22.2K | global_batch_size: 1.02K | lm_loss: 6.96 | lr: 1.47e-05 | model_tflops_per_gpu: 202 | hardware_tflops_per_gpu: 202 | grad_norm: 2.64 | |
| [default0]:07/03/2024 23:18:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-164-187]: Memory usage: 3478.61MiB. Peak allocated 22492.98MiB. Peak reserved: 23020.00MiB | |
| [default6]:07/03/2024 23:18:43 [INFO|DP=0|PP=3|TP=0|ip-26-0-164-187]: iteration: 20 / 20 | consumed_tokens: 83.9M | elapsed_time_per_iteration_ms: 23.6K | tokens_per_sec: 178K | tokens_per_sec_per_gpu: 22.2K | global_batch_size: 1.02K | lm_loss: 6.91 | lr: 1e-05 | model_tflops_per_gpu: 202 | hardware_tflops_per_gpu: 202 | grad_norm: 2.47 | |
| Saved 1 csv files over 1 completed logs | |
| Processing file: /fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/8_GPUS/dp-1_tp-2_pp-4_mbz-4/profiler/ip-26-0-164-187_27331.1720048318217251415.pt.trace.json | |
| Results written to /fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/8_GPUS/dp-1_tp-2_pp-4_mbz-4/profiler.csv | |
| Consider using `hf_transfer` for faster uploads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details. | |
| ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 0%| | 0.00/10.7G [00:00<?, ?B/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 0%| | 8.21M/10.7G [00:00<02:13, 79.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 0%| | 16.2M/10.7G [00:05<1:07:16, 2.64MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 0%| | 32.0M/10.7G [00:05<27:16, 6.49MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 0%| | 48.0M/10.7G [00:05<15:38, 11.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 1%| | 64.0M/10.7G [00:06<10:25, 16.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 1%| | 80.0M/10.7G [00:06<07:42, 22.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 1%| | 96.0M/10.7G [00:06<06:09, 28.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 1%| | 112M/10.7G [00:06<04:48, 36.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 1%| | 128M/10.7G [00:07<04:06, 42.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 1%|▏ | 144M/10.7G [00:07<03:27, 50.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 2%|▏ | 160M/10.7G [00:07<03:31, 49.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 2%|▏ | 176M/10.7G [00:07<03:06, 56.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 2%|▏ | 192M/10.7G [00:08<03:40, 47.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 2%|▏ | 208M/10.7G [00:08<03:17, 53.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 2%|▏ | 224M/10.7G [00:08<03:03, 57.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 2%|▏ | 240M/10.7G [00:08<02:51, 60.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 2%|▏ | 256M/10.7G [00:09<02:52, 60.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 3%|▎ | 272M/10.7G [00:09<02:55, 59.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 3%|▎ | 288M/10.7G [00:09<02:51, 60.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 3%|▎ | 304M/10.7G [00:09<02:42, 63.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 3%|▎ | 320M/10.7G [00:10<02:56, 58.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 3%|▎ | 336M/10.7G [00:10<02:49, 60.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 3%|▎ | 352M/10.7G [00:10<02:51, 60.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 3%|▎ | 368M/10.7G [00:10<02:48, 60.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 4%|▎ | 384M/10.7G [00:11<03:07, 54.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 4%|▍ | 400M/10.7G [00:11<03:01, 56.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 4%|▍ | 416M/10.7G [00:12<03:26, 49.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 4%|▍ | 432M/10.7G [00:12<03:16, 52.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 4%|▍ | 448M/10.7G [00:12<02:56, 57.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 4%|▍ | 464M/10.7G [00:12<03:14, 52.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 5%|▍ | 480M/10.7G [00:13<02:51, 59.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 5%|▍ | 496M/10.7G [00:13<02:41, 62.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 5%|▍ | 512M/10.7G [00:13<02:51, 59.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 5%|▍ | 528M/10.7G [00:13<02:44, 61.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 5%|▌ | 544M/10.7G [00:14<03:50, 43.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 5%|▌ | 560M/10.7G [00:14<03:26, 48.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 5%|▌ | 576M/10.7G [00:14<03:05, 54.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 6%|▌ | 592M/10.7G [00:15<02:54, 57.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 6%|▌ | 608M/10.7G [00:15<02:53, 57.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 6%|▌ | 624M/10.7G [00:15<02:39, 62.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 6%|▌ | 640M/10.7G [00:15<02:39, 62.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 6%|▌ | 649M/10.7G [00:15<02:30, 66.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 6%|▌ | 656M/10.7G [00:16<03:46, 44.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 6%|▋ | 672M/10.7G [00:16<03:09, 52.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 6%|▋ | 688M/10.7G [00:16<02:49, 58.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 7%|▋ | 704M/10.7G [00:17<02:44, 60.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 7%|▋ | 720M/10.7G [00:17<02:42, 61.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 7%|▋ | 736M/10.7G [00:17<02:31, 65.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 7%|▋ | 752M/10.7G [00:17<02:30, 65.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 7%|▋ | 768M/10.7G [00:18<02:48, 58.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 7%|▋ | 784M/10.7G [00:18<02:17, 71.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 7%|▋ | 793M/10.7G [00:18<02:52, 57.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 8%|▊ | 800M/10.7G [00:18<03:59, 41.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 8%|▊ | 816M/10.7G [00:19<03:13, 50.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 8%|▊ | 832M/10.7G [00:19<02:46, 59.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 8%|▊ | 848M/10.7G [00:19<02:32, 64.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 8%|▊ | 864M/10.7G [00:19<02:41, 60.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 8%|▊ | 880M/10.7G [00:19<02:37, 62.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 8%|▊ | 896M/10.7G [00:20<02:33, 63.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 9%|▊ | 912M/10.7G [00:25<18:04, 8.98MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 9%|▊ | 928M/10.7G [00:25<13:22, 12.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 9%|▉ | 944M/10.7G [00:26<10:38, 15.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 9%|▉ | 960M/10.7G [00:26<08:08, 19.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 9%|▉ | 976M/10.7G [00:26<06:46, 23.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 9%|▉ | 992M/10.7G [00:27<05:33, 29.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 9%|▉ | 1.01G/10.7G [00:27<04:31, 35.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 10%|▉ | 1.02G/10.7G [00:27<03:55, 41.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 10%|▉ | 1.04G/10.7G [00:27<03:38, 44.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 10%|▉ | 1.06G/10.7G [00:28<03:21, 47.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 10%|█ | 1.07G/10.7G [00:28<03:24, 46.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 10%|█ | 1.09G/10.7G [00:28<03:05, 51.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 10%|█ | 1.10G/10.7G [00:28<02:58, 53.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 11%|█ | 1.12G/10.7G [00:29<02:45, 57.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 11%|█ | 1.14G/10.7G [00:29<02:38, 60.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 11%|█ | 1.15G/10.7G [00:29<02:22, 66.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 11%|█ | 1.17G/10.7G [00:29<02:29, 63.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 11%|█ | 1.18G/10.7G [00:30<02:34, 61.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 11%|█▏ | 1.20G/10.7G [00:30<02:19, 67.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 11%|█▏ | 1.22G/10.7G [00:30<02:18, 68.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 12%|█▏ | 1.23G/10.7G [00:30<02:22, 66.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 12%|█▏ | 1.25G/10.7G [00:31<02:35, 60.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 12%|█▏ | 1.26G/10.7G [00:31<02:40, 58.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 12%|█▏ | 1.28G/10.7G [00:31<02:49, 55.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 12%|█▏ | 1.30G/10.7G [00:32<03:15, 47.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 12%|█▏ | 1.31G/10.7G [00:32<02:58, 52.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 12%|█▏ | 1.33G/10.7G [00:32<02:49, 55.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 13%|█▎ | 1.34G/10.7G [00:32<02:36, 59.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 13%|█▎ | 1.36G/10.7G [00:33<02:36, 59.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 13%|█▎ | 1.38G/10.7G [00:33<02:18, 67.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 13%|█▎ | 1.39G/10.7G [00:33<02:18, 67.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 13%|█▎ | 1.41G/10.7G [00:33<02:03, 74.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 13%|█▎ | 1.42G/10.7G [00:34<02:14, 68.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 14%|█▎ | 1.44G/10.7G [00:34<02:23, 64.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 14%|█▎ | 1.46G/10.7G [00:34<02:07, 72.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 14%|█▍ | 1.47G/10.7G [00:34<02:20, 65.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 14%|█▍ | 1.49G/10.7G [00:34<02:14, 68.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 14%|█▍ | 1.50G/10.7G [00:35<02:05, 72.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 14%|█▍ | 1.52G/10.7G [00:35<02:07, 71.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 14%|█▍ | 1.54G/10.7G [00:35<02:10, 69.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 15%|█▍ | 1.55G/10.7G [00:35<02:25, 62.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 15%|█▍ | 1.57G/10.7G [00:36<02:32, 59.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 15%|█▍ | 1.58G/10.7G [00:36<02:19, 65.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 15%|█▌ | 1.60G/10.7G [00:36<02:16, 66.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 15%|█▌ | 1.62G/10.7G [00:36<02:16, 66.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 15%|█▌ | 1.63G/10.7G [00:37<02:16, 65.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 15%|█▌ | 1.65G/10.7G [00:37<02:18, 64.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 16%|█▌ | 1.66G/10.7G [00:37<02:10, 68.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 16%|█▌ | 1.68G/10.7G [00:37<02:14, 66.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 16%|█▌ | 1.70G/10.7G [00:38<02:08, 69.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 16%|█▌ | 1.71G/10.7G [00:38<02:21, 63.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 16%|█▌ | 1.73G/10.7G [00:38<02:22, 62.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 16%|█▋ | 1.74G/10.7G [00:38<02:31, 58.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 17%|█▋ | 1.76G/10.7G [00:39<02:27, 60.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 17%|█▋ | 1.78G/10.7G [00:39<02:21, 62.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 17%|█▋ | 1.79G/10.7G [00:39<02:11, 67.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 17%|█▋ | 1.81G/10.7G [00:39<02:16, 64.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 17%|█▋ | 1.82G/10.7G [00:40<02:46, 53.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 17%|█▋ | 1.84G/10.7G [00:40<02:40, 54.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 17%|█▋ | 1.86G/10.7G [00:41<03:13, 45.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 18%|█▊ | 1.87G/10.7G [00:41<03:08, 46.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 18%|█▊ | 1.89G/10.7G [00:41<02:53, 50.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 18%|█▊ | 1.90G/10.7G [00:41<02:38, 55.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 18%|█▊ | 1.92G/10.7G [00:42<02:26, 59.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 18%|█▊ | 1.94G/10.7G [00:42<02:29, 58.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 18%|█▊ | 1.95G/10.7G [00:42<02:30, 57.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 18%|█▊ | 1.97G/10.7G [00:42<02:33, 56.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 19%|█▊ | 1.98G/10.7G [00:43<02:36, 55.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 19%|█▉ | 2.00G/10.7G [00:43<02:42, 53.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 19%|█▉ | 2.02G/10.7G [00:43<02:35, 55.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 19%|█▉ | 2.03G/10.7G [00:44<02:32, 56.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 19%|█▉ | 2.05G/10.7G [00:44<02:22, 60.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 19%|█▉ | 2.06G/10.7G [00:44<02:22, 60.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 20%|█▉ | 2.08G/10.7G [00:44<02:11, 65.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 20%|█▉ | 2.10G/10.7G [00:45<02:14, 63.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 20%|█▉ | 2.11G/10.7G [00:45<02:16, 62.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 20%|█▉ | 2.13G/10.7G [00:45<02:07, 67.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 20%|██ | 2.14G/10.7G [00:45<02:13, 63.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 20%|██ | 2.16G/10.7G [00:46<02:19, 61.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 20%|██ | 2.18G/10.7G [00:46<02:12, 63.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 21%|██ | 2.19G/10.7G [00:46<02:10, 64.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 21%|██ | 2.21G/10.7G [00:46<02:18, 61.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 21%|██ | 2.22G/10.7G [00:47<02:49, 49.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 21%|██ | 2.24G/10.7G [00:47<02:43, 51.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 21%|██ | 2.26G/10.7G [00:47<02:35, 54.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 21%|██▏ | 2.27G/10.7G [00:48<02:33, 54.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 21%|██▏ | 2.29G/10.7G [00:48<02:22, 58.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 22%|██▏ | 2.30G/10.7G [00:48<02:36, 53.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 22%|██▏ | 2.32G/10.7G [00:49<02:29, 55.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 22%|██▏ | 2.34G/10.7G [00:49<02:31, 55.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 22%|██▏ | 2.35G/10.7G [00:49<02:24, 57.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 22%|██▏ | 2.37G/10.7G [00:49<02:10, 63.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 22%|██▏ | 2.38G/10.7G [00:50<02:24, 57.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 23%|██▎ | 2.40G/10.7G [00:50<02:27, 56.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 23%|██▎ | 2.42G/10.7G [00:50<02:18, 59.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 23%|██▎ | 2.43G/10.7G [00:50<02:25, 56.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 23%|██▎ | 2.45G/10.7G [00:51<02:10, 62.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 23%|██▎ | 2.46G/10.7G [00:51<02:15, 60.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 23%|██▎ | 2.48G/10.7G [00:51<02:07, 64.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 23%|██▎ | 2.50G/10.7G [00:51<02:09, 62.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 24%|██▎ | 2.51G/10.7G [00:52<02:01, 66.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 24%|██▎ | 2.53G/10.7G [00:52<02:19, 58.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 24%|██▍ | 2.54G/10.7G [00:52<02:11, 61.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 24%|██▍ | 2.56G/10.7G [00:52<02:08, 62.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 24%|██▍ | 2.58G/10.7G [00:53<02:08, 62.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 24%|██▍ | 2.59G/10.7G [00:53<02:10, 61.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 24%|██▍ | 2.61G/10.7G [00:53<02:20, 57.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 25%|██▍ | 2.62G/10.7G [00:54<02:11, 61.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 25%|██▍ | 2.64G/10.7G [00:54<02:07, 62.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 25%|██▍ | 2.66G/10.7G [00:54<02:16, 58.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 25%|██▌ | 2.67G/10.7G [00:54<02:06, 63.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 25%|██▌ | 2.69G/10.7G [00:55<02:04, 63.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 25%|██▌ | 2.70G/10.7G [00:55<02:41, 49.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 26%|██▌ | 2.72G/10.7G [00:56<03:07, 42.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 26%|██▌ | 2.74G/10.7G [00:56<02:49, 46.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 26%|██▌ | 2.75G/10.7G [00:56<02:26, 53.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 26%|██▌ | 2.77G/10.7G [00:56<02:19, 56.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 26%|██▌ | 2.78G/10.7G [00:56<02:11, 60.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 26%|██▋ | 2.80G/10.7G [00:57<02:02, 64.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 26%|██▋ | 2.82G/10.7G [00:57<02:12, 59.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 27%|██▋ | 2.83G/10.7G [00:57<02:03, 63.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 27%|██▋ | 2.85G/10.7G [00:57<02:09, 60.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 27%|██▋ | 2.86G/10.7G [00:58<02:05, 61.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 27%|██▋ | 2.88G/10.7G [00:58<01:57, 66.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 27%|██▋ | 2.90G/10.7G [00:58<01:55, 67.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 27%|██▋ | 2.91G/10.7G [00:58<01:37, 79.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 27%|██▋ | 2.92G/10.7G [00:58<01:41, 75.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 27%|██▋ | 2.93G/10.7G [00:59<02:25, 53.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 28%|██▊ | 2.94G/10.7G [00:59<02:08, 59.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 28%|██▊ | 2.96G/10.7G [01:02<10:09, 12.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 28%|██▊ | 2.98G/10.7G [01:02<07:38, 16.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 28%|██▊ | 2.99G/10.7G [01:03<05:47, 22.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 28%|██▊ | 3.01G/10.7G [01:03<04:36, 27.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 28%|██▊ | 3.02G/10.7G [01:03<03:42, 34.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 29%|██▊ | 3.04G/10.7G [01:03<03:04, 41.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 29%|██▊ | 3.06G/10.7G [01:04<02:38, 48.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 29%|██▉ | 3.07G/10.7G [01:04<02:28, 51.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 29%|██▉ | 3.09G/10.7G [01:04<02:23, 52.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 29%|██▉ | 3.10G/10.7G [01:04<02:06, 59.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 29%|██▉ | 3.12G/10.7G [01:05<02:03, 60.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 29%|██▉ | 3.14G/10.7G [01:05<01:52, 66.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 30%|██▉ | 3.15G/10.7G [01:05<01:51, 67.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 30%|██▉ | 3.17G/10.7G [01:05<02:01, 61.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 30%|██▉ | 3.18G/10.7G [01:06<02:09, 57.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 30%|███ | 3.20G/10.7G [01:06<02:02, 61.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 30%|███ | 3.22G/10.7G [01:06<02:00, 61.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 30%|███ | 3.23G/10.7G [01:06<02:05, 59.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 30%|███ | 3.25G/10.7G [01:07<02:04, 59.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 31%|███ | 3.26G/10.7G [01:07<02:21, 52.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 31%|███ | 3.28G/10.7G [01:07<02:12, 55.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 31%|███ | 3.30G/10.7G [01:07<02:00, 61.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 31%|███ | 3.31G/10.7G [01:08<02:22, 51.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 31%|███ | 3.33G/10.7G [01:08<02:17, 53.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 31%|███▏ | 3.34G/10.7G [01:08<02:15, 53.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 32%|███▏ | 3.36G/10.7G [01:09<02:12, 55.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 32%|███▏ | 3.38G/10.7G [01:09<02:19, 52.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 32%|███▏ | 3.39G/10.7G [01:09<02:14, 53.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 32%|███▏ | 3.41G/10.7G [01:10<02:02, 59.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 32%|███▏ | 3.42G/10.7G [01:10<02:04, 58.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 32%|███▏ | 3.44G/10.7G [01:10<02:37, 45.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 32%|███▏ | 3.46G/10.7G [01:11<02:16, 52.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 33%|███▎ | 3.47G/10.7G [01:12<04:12, 28.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 33%|███▎ | 3.49G/10.7G [01:12<03:28, 34.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 33%|███▎ | 3.50G/10.7G [01:12<02:49, 42.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 33%|███▎ | 3.52G/10.7G [01:12<02:32, 46.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 33%|███▎ | 3.54G/10.7G [01:13<02:32, 46.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 33%|███▎ | 3.55G/10.7G [01:13<02:12, 53.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 33%|███▎ | 3.57G/10.7G [01:13<02:15, 52.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 34%|███▎ | 3.58G/10.7G [01:13<02:02, 57.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 34%|███▍ | 3.60G/10.7G [01:15<04:09, 28.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 34%|███▍ | 3.62G/10.7G [01:15<03:55, 29.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 34%|███▍ | 3.63G/10.7G [01:15<03:26, 34.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 34%|███▍ | 3.65G/10.7G [01:16<02:57, 39.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 34%|███▍ | 3.66G/10.7G [01:16<02:27, 47.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 35%|███▍ | 3.68G/10.7G [01:16<02:15, 51.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 35%|███▍ | 3.70G/10.7G [01:16<02:11, 53.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 35%|███▍ | 3.71G/10.7G [01:17<01:58, 58.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 35%|███▍ | 3.73G/10.7G [01:17<02:00, 57.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 35%|███▌ | 3.74G/10.7G [01:17<02:06, 54.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 35%|███▌ | 3.76G/10.7G [01:18<02:34, 44.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 35%|███▌ | 3.78G/10.7G [01:18<02:18, 49.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 36%|███▌ | 3.79G/10.7G [01:18<02:10, 52.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 36%|███▌ | 3.81G/10.7G [01:19<02:02, 55.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 36%|███▌ | 3.82G/10.7G [01:19<02:02, 55.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 36%|███▌ | 3.84G/10.7G [01:19<01:56, 58.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 36%|███▌ | 3.86G/10.7G [01:19<01:56, 58.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 36%|███▋ | 3.87G/10.7G [01:20<02:08, 52.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 36%|███▋ | 3.89G/10.7G [01:20<01:54, 59.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 37%|███▋ | 3.90G/10.7G [01:20<01:42, 65.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 37%|███▋ | 3.92G/10.7G [01:20<01:50, 61.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 37%|███▋ | 3.94G/10.7G [01:21<02:02, 54.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 37%|███▋ | 3.95G/10.7G [01:21<02:06, 53.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 37%|███▋ | 3.97G/10.7G [01:21<01:59, 55.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 37%|███▋ | 3.98G/10.7G [01:22<01:56, 57.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 38%|███▊ | 4.00G/10.7G [01:22<01:59, 55.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 38%|███▊ | 4.02G/10.7G [01:22<02:01, 54.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 38%|███▊ | 4.03G/10.7G [01:22<01:55, 57.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 38%|███▊ | 4.05G/10.7G [01:23<01:47, 61.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 38%|███▊ | 4.06G/10.7G [01:23<01:57, 56.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 38%|███▊ | 4.08G/10.7G [01:23<01:46, 61.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 38%|███▊ | 4.10G/10.7G [01:23<01:42, 64.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 39%|███▊ | 4.11G/10.7G [01:24<01:37, 66.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 39%|███▊ | 4.13G/10.7G [01:24<01:41, 64.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 39%|███▉ | 4.14G/10.7G [01:24<01:36, 67.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 39%|███▉ | 4.16G/10.7G [01:24<01:39, 65.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 39%|███▉ | 4.18G/10.7G [01:25<01:44, 62.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 39%|███▉ | 4.19G/10.7G [01:25<01:34, 68.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 39%|███▉ | 4.21G/10.7G [01:25<01:32, 69.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 40%|███▉ | 4.22G/10.7G [01:25<01:43, 62.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 40%|███▉ | 4.24G/10.7G [01:26<01:57, 54.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 40%|███▉ | 4.26G/10.7G [01:26<02:04, 51.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 40%|████ | 4.27G/10.7G [01:26<02:10, 49.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 40%|████ | 4.29G/10.7G [01:27<02:00, 52.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 40%|████ | 4.30G/10.7G [01:27<01:49, 58.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 41%|████ | 4.32G/10.7G [01:27<01:49, 57.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 41%|████ | 4.34G/10.7G [01:27<01:42, 61.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 41%|████ | 4.35G/10.7G [01:28<01:33, 67.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 41%|████ | 4.37G/10.7G [01:28<01:43, 61.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 41%|████ | 4.38G/10.7G [01:28<02:04, 50.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 41%|████▏ | 4.40G/10.7G [01:29<01:55, 54.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 41%|████▏ | 4.42G/10.7G [01:29<01:49, 56.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 42%|████▏ | 4.43G/10.7G [01:29<01:45, 58.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 42%|████▏ | 4.45G/10.7G [01:29<01:54, 54.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 42%|████▏ | 4.46G/10.7G [01:30<01:45, 58.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 42%|████▏ | 4.48G/10.7G [01:30<01:53, 54.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 42%|████▏ | 4.50G/10.7G [01:30<01:37, 63.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 42%|████▏ | 4.51G/10.7G [01:30<01:35, 64.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 42%|████▏ | 4.53G/10.7G [01:31<01:45, 58.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 43%|████▎ | 4.54G/10.7G [01:31<01:46, 57.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 43%|████▎ | 4.56G/10.7G [01:32<02:33, 39.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 43%|████▎ | 4.58G/10.7G [01:32<02:12, 45.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 43%|████▎ | 4.59G/10.7G [01:32<02:20, 43.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 43%|████▎ | 4.61G/10.7G [01:34<05:21, 18.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 43%|████▎ | 4.62G/10.7G [01:35<04:25, 22.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 44%|████▎ | 4.64G/10.7G [01:35<03:38, 27.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 44%|████▎ | 4.66G/10.7G [01:35<03:05, 32.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 44%|████▍ | 4.67G/10.7G [01:36<02:37, 38.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 44%|████▍ | 4.69G/10.7G [01:36<02:22, 41.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 44%|████▍ | 4.70G/10.7G [01:36<02:10, 45.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 44%|████▍ | 4.72G/10.7G [01:36<01:58, 50.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 44%|████▍ | 4.74G/10.7G [01:37<02:01, 48.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 45%|████▍ | 4.75G/10.7G [01:37<01:52, 52.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 45%|████▍ | 4.77G/10.7G [01:37<01:46, 55.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 45%|████▍ | 4.78G/10.7G [01:38<01:40, 58.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 45%|████▌ | 4.80G/10.7G [01:38<02:32, 38.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 45%|████▌ | 4.82G/10.7G [01:39<02:22, 40.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 45%|████▌ | 4.83G/10.7G [01:39<02:24, 40.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 46%|████▌ | 4.85G/10.7G [01:39<02:32, 38.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 46%|████▌ | 4.86G/10.7G [01:40<02:34, 37.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 46%|████▌ | 4.88G/10.7G [01:40<02:03, 46.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 46%|████▌ | 4.88G/10.7G [01:41<03:42, 26.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 46%|████▌ | 4.90G/10.7G [01:41<03:15, 29.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 46%|████▌ | 4.91G/10.7G [01:42<03:11, 30.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 46%|████▋ | 4.93G/10.7G [01:42<02:51, 33.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 46%|████▋ | 4.94G/10.7G [01:42<02:34, 36.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 47%|████▋ | 4.96G/10.7G [01:43<02:18, 41.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 47%|████▋ | 4.98G/10.7G [01:43<02:10, 43.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 47%|████▋ | 4.99G/10.7G [01:43<02:08, 44.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 47%|████▋ | 5.01G/10.7G [01:44<02:00, 46.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 47%|████▋ | 5.02G/10.7G [01:44<01:57, 47.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 47%|████▋ | 5.04G/10.7G [01:44<01:48, 51.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 47%|████▋ | 5.06G/10.7G [01:44<01:42, 54.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 48%|████▊ | 5.07G/10.7G [01:45<01:48, 51.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 48%|████▊ | 5.09G/10.7G [01:45<01:53, 49.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 48%|████▊ | 5.10G/10.7G [01:45<01:51, 49.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 48%|████▊ | 5.12G/10.7G [01:46<02:15, 40.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 48%|████▊ | 5.14G/10.7G [01:46<02:12, 41.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 48%|████▊ | 5.15G/10.7G [01:47<02:07, 43.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 49%|████▊ | 5.17G/10.7G [01:47<01:55, 47.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 49%|████▊ | 5.18G/10.7G [01:47<02:00, 45.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 49%|████▉ | 5.20G/10.7G [01:48<01:53, 48.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 49%|████▉ | 5.22G/10.7G [01:48<01:48, 50.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 49%|████▉ | 5.23G/10.7G [01:48<01:38, 54.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 49%|████▉ | 5.25G/10.7G [01:48<01:27, 61.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 49%|████▉ | 5.26G/10.7G [01:49<01:41, 53.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 50%|████▉ | 5.28G/10.7G [01:49<01:33, 57.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 50%|████▉ | 5.29G/10.7G [01:49<01:21, 66.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 50%|████▉ | 5.30G/10.7G [01:49<01:32, 57.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 50%|████▉ | 5.31G/10.7G [01:50<01:37, 55.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 50%|█████ | 5.33G/10.7G [01:50<01:34, 56.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 50%|█████ | 5.34G/10.7G [01:50<01:31, 57.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 50%|█████ | 5.36G/10.7G [01:50<01:35, 55.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 50%|█████ | 5.38G/10.7G [01:51<01:43, 51.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 51%|█████ | 5.39G/10.7G [01:51<01:41, 51.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 51%|█████ | 5.41G/10.7G [01:51<01:38, 53.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 51%|█████ | 5.42G/10.7G [01:52<01:34, 55.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 51%|█████ | 5.44G/10.7G [01:52<01:32, 56.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 51%|█████ | 5.45G/10.7G [01:52<01:20, 64.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 51%|█████ | 5.46G/10.7G [01:52<01:37, 53.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 51%|█████▏ | 5.47G/10.7G [01:53<01:42, 50.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 52%|█████▏ | 5.49G/10.7G [01:53<01:34, 54.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 52%|█████▏ | 5.50G/10.7G [01:53<01:42, 50.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 52%|█████▏ | 5.51G/10.7G [01:53<01:37, 52.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 52%|█████▏ | 5.52G/10.7G [01:53<01:36, 53.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 52%|█████▏ | 5.52G/10.7G [01:54<02:03, 41.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 52%|█████▏ | 5.54G/10.7G [01:54<02:01, 42.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 52%|█████▏ | 5.55G/10.7G [01:54<01:40, 50.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 52%|█████▏ | 5.55G/10.7G [01:54<02:19, 36.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 52%|█████▏ | 5.56G/10.7G [01:55<01:52, 45.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 52%|█████▏ | 5.57G/10.7G [01:55<02:45, 30.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 52%|█████▏ | 5.58G/10.7G [01:55<02:15, 37.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 53%|█████▎ | 5.60G/10.7G [01:55<01:58, 42.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 53%|█████▎ | 5.62G/10.7G [01:56<01:55, 43.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 53%|█████▎ | 5.63G/10.7G [01:56<01:57, 42.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 53%|█████▎ | 5.65G/10.7G [01:56<01:45, 47.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 53%|█████▎ | 5.66G/10.7G [01:57<01:24, 59.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 53%|█████▎ | 5.67G/10.7G [01:57<01:43, 48.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 53%|█████▎ | 5.68G/10.7G [01:57<01:52, 44.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 53%|█████▎ | 5.70G/10.7G [01:58<01:50, 44.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 54%|█████▎ | 5.71G/10.7G [01:58<01:53, 43.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 54%|█████▍ | 5.73G/10.7G [01:58<01:41, 48.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 54%|█████▍ | 5.74G/10.7G [01:58<01:33, 52.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 54%|█████▍ | 5.76G/10.7G [01:59<01:32, 52.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 54%|█████▍ | 5.77G/10.7G [01:59<01:21, 60.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 54%|█████▍ | 5.78G/10.7G [01:59<01:47, 45.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 54%|█████▍ | 5.79G/10.7G [02:00<01:54, 42.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 55%|█████▍ | 5.81G/10.7G [02:00<01:37, 49.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 55%|█████▍ | 5.82G/10.7G [02:00<02:08, 37.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 55%|█████▍ | 5.84G/10.7G [02:01<01:57, 41.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 55%|█████▍ | 5.86G/10.7G [02:01<01:47, 44.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 55%|█████▌ | 5.87G/10.7G [02:01<01:40, 47.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 55%|█████▌ | 5.89G/10.7G [02:02<01:41, 47.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 55%|█████▌ | 5.90G/10.7G [02:02<01:28, 53.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 55%|█████▌ | 5.91G/10.7G [02:02<02:16, 34.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 56%|█████▌ | 5.92G/10.7G [02:03<02:28, 31.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 56%|█████▌ | 5.93G/10.7G [02:03<02:14, 35.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 56%|█████▌ | 5.93G/10.7G [02:03<02:00, 39.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 56%|█████▌ | 5.94G/10.7G [02:03<02:27, 31.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 56%|█████▌ | 5.95G/10.7G [02:04<02:19, 33.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 56%|█████▌ | 5.96G/10.7G [02:04<01:43, 45.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 56%|█████▌ | 5.97G/10.7G [02:04<02:01, 38.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 56%|█████▌ | 5.98G/10.7G [02:04<01:58, 39.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 56%|█████▋ | 6.00G/10.7G [02:05<01:44, 44.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 56%|█████▋ | 6.02G/10.7G [02:05<02:07, 36.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 57%|█████▋ | 6.03G/10.7G [02:05<01:53, 40.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 57%|█████▋ | 6.05G/10.7G [02:06<01:32, 49.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 57%|█████▋ | 6.06G/10.7G [02:06<01:27, 52.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 57%|█████▋ | 6.08G/10.7G [02:06<01:30, 50.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 57%|█████▋ | 6.10G/10.7G [02:07<01:26, 52.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 57%|█████▋ | 6.11G/10.7G [02:07<02:09, 35.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 58%|█████▊ | 6.13G/10.7G [02:08<01:49, 41.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 58%|█████▊ | 6.14G/10.7G [02:08<01:48, 41.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 58%|█████▊ | 6.16G/10.7G [02:08<01:38, 45.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 58%|█████▊ | 6.18G/10.7G [02:09<01:35, 47.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 58%|█████▊ | 6.19G/10.7G [02:09<01:41, 44.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 58%|█████▊ | 6.21G/10.7G [02:09<01:34, 47.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 58%|█████▊ | 6.22G/10.7G [02:10<01:34, 46.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 59%|█████▊ | 6.23G/10.7G [02:10<01:26, 51.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 59%|█████▊ | 6.24G/10.7G [02:10<01:26, 51.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 59%|█████▊ | 6.25G/10.7G [02:10<01:53, 38.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 59%|█████▊ | 6.26G/10.7G [02:10<01:46, 41.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 59%|█████▉ | 6.27G/10.7G [02:11<01:39, 44.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 59%|█████▉ | 6.29G/10.7G [02:11<01:30, 48.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 59%|█████▉ | 6.30G/10.7G [02:11<01:28, 49.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 59%|█████▉ | 6.32G/10.7G [02:12<01:35, 45.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 59%|█████▉ | 6.34G/10.7G [02:12<01:32, 46.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 60%|█████▉ | 6.35G/10.7G [02:12<01:28, 48.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 60%|█████▉ | 6.37G/10.7G [02:13<01:42, 41.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 60%|█████▉ | 6.38G/10.7G [02:13<01:52, 37.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 60%|██████ | 6.40G/10.7G [02:14<01:37, 43.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 60%|██████ | 6.42G/10.7G [02:14<01:31, 46.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 60%|██████ | 6.43G/10.7G [02:14<01:12, 58.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 60%|██████ | 6.44G/10.7G [02:14<01:21, 51.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 61%|██████ | 6.45G/10.7G [02:14<01:19, 52.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 61%|██████ | 6.46G/10.7G [02:15<01:18, 53.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 61%|██████ | 6.48G/10.7G [02:15<01:46, 39.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 61%|██████ | 6.50G/10.7G [02:16<01:34, 43.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 61%|██████ | 6.51G/10.7G [02:16<01:26, 48.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 61%|██████▏ | 6.53G/10.7G [02:16<01:25, 48.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 61%|██████▏ | 6.54G/10.7G [02:16<01:16, 53.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 62%|██████▏ | 6.56G/10.7G [02:17<01:29, 45.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 62%|██████▏ | 6.58G/10.7G [02:17<01:20, 50.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 62%|██████▏ | 6.59G/10.7G [02:17<01:19, 51.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 62%|██████▏ | 6.61G/10.7G [02:18<01:15, 53.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 62%|██████▏ | 6.62G/10.7G [02:19<02:21, 28.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 62%|██████▏ | 6.64G/10.7G [02:19<01:58, 33.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 62%|██████▏ | 6.66G/10.7G [02:19<01:36, 41.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 63%|██████▎ | 6.67G/10.7G [02:20<01:27, 45.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 63%|██████▎ | 6.69G/10.7G [02:20<01:30, 43.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 63%|██████▎ | 6.70G/10.7G [02:20<01:28, 44.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 63%|██████▎ | 6.72G/10.7G [02:20<01:17, 50.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 63%|██████▎ | 6.74G/10.7G [02:21<01:21, 48.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 63%|██████▎ | 6.75G/10.7G [02:21<01:15, 51.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 64%|██████▎ | 6.77G/10.7G [02:21<01:14, 52.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 64%|██████▎ | 6.78G/10.7G [02:22<01:11, 54.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 64%|██████▍ | 6.80G/10.7G [02:22<01:19, 48.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 64%|██████▍ | 6.82G/10.7G [02:22<01:19, 48.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 64%|██████▍ | 6.83G/10.7G [02:23<01:14, 51.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 64%|██████▍ | 6.85G/10.7G [02:23<01:11, 53.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 64%|██████▍ | 6.86G/10.7G [02:23<01:12, 52.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 65%|██████▍ | 6.88G/10.7G [02:24<01:07, 56.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 65%|██████▍ | 6.90G/10.7G [02:24<01:47, 34.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 65%|██████▍ | 6.91G/10.7G [02:24<01:27, 42.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 65%|██████▍ | 6.92G/10.7G [02:25<01:37, 38.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 65%|██████▌ | 6.93G/10.7G [02:25<01:34, 39.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 65%|██████▌ | 6.94G/10.7G [02:25<01:26, 43.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 65%|██████▌ | 6.94G/10.7G [02:26<01:47, 34.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 65%|██████▌ | 6.96G/10.7G [02:26<02:16, 27.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 65%|██████▌ | 6.98G/10.7G [02:27<01:45, 35.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 66%|██████▌ | 6.99G/10.7G [02:28<02:35, 23.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 66%|██████▌ | 7.01G/10.7G [02:28<02:03, 29.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 66%|██████▌ | 7.02G/10.7G [02:28<01:50, 32.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 66%|██████▌ | 7.04G/10.7G [02:29<01:32, 39.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 66%|██████▌ | 7.06G/10.7G [02:29<01:21, 44.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 66%|██████▋ | 7.07G/10.7G [02:29<01:13, 48.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 67%|██████▋ | 7.09G/10.7G [02:29<01:12, 48.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 67%|██████▋ | 7.10G/10.7G [02:30<01:13, 48.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 67%|██████▋ | 7.12G/10.7G [02:30<01:08, 51.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 67%|██████▋ | 7.14G/10.7G [02:30<01:09, 50.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 67%|██████▋ | 7.15G/10.7G [02:31<01:06, 52.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 67%|██████▋ | 7.17G/10.7G [02:31<01:05, 53.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 67%|██████▋ | 7.18G/10.7G [02:31<01:00, 57.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 68%|██████▊ | 7.20G/10.7G [02:31<01:04, 53.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 68%|██████▊ | 7.22G/10.7G [02:32<00:58, 59.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 68%|██████▊ | 7.23G/10.7G [02:32<00:57, 59.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 68%|██████▊ | 7.25G/10.7G [02:32<00:57, 58.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 68%|██████▊ | 7.26G/10.7G [02:32<00:58, 57.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 68%|██████▊ | 7.28G/10.7G [02:33<01:00, 56.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 68%|██████▊ | 7.30G/10.7G [02:33<00:58, 57.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 69%|██████▊ | 7.31G/10.7G [02:33<01:01, 53.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 69%|██████▊ | 7.32G/10.7G [02:33<00:56, 58.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 69%|██████▉ | 7.33G/10.7G [02:34<01:05, 50.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 69%|██████▉ | 7.33G/10.7G [02:34<01:26, 38.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 69%|██████▉ | 7.34G/10.7G [02:34<01:24, 39.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 69%|██████▉ | 7.36G/10.7G [02:35<01:11, 46.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 69%|██████▉ | 7.38G/10.7G [02:35<01:10, 46.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 69%|██████▉ | 7.38G/10.7G [02:35<01:07, 48.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 69%|██████▉ | 7.39G/10.7G [02:35<01:07, 48.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 69%|██████▉ | 7.40G/10.7G [02:35<01:26, 37.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 70%|██████▉ | 7.41G/10.7G [02:36<01:14, 43.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 70%|██████▉ | 7.42G/10.7G [02:36<01:21, 39.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 70%|██████▉ | 7.44G/10.7G [02:37<01:57, 27.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 70%|██████▉ | 7.45G/10.7G [02:37<01:31, 35.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 70%|███████ | 7.46G/10.7G [02:37<01:39, 32.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 70%|███████ | 7.47G/10.7G [02:38<01:27, 36.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 70%|███████ | 7.49G/10.7G [02:38<01:16, 41.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 70%|███████ | 7.50G/10.7G [02:39<01:36, 32.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 71%|███████ | 7.52G/10.7G [02:39<01:26, 36.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 71%|███████ | 7.54G/10.7G [02:39<01:15, 41.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 71%|███████ | 7.55G/10.7G [02:39<01:06, 46.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 71%|███████ | 7.57G/10.7G [02:40<00:58, 52.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 71%|███████ | 7.58G/10.7G [02:40<01:10, 43.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 71%|███████▏ | 7.60G/10.7G [02:40<01:03, 48.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 71%|███████▏ | 7.62G/10.7G [02:41<01:00, 50.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 72%|███████▏ | 7.63G/10.7G [02:41<00:58, 51.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 72%|███████▏ | 7.65G/10.7G [02:41<00:52, 57.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 72%|███████▏ | 7.66G/10.7G [02:42<00:52, 57.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 72%|███████▏ | 7.68G/10.7G [02:42<00:55, 53.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 72%|███████▏ | 7.70G/10.7G [02:42<00:54, 53.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 72%|███████▏ | 7.71G/10.7G [02:42<00:55, 53.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 73%|███████▎ | 7.73G/10.7G [02:43<00:52, 55.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 73%|███████▎ | 7.74G/10.7G [02:43<00:52, 54.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 73%|███████▎ | 7.76G/10.7G [02:43<01:00, 48.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 73%|███████▎ | 7.78G/10.7G [02:44<01:00, 47.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 73%|███████▎ | 7.79G/10.7G [02:44<00:56, 50.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 73%|███████▎ | 7.81G/10.7G [02:44<00:55, 51.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 73%|███████▎ | 7.82G/10.7G [02:45<00:50, 55.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 74%|███████▎ | 7.84G/10.7G [02:45<01:03, 44.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 74%|███████▎ | 7.86G/10.7G [02:46<01:22, 34.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 74%|███████▍ | 7.87G/10.7G [02:46<01:16, 36.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 74%|███████▍ | 7.89G/10.7G [02:46<01:04, 42.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 74%|███████▍ | 7.90G/10.7G [02:47<01:05, 42.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 74%|███████▍ | 7.92G/10.7G [02:47<00:55, 48.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 74%|███████▍ | 7.94G/10.7G [02:48<01:09, 38.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 75%|███████▍ | 7.95G/10.7G [02:48<00:59, 45.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 75%|███████▍ | 7.97G/10.7G [02:48<00:55, 48.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 75%|███████▍ | 7.98G/10.7G [02:49<00:56, 47.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 75%|███████▌ | 8.00G/10.7G [02:49<00:48, 54.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 75%|███████▌ | 8.02G/10.7G [02:49<00:44, 59.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 75%|███████▌ | 8.03G/10.7G [02:49<00:44, 59.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 76%|███████▌ | 8.05G/10.7G [02:49<00:41, 62.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 76%|███████▌ | 8.06G/10.7G [02:50<00:41, 61.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 76%|███████▌ | 8.08G/10.7G [02:50<00:46, 54.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 76%|███████▌ | 8.10G/10.7G [02:50<00:43, 59.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 76%|███████▌ | 8.11G/10.7G [02:50<00:41, 62.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 76%|███████▋ | 8.13G/10.7G [02:51<00:40, 61.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 76%|███████▋ | 8.14G/10.7G [02:51<00:37, 67.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 77%|███████▋ | 8.16G/10.7G [02:51<00:38, 65.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 77%|███████▋ | 8.18G/10.7G [02:51<00:39, 63.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 77%|███████▋ | 8.19G/10.7G [02:52<00:37, 65.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 77%|███████▋ | 8.21G/10.7G [02:52<00:38, 62.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 77%|███████▋ | 8.22G/10.7G [02:52<00:37, 64.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 77%|███████▋ | 8.24G/10.7G [02:53<00:41, 57.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 77%|███████▋ | 8.26G/10.7G [02:53<00:40, 59.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 78%|███████▊ | 8.27G/10.7G [02:53<00:39, 60.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 78%|███████▊ | 8.29G/10.7G [02:54<00:58, 40.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 78%|███████▊ | 8.30G/10.7G [02:54<00:56, 41.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 78%|███████▊ | 8.32G/10.7G [02:54<00:47, 49.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 78%|███████▊ | 8.34G/10.7G [02:55<00:44, 52.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 78%|███████▊ | 8.35G/10.7G [02:55<00:38, 59.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 79%|███████▊ | 8.37G/10.7G [02:55<00:42, 54.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 79%|███████▊ | 8.38G/10.7G [02:55<00:37, 60.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 79%|███████▉ | 8.40G/10.7G [02:56<00:36, 62.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 79%|███████▉ | 8.42G/10.7G [02:56<00:39, 56.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 79%|███████▉ | 8.43G/10.7G [02:56<00:40, 55.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 79%|███████▉ | 8.45G/10.7G [02:57<00:41, 52.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 79%|███████▉ | 8.46G/10.7G [02:57<00:38, 56.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 80%|███████▉ | 8.48G/10.7G [02:57<00:38, 56.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 80%|███████▉ | 8.50G/10.7G [02:57<00:34, 62.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 80%|███████▉ | 8.51G/10.7G [02:57<00:33, 64.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 80%|████████ | 8.53G/10.7G [02:58<00:31, 66.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 80%|████████ | 8.54G/10.7G [02:58<00:31, 66.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 80%|████████ | 8.56G/10.7G [02:58<00:35, 59.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 80%|████████ | 8.58G/10.7G [02:59<01:00, 34.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 81%|████████ | 8.59G/10.7G [02:59<00:51, 39.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 81%|████████ | 8.61G/10.7G [03:00<00:44, 45.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 81%|████████ | 8.62G/10.7G [03:00<00:41, 48.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 81%|████████ | 8.64G/10.7G [03:00<00:37, 53.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 81%|████████ | 8.66G/10.7G [03:00<00:36, 55.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 81%|████████▏ | 8.67G/10.7G [03:01<00:34, 57.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 82%|████████▏ | 8.69G/10.7G [03:01<00:34, 57.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 82%|████████▏ | 8.70G/10.7G [03:01<00:37, 51.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 82%|████████▏ | 8.72G/10.7G [03:02<00:47, 40.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 82%|████████▏ | 8.74G/10.7G [03:02<00:41, 46.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 82%|████████▏ | 8.75G/10.7G [03:02<00:38, 49.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 82%|████████▏ | 8.77G/10.7G [03:03<00:40, 46.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 82%|████████▏ | 8.78G/10.7G [03:03<00:35, 52.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 83%|████████▎ | 8.80G/10.7G [03:03<00:33, 55.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 83%|████████▎ | 8.82G/10.7G [03:04<00:32, 57.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 83%|████████▎ | 8.83G/10.7G [03:04<00:47, 38.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 83%|████████▎ | 8.85G/10.7G [03:05<00:40, 44.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 83%|████████▎ | 8.86G/10.7G [03:05<00:36, 49.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 83%|████████▎ | 8.88G/10.7G [03:05<00:34, 51.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 83%|████████▎ | 8.90G/10.7G [03:05<00:32, 53.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 84%|████████▎ | 8.91G/10.7G [03:06<00:29, 59.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 84%|████████▍ | 8.93G/10.7G [03:06<00:30, 57.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 84%|████████▍ | 8.94G/10.7G [03:06<00:27, 61.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 84%|████████▍ | 8.96G/10.7G [03:06<00:26, 64.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 84%|████████▍ | 8.98G/10.7G [03:07<00:26, 63.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 84%|████████▍ | 8.99G/10.7G [03:07<00:27, 60.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 85%|████████▍ | 9.01G/10.7G [03:07<00:24, 67.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 85%|████████▍ | 9.02G/10.7G [03:07<00:23, 68.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 85%|████████▍ | 9.04G/10.7G [03:07<00:22, 71.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 85%|████████▍ | 9.06G/10.7G [03:08<00:23, 69.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 85%|████████▌ | 9.07G/10.7G [03:08<00:25, 62.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 85%|████████▌ | 9.09G/10.7G [03:08<00:24, 63.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 85%|████████▌ | 9.10G/10.7G [03:08<00:22, 68.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 86%|████████▌ | 9.12G/10.7G [03:09<00:23, 65.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 86%|████████▌ | 9.14G/10.7G [03:09<00:23, 65.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 86%|████████▌ | 9.15G/10.7G [03:09<00:24, 61.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 86%|████████▌ | 9.17G/10.7G [03:09<00:24, 61.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 86%|████████▌ | 9.18G/10.7G [03:10<00:23, 62.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 86%|████████▋ | 9.20G/10.7G [03:10<00:24, 59.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 87%|████████▋ | 9.22G/10.7G [03:10<00:22, 65.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 87%|████████▋ | 9.23G/10.7G [03:10<00:21, 64.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 87%|████████▋ | 9.25G/10.7G [03:11<00:20, 69.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 87%|████████▋ | 9.26G/10.7G [03:11<00:19, 71.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 87%|████████▋ | 9.28G/10.7G [03:11<00:22, 61.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 87%|████████▋ | 9.30G/10.7G [03:12<00:22, 60.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 87%|████████▋ | 9.31G/10.7G [03:12<00:20, 66.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 88%|████████▊ | 9.33G/10.7G [03:12<00:18, 70.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 88%|████████▊ | 9.34G/10.7G [03:12<00:21, 62.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 88%|████████▊ | 9.36G/10.7G [03:13<00:22, 57.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 88%|████████▊ | 9.38G/10.7G [03:13<00:20, 61.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 88%|████████▊ | 9.39G/10.7G [03:13<00:19, 63.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 88%|████████▊ | 9.41G/10.7G [03:13<00:18, 66.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 88%|████████▊ | 9.42G/10.7G [03:13<00:17, 69.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 89%|████████▊ | 9.44G/10.7G [03:14<00:19, 61.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 89%|████████▉ | 9.46G/10.7G [03:14<00:19, 61.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 89%|████████▉ | 9.47G/10.7G [03:14<00:18, 63.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 89%|████████▉ | 9.49G/10.7G [03:15<00:19, 59.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 89%|████████▉ | 9.50G/10.7G [03:15<00:18, 61.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 89%|████████▉ | 9.50G/10.7G [03:15<00:33, 34.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 89%|████████▉ | 9.52G/10.7G [03:16<00:28, 40.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 90%|████████▉ | 9.54G/10.7G [03:16<00:23, 48.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 90%|████████▉ | 9.55G/10.7G [03:16<00:21, 50.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 90%|████████▉ | 9.57G/10.7G [03:16<00:20, 51.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 90%|████████▉ | 9.58G/10.7G [03:17<00:19, 54.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 90%|█████████ | 9.60G/10.7G [03:17<00:18, 57.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 90%|█████████ | 9.62G/10.7G [03:17<00:18, 56.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 90%|█████████ | 9.63G/10.7G [03:17<00:17, 60.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 91%|█████████ | 9.65G/10.7G [03:18<00:16, 61.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 91%|█████████ | 9.66G/10.7G [03:18<00:16, 61.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 91%|█████████ | 9.68G/10.7G [03:18<00:15, 63.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 91%|█████████ | 9.70G/10.7G [03:18<00:15, 62.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 91%|█████████ | 9.71G/10.7G [03:19<00:14, 64.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 91%|█████████▏| 9.73G/10.7G [03:19<00:14, 62.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 91%|█████████▏| 9.74G/10.7G [03:19<00:15, 60.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 92%|█████████▏| 9.76G/10.7G [03:19<00:14, 61.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 92%|█████████▏| 9.78G/10.7G [03:20<00:14, 59.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 92%|█████████▏| 9.79G/10.7G [03:20<00:15, 56.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 92%|█████████▏| 9.81G/10.7G [03:20<00:14, 59.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 92%|█████████▏| 9.82G/10.7G [03:21<00:15, 55.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 92%|█████████▏| 9.84G/10.7G [03:21<00:14, 57.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 93%|█████████▎| 9.86G/10.7G [03:21<00:13, 58.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 93%|█████████▎| 9.87G/10.7G [03:21<00:14, 54.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 93%|█████████▎| 9.89G/10.7G [03:22<00:12, 59.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 93%|█████████▎| 9.90G/10.7G [03:22<00:12, 59.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 93%|█████████▎| 9.92G/10.7G [03:22<00:13, 55.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 93%|█████████▎| 9.94G/10.7G [03:23<00:12, 57.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 93%|█████████▎| 9.95G/10.7G [03:23<00:12, 56.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 94%|█████████▎| 9.97G/10.7G [03:23<00:12, 55.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 94%|█████████▎| 9.98G/10.7G [03:23<00:10, 61.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 94%|█████████▍| 10.0G/10.7G [03:24<00:10, 61.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 94%|█████████▍| 10.0G/10.7G [03:24<00:09, 68.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 94%|█████████▍| 10.0G/10.7G [03:24<00:08, 69.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 94%|█████████▍| 10.0G/10.7G [03:24<00:08, 70.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 94%|█████████▍| 10.1G/10.7G [03:24<00:08, 70.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 95%|█████████▍| 10.1G/10.7G [03:25<00:08, 69.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 95%|█████████▍| 10.1G/10.7G [03:25<00:07, 70.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 95%|█████████▍| 10.1G/10.7G [03:25<00:07, 69.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 95%|█████████▌| 10.1G/10.7G [03:25<00:07, 70.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 95%|█████████▌| 10.1G/10.7G [03:26<00:07, 69.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 95%|█████████▌| 10.2G/10.7G [03:26<00:07, 68.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 96%|█████████▌| 10.2G/10.7G [03:26<00:06, 69.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 96%|█████████▌| 10.2G/10.7G [03:26<00:06, 67.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 96%|█████████▌| 10.2G/10.7G [03:27<00:07, 61.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 96%|█████████▌| 10.2G/10.7G [03:27<00:07, 58.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 96%|█████████▌| 10.2G/10.7G [03:27<00:06, 61.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 96%|█████████▋| 10.3G/10.7G [03:27<00:06, 59.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 96%|█████████▋| 10.3G/10.7G [03:28<00:06, 56.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 97%|█████████▋| 10.3G/10.7G [03:28<00:05, 63.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 97%|█████████▋| 10.3G/10.7G [03:28<00:05, 68.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 97%|█████████▋| 10.3G/10.7G [03:28<00:04, 70.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 97%|█████████▋| 10.3G/10.7G [03:29<00:04, 72.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 97%|█████████▋| 10.4G/10.7G [03:29<00:04, 73.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 97%|█████████▋| 10.4G/10.7G [03:29<00:03, 77.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 97%|█████████▋| 10.4G/10.7G [03:29<00:03, 77.6MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 98%|█████████▊| 10.4G/10.7G [03:29<00:03, 70.5MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 98%|█████████▊| 10.4G/10.7G [03:30<00:03, 62.1MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 98%|█████████▊| 10.4G/10.7G [03:30<00:03, 61.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 98%|█████████▊| 10.4G/10.7G [03:30<00:03, 62.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 98%|█████████▊| 10.5G/10.7G [03:31<00:03, 61.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 98%|█████████▊| 10.5G/10.7G [03:31<00:02, 67.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 99%|█████████▊| 10.5G/10.7G [03:31<00:02, 56.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 99%|█████████▊| 10.5G/10.7G [03:31<00:02, 59.2MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 99%|█████████▉| 10.5G/10.7G [03:32<00:02, 55.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 99%|█████████▉| 10.5G/10.7G [03:32<00:01, 60.0MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 99%|█████████▉| 10.6G/10.7G [03:32<00:01, 60.9MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 99%|█████████▉| 10.6G/10.7G [03:32<00:01, 64.7MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 99%|█████████▉| 10.6G/10.7G [03:33<00:01, 60.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 100%|█████████▉| 10.6G/10.7G [03:33<00:00, 60.8MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 100%|█████████▉| 10.6G/10.7G [03:33<00:00, 55.4MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 100%|█████████▉| 10.6G/10.7G [03:34<00:00, 41.3MB/s] ip-26-0-164-187_27331.1720048318217251415.pt.trace.json: 100%|██████████| 10.7G/10.7G [03:34<00:00, 49.6MB/s] | |