WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** [2023-11-16 11:31:41,918] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:31:41,918] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:31:41,918] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:31:41,918] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:31:41,918] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:31:41,919] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:31:41,919] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:31:41,921] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:32:02,331] [INFO] [comm.py:637:init_distributed] cdb=None [2023-11-16 11:32:02,514] [INFO] [comm.py:637:init_distributed] cdb=None [2023-11-16 11:32:02,595] [INFO] [comm.py:637:init_distributed] cdb=None [2023-11-16 11:32:02,609] [INFO] [comm.py:637:init_distributed] cdb=None [2023-11-16 11:32:02,628] [INFO] [comm.py:637:init_distributed] cdb=None [2023-11-16 11:32:02,633] [INFO] [comm.py:637:init_distributed] cdb=None [2023-11-16 11:32:02,634] [INFO] [comm.py:637:init_distributed] cdb=None [2023-11-16 11:32:02,635] [INFO] [comm.py:637:init_distributed] cdb=None 11/16/2023 11:32:02 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:32:02 - INFO - __main__ - Training/evaluation parameters TrainingArguments( _n_gpu=1, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, bf16=True, bf16_full_eval=True, data_seed=None, dataloader_drop_last=False, dataloader_num_workers=8, dataloader_pin_memory=True, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=72000, debug=[], deepspeed=train/deepspeed_config_zero2.json, disable_tqdm=False, do_eval=False, do_predict=False, do_train=True, eval_accumulation_steps=None, eval_delay=11/16/2023 11:32:02 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:32:02 - WARNING - __main__ - Process rank: 7, device: cuda:7, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:32:02 - INFO - __main__ - Training/evaluation parameters TrainingArguments( _n_gpu=1, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, bf16=True, bf16_full_eval=True, data_seed=None, dataloader_drop_last=False, dataloader_num_workers=8, dataloader_pin_memory=True, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=72000, debug=[], deepspeed=train/deepspeed_config_zero2.json, disable_tqdm=False, do_eval=False, do_predict=False, do_train=True, eval_accumulation_steps=None, eval_delay=0, eval_steps=None, evaluation_strategy=no, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'fsdp_min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=None, full_determinism=False, gradient_accumulation_steps=1, gradient_checkpointing=True, greater_is_better=None, group_by_length=False, half_precision_backend=auto, hub_model_id=None, hub_private_repo=False, hub_strategy=every_save, hub_token=, ignore_data_skip=False, include_inputs_for_metrics=False, jit_mode_eval=False, label_names=None, label_smoothing_factor=0.0, learning_rate=2e-05, length_column_name=length, load_best_model_at_end=False, local_rank=0, log_level=passive, log_level_replica=warning, log_on_each_node=True, logging_dir=/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20/runs/Nov16_11-31-41_ts-cbba87c5e7504a249f5127103d9ce40f-launcher, logging_first_step=False, logging_nan_inf_filter=True, logging_steps=10, logging_strategy=steps, lr_scheduler_type=cosine, max_grad_norm=1.0, max_steps=-1, metric_for_best_model=None, mp_parameters=, no_cuda=False, num_train_epochs=3.0, optim=adamw_hf, optim_args=None, output_dir=/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20, overwrite_output_dir=False, past_index=-1, per_device_eval_batch_size=2, per_device_train_batch_size=48, prediction_loss_only=False, push_to_hub=False, push_to_hub_model_id=None, push_to_hub_organization=None, push_to_hub_token=, ray_scope=last, remove_unused_columns=True, report_to=['tensorboard'], resume_from_checkpoint=None, run_name=/apdcephfs/shar11/16/2023 11:32:02 - WARNING - __main__ - Process rank: 2, device: cuda:2, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:32:02 - WARNING - __main__ - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:32:02 - WARNING - __main__ - Process rank: 7, device: cuda:7, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:32:02 - WARNING - __main__ - Process rank: 6, device: cuda:6, n_gpu: 1distributed training: True, 16-bits training: False Traceback (most recent call last): File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 739, in Traceback (most recent call last): File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 739, in Traceback (most recent call last): File "trans File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 739, in Traceback (most recent call last): File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 739,11/16/2023 11:32:02 - WARNING - __main__ - Process rank: 5, device: cuda:5, n_gpu: 1distributed training: True, 16-bits training: False Traceback (most recent call last): File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 739, in main() File " main() File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", main() File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 282, in main main() File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 282, in main Traceback (most recent call last): File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 739, in raise ValueError( ValueError: Output directory (/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmode raise ValueError( ValueError: Output directory (/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20) already exists and is not empty. Use --overwrite_output_dir to overcome. Traceback (most recent call last): File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 739, in raise ValueError( ValueError: Output directory (/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20) already exists and is not empty. Use --overwrite_output_dir to overcome. main() File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 282, in main raise ValueError( ValueError: Output directory (/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagp11/16/2023 11:32:02 - WARNING - __main__ - Process rank: 5, device: cuda:5, n_gpu: 1distributed training: True, 16-bits training: False Traceback (most recent call last): File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 739, in main() File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 282, in main raise ValueError( ValueError: Output directory (/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20) already exists and is not empty. Use --overwrite_output_dir to overcome. 11/16/2023 11:32:02 - WARNING - __main__ - Process rank: 6, device: cuda:6, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:32:02 - WARNING - __main__ - Process rank: 2, device: cuda:2, n_gpu: 1distributed training: True, 16-bits training: False Traceback (most recent call last): File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 739, in main() File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 282, in main raise ValueError( ValueError: Output directory (/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20) already exists and is not empty. Use --overwrite_output_dir to overcome. Traceback (most recent call last): File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 739, in main() File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 282, in main raise ValueError( ValueError: Output directory (/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20) already exists and is not empty. Use --oveERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 80391) of binary: /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/bin/python ent call last): File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 739, in main() File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 282, in main raise ValueError( ValueError: Output directory (/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20) already exists and is not empty. Use --overwrite_output_dir to overcome. ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 147244) of binary: /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/bin/python 11/16/2023 11:33:02 - WARNING - __main__ - Process rank: 7, device: cuda:7, n_gpu: 1distributed training: True, 16-bits training: False Traceback (most recent call last): File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 739, in main() File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 282, in main raise ValueError( ValueError: Output directory (/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20) already exists and is not empty. Use --overwrite_output_dir to overcome. 11/16/2023 11:33:02 - WARNING - __main__ - Process rank: 3, device: cuda:3, n_gpu: 1distributed training: True, 16-bits training: False Traceback (most recent call last): File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 739, in main() File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 282, in main raise ValueError( ValueError: Output directory (/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20) already exists and is not empty. Use --overwrite_output_dir to overcome. 11/16/2023 11:33:02 - WARNING - __main__ - Process rank: 4, device: cuda:4, n_gpu: 1distributed training: True, 16-bits training: False Traceback (most recent call last): File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 739, in main() File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 282, in main raise ValueError( ValueError: Output directory (/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20) already exists and is not empty. Use --overwrite_output_dir to overcome. main() File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 282, in main raise ValueError( ValueError: Output directory (/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20) already exists and is not empty. Use --overwrite_output_dir to overcome. raise ValueError( ValueError: Output directory (/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20) already exists and is not empty. Use --overwrite_output_dir to overcome. 11/16/2023 11:33:02 - WARNING - __main__ - Process rank: 2, device: cuda:2, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:33:02 - WARNING - __main__ - Process rank: 7, device: cuda:7, n_gpu: 1distributed training: True, 16-bits training: False Traceback (most recent call last): File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 739, in Traceback (most recent call last): File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 739, in main() File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 282, in main raise ValueError( ValueError: Output directory (/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20) already exists and is not empty. Use --overwrite_output_dir to overcome. main() File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 282, in main raise ValueError( ValueError: Output directory (/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20) already exists and is not empty. Use --overwrite_output_dir to overcome. 11/16/2023 11:33:02 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:33:02 - WARNING - __main__ - Process rank: 3, device: cuda:3, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:33:02 - INFO - __main__ - Training/evaluation parameters TrainingArguments( _n_gpu=1, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, bf16=True, bf16_full_eval=True, data_seed=None, dataloader_drop_last=False, dataloader_num_workers=8, dataloader_pin_memory=True, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=72000, debug=[], deepspeed=train/deepspeed_config_zero2.json, disable_tqdm=False, do_eval=False, do_predict=False, do_train=True, eval_accumulation_steps=None, eval_delay=0, eval_steps=None, evaluation_strategy=no, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'fsdp_min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=None, full_determinism=False, gradient_accumulation_steps=1, gradient_checkpointing=True, greater_is_better=None, group_by_length=False, half_precision_backend=auto, hub_model_id=None, hub_private_repo=False, hub_strategy=every_save, hub_token=, ignore_data_skip=False, include_inputs_for_metrics=False, jit_mode_eval=False, label_names=None, label_smoothing_factor=0.0, learning_rate=2e-05, length_column_name=length, load_best_model_at_end=False, local_rank=0, log_level=passive, log_level_replica=warning, log_on_each_node=True, logging_dir=/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20/runs/Nov16_11-31-40_ts-cbba87c5e7504a249f5127103d9ce40f-worker-2, logging_first_step=False, logging_nan_inf_filter=True, logging_steps=10, logging_strategy=steps, lr_scheduler_type=cosine, max_grad_norm=1.0, max_steps=-1, metric_for_best_model=None, mp_parameters=, no_cuda=False, num_train_epochs=3.0, optim=adamw_hf, optim_args=None, output_dir=/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20, overwrite_output_dir=False, past_index=-1, per_device_eval_batch_size=2, per_device_train_batch_size=48, prediction_loss_only=False, push_to_hub=False, push_to_hub_model_id=None, push_to_hub_organization=None, push_to_hub_token=, ray_scope=last, remove_unused_columns=True, report_to=['tensorboard'], resume_from_checkpoint=None, run_name=/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20, save_on_each_node=False, save_steps=500, save_strategy=steps, save_total_limit=20, seed=34, sharded_ddp=[], skip_memory_metrics=True, tf32=None, torch_compile=False, torch_compile_backend=None, torch_compile_mode=None, torchdynamo=None, tpu_metrics_debug=False, tpu_num_cores=None, use_ipex=False, use_legacy_prediction_loop=False, use_mps_device=False, warmup_ratio=0.03, warmup_steps=0, weight_decay=0.0, xpu_backend=None, ) Traceback (most recent call last): File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 739, in Traceback (most recent call last): File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 739, in main() File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 282, in main main() File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 282, in main raise ValueError( ValueError: Output directory (/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20) already exists and is not empty. Use --overwrite_output_dir to overcome. raise ValueError( ValueError: Output directory (/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20) already exists and is not empty. Use --overwrite_output_dir to overcome. ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 52636) of binary: /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/bin/python Traceback (most recent call last): File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/bin/torchrun", line 8, in sys.exit(main()) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper return f(*args, **kwargs) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/distributed/run.py", line 762, in main run(args) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/distributed/run.py", line 753, in run elastic_launch( File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2023-11-16_11:33:04 host : ts-cbba87c5e7504a249f5127103d9ce40f-worker-2 rank : 25 (local_rank: 1) exitcode : 1 (pid: 52637) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [2]: time : 2023-11-16_11:33:04 host : ts-cbba87c5e7504a249f5127103d9ce40f-worker-2 rank : 26 (local_rank: 2) exitcode : 1 (pid: 52638) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [3]: time : 2023-11-16_11:33:04 host : ts-cbba87c5e7504a249f5127103d9ce40f-worker-2 rank : 27 (local_rank: 3) exitcode : 1 (pid: 52639) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [4]: time : 2023-11-16_11:33:04 host : ts-cbba87c5e7504a249f5127103d9ce40f-worker-2 rank : 28 (local_rank: 4) exitcode : 1 (pid: 52640) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [5]: time : 2023-11-16_11:33:04 host : ts-cbba87c5e7504a249f5127103d9ce40f-worker-2 rank : 29 (local_rank: 5) exitcode : 1 (pid: 52641) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [6]: time : 2023-11-16_11:33:04 host : ts-cbba87c5e7504a249f5127103d9ce40f-worker-2 rank : 30 (local_rank: 6) exitcode : 1 (pid: 52642) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [7]: time : 2023-11-16_11:33:04 host : ts-cbba87c5e7504a249f5127103d9ce40f-worker-2 rank : 31 (local_rank: 7) exitcode : 1 (pid: 52643) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2023-11-16_11:33:04 host : ts-cbba87c5e7504a249f5127103d9ce40f-worker-2 rank : 24 (local_rank: 0) exitcode : 1 (pid: 52636) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ n-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20/runs/Nov16_11-31-40_ts-cbba87c5e7504a249f5127103d9ce40f-worker-0, logging_first_step=False, logging_nan_inf_filter=True, logging_steps=10, logging_strategy=steps, lr_scheduler_type=cosine, max_grad_norm=1.0, max_steps=-1, metric_for_best_model=None, mp_parameters=, no_cuda=False, num_train_epochs=3.0, optim=adamw_hf, optim_args=None, output_dir=/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20, overwrite_output_dir=False, past_index=-1, per_device_eval_batch_size=2, per_device_train_batch_size=48, prediction_loss_only=False, push_to_hub=False, push_to_hub_model_id=None, push_to_hub_organization=None, push_to_hub_token=, ray_scope=last, remove_unused_columns=True, report_to=['tensorboard'], resume_from_checkpoint=None, run_name=/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20, save_on_each_node=False, save_steps=500, save_strategy=steps, save_total_limit=20, seed=34, sharded_ddp=[], skip_memory_metrics=True, tf32=None, torch_compile=False, torch_compile_backend=None, torch_compile_mode=None, torchdynamo=None, tpu_metrics_debug=False, tpu_num_cores=None, use_ipex=False, use_legacy_prediction_loop=False, use_mps_device=False, warmup_ratio=0.03, warmup_steps=0, weight_decay=0.0, xpu_backend=None, ) main() File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 282, in main Traceback (most recent call last): File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 739, in raise ValueError( ValueError: Output directory (/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20) already exists and is not empty. Use --overwrite_output_dir to overcome. main() File "transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py", line 282, in main raise ValueError( ValueError: Output directory (/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20) already exists and is not empty. Use --overwrite_output_dir to overcome. Traceback (most recent call last): File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/bin/torchrun", line 8, in sys.exit(main()) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper return f(*args, **kwargs) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/distributed/run.py", line 762, in main run(args) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/distributed/run.py", line 753, in run elastic_launch( File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ transformers/examples/pytorch/language-modeling/run_clm_llms_splittrain.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2023-11-16_11:32:04 host : ts-cbba87c5e7504a249f5127103d9ce40f-worker-1 rank : 17 (local_rank: 1) exitcode : 1 (pid: 80392) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [2]: time : 2023-11-16_11:32:04 host : ts-cbba87c5e7504a249f5127103d9ce40f-worker-1 rank : 18 (local_rank: 2) exitcode : 1 (pid: 80393) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [3]: time : 2023-11-16_11:32:04 host : ts-cbba87c5e7504a249f5127103d9ce40f-worker-1 rank : 19 (local_rank: 3) exitcode : 1 (pid: 80394) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [4]: time : 2023-11-16_11:32:04 host : ts-cbba87c5e7504a249f5127103d9ce40f-worker-1 rank : 20 (local_rank: 4) exitcode : 1 (pid: 80395) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [5]: time : 2023-11-16_11:32:04 host : ts-cbba87c5e7504a249f5127103d9ce40f-worker-1 rank : 21 (local_rank: 5) exitcode : 1 (pid: 80396) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [6]: time : 2023-11-16_11:32:04 host : ts-cbba87c5e7504a249f5127103d9ce40f-worker-1 rank : 22 (local_rank: 6) exitcode : 1 (pid: 80397) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [7]: time : 2023-11-16_11:32:04 host : ts-cbba87c5e7504a249f5127103d9ce40f-worker-1 rank : 23 (local_rank: 7) exitcode : 1 (pid: 80398) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2023-11-16_11:32:04 host : ts-cbba87c5e7504a249f5127103d9ce40f-worker-1 rank : 16 (local_rank: 0) exitcode : 1 (pid: 80391) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ pid: 72154) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** [2023-11-16 11:38:01,347] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:38:01,379] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:38:01,606] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:38:01,649] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:38:01,694] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:38:01,740] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:38:01,756] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:38:01,758] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:38:04,812] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:38:04,812] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:38:04,893] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:38:04,951] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:38:04,992] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:38:05,010] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:38:05,016] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:38:05,018] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2023-11-16 11:38:18,378] [INFO] [comm.py:637:init_distributed] cdb=None [2023-11-16 11:38:19,354] [INFO] [comm.py:637:init_distributed] cdb=None [2023-11-16 11:38:19,655] [INFO] [comm.py:637:init_distributed] cdb=None [2023-11-16 11:38:19,829] [INFO] [comm.py:637:init_distributed] cdb=None [2023-11-16 11:38:19,879] [INFO] [comm.py:637:init_distributed] cdb=None [2023-11-16 11:38:19,984] [INFO] [comm.py:637:init_distributed] cdb=None [2023-11-16 11:38:20,063] [INFO] [comm.py:637:init_distributed] cdb=None [2023-11-16 11:38:20,087] [INFO] [comm.py:637:init_distributed] cdb=None [2023-11-16 11:38:20,101] [INFO] [comm.py:637:init_distributed] cdb=None 11/16/2023 11:38:23 - WARNING - __main__ - Process rank: 2, device: cuda:2, n_gpu: 1distributed training: True, 16-bits training: False /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( 11/16/2023 11:38:23 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:38:23 - INFO - __main__ - Training/eva11/16/2023 11:38:23 - WARNING - __main__ - Process rank: 5, device: cuda:5, n_gpu: 1distributed training: True, 16-bits training: False /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( 11/16/2023 11:38:23 - WARNING - __main__ - Process rank: 6, device: cuda:6, n_gpu: 1distributed training: True, 16-bits training: False /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( 11/16/2023 11:38:23 - WARNING - __main__ - Process rank: 3, device: cuda:3, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:38:23 - WARNING - __main__ - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( 11/16/2023 11:38:23 - WARNING - __main__ - Process rank: 4, device: cuda:4, n_gpu: 1distributed training: True, 16-bits training: False /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( 11/16/2023 11:38:23 - WARNING - __main__ - Process rank: 7, device: cuda:7, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:38:23 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:38:23 - INFO - __main__ - Training/evaluation parameters TrainingArguments( _n_gpu=1, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, bf16=True, bf16_full_eval=True, data_seed=None, dataloader_drop_last=False, dataloader_num_workers=8, dataloader_pin_memory=True, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=72000, debug=[], deepspeed=train/deepspeed_config_zero2.json, disable_tqdm=False, do_eval=False, do_predict=False, do_train=True, eval_accumulation_steps=None, eval_delay=0, eval_steps=None, evaluation_strategy=no, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'fsdp_min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=None, full_determinism=False, gradient_accumulation_steps=1, gradient_checkpointing=True, greater_is_better=None, group_by_length=False, half_precision_backend=auto, hub_model_id=None, hub_private_repo=False, hub_strategy=every_save, hub_token=, ignore_data_skip=False, include_inputs_for_metrics=False, jit_mode_eval=False, label_names=None, label_smoothing_factor=0.0, learning_rate=2e-05, length_column_name=length, load_best_model_at_end=False, local_rank=0, log_level=passive, log_level_replica=warning, log_on_each_node=True, logging_dir=/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20/runs/Nov16_11-38-00_ts-cbba87c5e7504a249f5127103d9ce40f-launcher, logging_first_step=False, logging_nan_inf_filter=True, logging_steps=10, logging_strategy=steps, lr_scheduler_type=cosine, max_grad_norm=1.0, max_steps=-1, metric_for_best_model=None, mp_parameters=, no_cuda=False, num_train_epochs=3.0, optim=adamw_hf, optim_args=None, output_dir=/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20, overwrite_output_dir=True, past_index=-1, 11/16/2023 11:38:23 - WARNING - __main__ - Process rank: 6, device: cuda:6, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:38:23 - WARNING - __main__ - Process rank: 5, device: cuda:5, n_gpu: 1distributed training: True, 16-bits training: False /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( 11/16/2023 11:38:23 - WARNING - __main__ - Process rank: 4, device: cuda:4, n_gpu: 1distributed training: True, 16-bits training: False /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( Using custom data configuration default-757af2c3ccba6a1a 11/16/2023 11:38:24 - INFO - datasets.builder - Using custom data configuration default-757af2c3ccba6a1a Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/packaged_modules/json 11/16/2023 11:38:24 - INFO - datasets.info - Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/packaged_modules/json Overwrite dataset info from restored data version if exists. 11/16/2023 11:38:24 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists. Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 11/16/2023 11:38:24 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 Found cached dataset json (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96) 11/16/2023 11:38:24 - INFO - datasets.builder - Found cached dataset json (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96) Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 11/16/2023 11:38:24 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 der - Found cached dataset json (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96) Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 11/16/2023 11:38:24 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 11/16/2023 11:39:23 - WARNING - __main__ - Process rank: 4, device: cuda:4, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:39:23 - WARNING - __main__ - Process rank: 2, device: cuda:2, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:39:23 - WARNING - __main__ - Process rank: 3, device: cuda:3, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:39:23 - WARNING - __main__ - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:39:23 - WARNING - __main__ - Process rank: 5, device: cuda:5, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:39:23 - WARNING - __main__ - Process rank: 6, device: cuda:6, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:39:23 - WARNING - __main__ - Process rank: 7, device: cuda:7, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:39:23 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:39:23 - INFO - __main__ - Training/evaluation parameters TrainingArguments( _n_gpu=1, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, bf16=True, bf16_full_eval=True, data_seed=None, dataloader_drop_last=False, dataloader_num_workers=8, dataloader_pin_memory=True, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=72000, debug=[], deepspeed=train/deepspeed_config_zero2.json, disable_tqdm=False, do_eval=False, do_predict=False, do_train=True, eval_accumulation_steps=None, eval_delay=0, eval_steps=None, evaluation_strategy=no, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'fsdp_min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=None, full_determinism=False, gradient_accumulation_steps=1, gradient_checkpointing=True, greater_is_better=None, group_by_length=False, half_precision_backend=auto, hub_model_id=None, hub_private_repo=False, hub_strategy=every_save, hub_token=, ignore_data_skip=False, include_inputs_for_metrics=False, jit_mode_eval=False, label_names=None, label_smoothing_factor=0.0, learning_rate=2e-05, length_column_name=length, load_best_model_at_end=False, local_rank=0, log_level=passive, log_level_replica=warning, log_on_each_node=True, logging_dir=/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20/runs/Nov16_11-38-00_ts-cbba87c5e7504a249f5127103d9ce40f-worker-0, logging_first_step=False, logging_nan_inf_filter=True, logging_steps=10, logging_strategy=steps, lr_scheduler_type=cosine, max_grad_norm=1.0, max_steps=-1, metric_for_best_model=None, mp_parameters=, no_cuda=False, num_train_epochs=3.0, optim=adamw_hf, optim_args=None, output_dir=/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20, overwrite_output_dir=True, past_index=-1, per_device_eval_batch_size=2, per_device_train_batch_size=48, prediction_loss_only=False, push_to_hub=False, push_to_hub_model_id=None, push_to_hub_organization=None, push_to_hub_token=, ray_scope=last, remove_unused_columns=True, report_to=['tensorboard'], resume_from_checkpoint=None, run_name=/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20, save_on_each_node=False, save_steps=500, save_strategy=steps, save_total_limit=10, seed=34, sharded_ddp=[], skip_memory_metrics=True, tf32=None, torch_compile=False, torch_compile_backend=None, torch_compile_mode=None, torchdynamo=None, tpu_metrics_debug=False, tpu_num_cores=None, use_ipex=False, use_legacy_prediction11/16/2023 11:39:23 - WARNING - __main__ - Process rank: 7, device: cuda:7, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:39:23 - WARNING - __main__ - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:39:23 - WARNING - __main__ - Process rank: 6, device: cuda:6, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:39:23 - WARNING - __main__ - Process rank: 2, device: cuda:2, n_gpu: 1distributed training: True, 16-bits training: False /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( 11/16/2023 11:39:23 - WARNING - __main__ - Process rank: 4, device: cuda:4, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:39:23 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:39:23 - INFO - __main__ - Training/evaluation parameters TrainingArguments( _n_gpu=1, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, bf16=True, bf16_full_eval=True, data_seed=None, dataloader_drop_last=False, dataloader_num_workers=8, dataloader_pin_memory=True, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=72000, debug=[], deepspeed=train/deepspeed_config_zero2.json, disable_tqdm=False, do_eval=False, do_predict=False, do_train=True, eval_accumulation_steps=None, eval_delay=0, eval_steps=None, evaluation_strategy=no, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'fsdp_min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=NoUsing custom data configuration default-757af2c3ccba6a1a 11/16/2023 11:39:24 - INFO - datasets.builder - Using custom data configuration default-757af2c3ccba6a1a Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/packaged_modules/json 11/16/2023 11:39:24 - INFO - datasets.info - Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/packaged_modules/json Overwrite dataset info from restored data version if exists. 11/16/2023 11:39:24 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists. Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 11/16/2023 11:39:24 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 Found cached dataset json (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96) 11/16/2023 11:39:24 - INFO - datasets.builder - Found cached dataset json (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96) Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 11/16/2023 11:39:24 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 ebug=False, tpu_num_cores=None, use_ipex=False, use_legacy_prediction_loop=False, use_mps_device=False, warmup_ratio=0.03, warmup_steps=0, weight_decay=0.0, xpu_backend=None, ) 11/16/2023 11:39:23 - INFO - __main__ - Loading dataset from file: {'train': '/apdcephfs/share_733425/vinnylywang/jianhuipang/datasets/wmt23/wmt23-zhen/train.catalpacagpt4.newstests17to20.zh2en.hf.json', 'validation': '/apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/test/WMT22/newstest22.de2en.hf.json'} /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( 11/16/2023 11:39:23 - WARNING - __main__ - Process rank: 3, device: cuda:3, n_gpu: 1distributed training: True, 16-bits training: False 11/16/2023 11:39:23 - WARNING - __main__ - Process rank: 5, device: cuda:5, n_gpu: 1distributed training: True, 16-bits training: False /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( Using custom data configuration default-757af2c3ccba6a1a 11/16/2023 11:39:24 - INFO - datasets.builder - Using custom data configuration default-757af2c3ccba6a1a Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/packaged_modules/json 11/16/2023 11:39:24 - INFO - datasets.info - Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/packaged_modules/json Overwrite dataset info from restored data version if exists. 11/16/2023 11:39:24 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists. Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 11/16/2023 11:39:24 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 Found cached dataset json (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96) 11/16/2023 11:39:24 - INFO - datasets.builder - Found cached dataset json (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96) Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 11/16/2023 11:39:24 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 Using custom data configuration default-757af2c3ccba6a1a 11/16/2023 11:39:58 - INFO - datasets.builder - Using custom data configuration default-757af2c3ccba6a1a Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/packaged_modules/json 11/16/2023 11:39:58 - INFO - datasets.info - Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/packaged_modules/json Overwrite dataset info from restored data version if exists. 11/16/2023 11:39:58 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists. Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 11/16/2023 11:39:58 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 Found cached dataset json (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96) 11/16/2023 11:39:58 - INFO - datasets.builder - Found cached dataset json (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96) Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 11/16/2023 11:39:58 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 [INFO|configuration_utils.py:666] 2023-11-16 11:39:59,250 >> loading configuration file /apdcephfs/share_733425/vinnylywang/jianhuipang/opensourcellms/llama2/Llama-2-7b-hf/config.json [INFO|configuration_utils.py:720] 2023-11-16 11:39:59,251 >> Model config LlamaConfig { "_name_or_path": "/apdcephfs/share_733425/vinnylywang/jianhuipang/opensourcellms/llama2/Llama-2-7b-hf", "architectures": [ "LlamaForCausalLM" ], "bos_token_id": 1, "eos_token_id": 2, "hidden_act": "silu", "hidden_size": 4096, "initializer_range": 0.02, "intermediate_size": 11008, "max_position_embeddings": 4096, "model_type": "llama", "num_attention_heads": 32, "num_hidden_layers": 32, "num_key_value_heads": 32, "pad_token_id": 0, "pretraining_tp": 1, "rms_norm_eps": 1e-05, "rope_scaling": null, "tie_word_embeddings": false, "torch_dtype": "float16", "transformers_version": "4.28.0.dev0", "use_cache": true, "vocab_size": 32000 } 11/16/2023 11:39:59 - INFO - __main__ - Tokenizer_kwargs: {'cache_dir': None, 'use_fast': True, 'revision': 'main', 'use_auth_token': None} [INFO|tokenization_utils_base.py:1801] 2023-11-16 11:39:59,257 >> loading file tokenizer.model [INFO|tokenization_utils_base.py:1801] 2023-11-16 11:39:59,257 >> loading file added_tokens.json [INFO|tokenization_utils_base.py:1801] 2023-11-16 11:39:59,257 >> loading file special_tokens_map.json [INFO|tokenization_utils_base.py:1801] 2023-11-16 11:39:59,257 >> loading file tokenizer_config.json 11/16/2023 11:39:59 - INFO - __main__ - Loading checkpoints in dtype: None [INFO|modeling_utils.py:2395] 2023-11-16 11:39:59,275 >> loading weights file /apdcephfs/share_733425/vinnylywang/jianhuipang/opensourcellms/llama2/Llama-2-7b-hf/pytorch_model.bin.index.json [INFO|configuration_utils.py:575] 2023-11-16 11:39:59,277 >> Generate config GenerationConfig { "_from_model_config": true, "bos_token_id": 1, "eos_token_id": 2, "pad_token_id": 0, "transformers_version": "4.28.0.dev0" } Loading checkpoint shards: 0%| | 0/2 [00:00> Using pad_token, but it is not set yet. Loading checkpoint shards: 100%|██████████| 2/2 [01:07<00:00, 30.50s/it] Loading checkpoint shards: 100%|██████████| 2/2 [01:07<00:00, 33.65s/it] [INFO|modeling_utils.py:3029] 2023-11-16 11:41:05,248 >> All model checkpoint weights were used when initializing LlamaForCausalLM. [INFO|modeling_utils.py:3037] 2023-11-16 11:41:05,248 >> All the weights of LlamaForCausalLM were initialized from the model checkpoint at /apdcephfs/share_733425/vinnylywang/jianhuipang/opensourcellms/llama2/Llama-2-7b-hf. If your task is similar to the task the model of the checkpoint was trained on, you can already use LlamaForCausalLM for predictions without further training. Loading checkpoint shards: 100%|██████████| 2/2 [01:07<00:00, 30.58s/it] Loading checkpoint shards: 100%|██████████| 2/2 [01:07<00:00, 33.54s/it] [INFO|configuration_utils.py:535] 2023-11-16 11:41:05,256 >> loading configuration file /apdcephfs/share_733425/vinnylywang/jianhuipang/opensourcellms/llama2/Llama-2-7b-hf/generation_config.json [INFO|configuration_utils.py:575] 2023-11-16 11:41:05,256 >> Generate config GenerationConfig { "bos_token_id": 1, "do_sample": true, "eos_token_id": 2, "max_length": 4096, "pad_token_id": 0, "temperature": 0.6, "top_p": 0.9, "transformers_version": "4.28.0.dev0" } [ERROR|tokenization_utils_base.py:1042] 2023-11-16 11:41:05,256 >> Using pad_token, but it is not set yet. [INFO|tokenization_utils_base.py:907] 2023-11-16 11:41:05,256 >> Assigning [PAD] to the pad_token key of the tokenizer [INFO|tokenization_utils.py:426] 2023-11-16 11:41:05,256 >> Adding [PAD] to the vocabulary [ERROR|tokenization_utils_base.py:1042] 2023-11-16 11:41:05,261 >> Using pad_token, but it is not set yet. Loading checkpoint shards: 100%|██████████| 2/2 [01:07<00:00, 30.41s/it] Loading checkpoint shards: 100%|██████████| 2/2 [01:07<00:00, 33.51s/it] [ERROR|tokenization_utils_base.py:1042] 2023-11-16 11:41:05,315 >> Using pad_token, but it is not set yet. Loading checkpoint shards: 100%|██████████| 2/2 [01:07<00:00, 30.57s/it] Loading checkpoint shards: 100%|██████████| 2/2 [01:07<00:00, 33.71s/it] [ERROR|tokenization_utils_base.py:1042] 2023-11-16 11:41:05,350 >> Using pad_token, but it is not set yet. Loading checkpoint shards: 100%|██████████| 2/2 [01:07<00:00, 30.83s/it] Loading checkpoint shards: 100%|██████████| 2/2 [01:07<00:00, 33.74s/it] [ERROR|tokenization_utils_base.py:1042] 2023-11-16 11:41:05,377 >> Using pad_token, but it is not set yet. Loading checkpoint shards: 50%|█████ | 1/2 [00:32<00:32, 32.41s/it] Loading checkpoint shards: 50%|█████ | 1/2 [00:32<00:32, 32.58s/it] Loading checkpoint shards: 50%|█████ | 1/2 [00:32<00:32, 32.61s/it] Loading checkpoint shards: 50%|█████ | 1/2 [00:32<00:32, 32.57s/it] Loading checkpoint shards: 50%|█████ | 1/2 [00:33<00:33, 33.26s/it] Loading checkpoint shards: 50%|█████ | 1/2 [00:33<00:33, 33.30s/it] Loading checkpoint shards: 50%|█████ | 1/2 [00:33<00:33, 33.77s/it] Loading checkpoint shards: 50%|█████ | 1/2 [00:33<00:33, 33.72s/it]e at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00000_of_00016.arrow Process #1 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00001_of_00016.arrow 11/16/2023 11:41:08 - INFO - datasets.arrow_dataset - Process #1 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00001_of_00016.arrow Process #2 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00002_of_00016.arrow 11/16/2023 11:41:08 - INFO - datasets.arrow_dataset - Process #2 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00002_of_00016.arrow Process #3 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00003_of_00016.arrow 11/16/2023 11:41:08 - INFO - datasets.arrow_dataset - Process #3 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00003_of_00016.arrow Process #4 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00004_of_00016.arrow 11/16/2023 11:41:08 - INFO - datasets.arrow_dataset - Process #4 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00004_of_00016.arrow Process #5 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00005_of_00016.arrow 11/16/2023 11:41:08 - INFO - datasets.arrow_dataset - Process #5 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00005_of_00016.arrow Process #6 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00006_of_00016.arrow 11/16/2023 11:41:08 - INFO - datasets.arrow_dataset - Process #6 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00006_of_00016.arrow Process #7 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00007_of_00016.arrow 11/16/2023 11:41:08 - INFO - datasets.arrow_dataset - Process #7 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00007_of_00016.arrow Process #8 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00008_of_00016.arrow 11/16/2023 11:41:08 - INFO - datasets.arrow_dataset - Process #8 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00008_of_00016.arrow Process #9 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00009_of_00016.arrow 11/16/2023 11:41:08 - INFO - datasets.arrow_dataset - Process #9 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00009_of_00016.arrow Process #10 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00010_of_00016.arrow 11/16/2023 11:41:08 - INFO - datasets.arrow_dataset - Process #10 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00010_of_00016.arrow Process #11 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00011_of_00016.arrow 11/16/2023 11:41:08 - INFO - datasets.arrow_dataset - Process #11 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00011_of_00016.arrow Process #12 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00012_of_00016.arrow 11/16/2023 11:41:08 - INFO - datasets.arrow_dataset - Process #12 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00012_of_00016.arrow Process #13 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00013_of_00016.arrow 11/16/2023 11:41:08 - INFO - datasets.arrow_dataset - Process #13 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00013_of_00016.arrow Process #14 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00014_of_00016.arrow 11/16/2023 11:41:08 - INFO - datasets.arrow_dataset - Process #14 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00014_of_00016.arrow Process #15 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00015_of_00016.arrow 11/16/2023 11:41:08 - INFO - datasets.arrow_dataset - Process #15 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00015_of_00016.arrow Spawning 16 processes 11/16/2023 11:41:14 - INFO - datasets.arrow_dataset - Spawning 16 processes Tokenize with padding (num_proc=16): 0%| | 0/75399 [00:00> Generate config GenerationConfig { "bos_token_id": 1, "do_sample": true, "eos_token_id": 2, "max_length": 4096, "pad_token_id": 0, "temperature": 0.6, "top_p": 0.9, "transformers_version": "4.28.0.dev0" } [ERROR|tokenization_utils_base.py:1042] 2023-11-16 11:41:19,571 >> Using pad_token, but it is not set yet. [INFO|tokenization_utils_base.py:907] 2023-11-16 11:41:19,571 >> Assigning [PAD] to the pad_token key of the tokenizer [INFO|tokenization_utils.py:426] 2023-11-16 11:41:19,571 >> Adding [PAD] to the vocabulary [ERROR|tokenization_utils_base.py:1042] 2023-11-16 11:41:19,573 >> Using pad_token, but it is not set yet. Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 19.76s/it] Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.58s/it] [ERROR|tokenization_utils_base.py:1042] 2023-11-16 11:41:19,594 >> Using pad_token, but it is not set yet. Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 19.84s/it] Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.77s/it] [ERROR|tokenization_utils_base.py:1042] 2023-11-16 11:41:19,980 >> Using pad_token, but it is not set yet. Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 19.67s/it] Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.66s/it] [ERROR|tokenization_utils_base.py:1042] 2023-11-16 11:41:20,075 >> Using pad_token, but it is not set yet. Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 19.94s/it] Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.83s/it] [ERROR|tokenization_utils_base.py:1042] 2023-11-16 11:41:20,837 >> Using pad_token, but it is not set yet. Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.08s/it] Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.93s/it] [ERROR|tokenization_utils_base.py:1042] 2023-11-16 11:41:20,871 >> Using pad_token, but it is not set yet. Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 19.98s/it] Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.88s/it] [ERROR|tokenization_utils_base.py:1042] 2023-11-16 11:41:20,887 >> Using pad_token, but it is not set yet. [INFO|tokenization_utils_base.py:907] 2023-11-16 11:41:22,619 >> Assigning to the eos_token key of the tokenizer [INFO|tokenization_utils_base.py:907] 2023-11-16 11:41:22,619 >> Assigning to the bos_token key of the tokenizer [INFO|tokenization_utils_base.py:907] 2023-11-16 11:41:22,619 >> Assigning to the unk_token key of the tokenizer Process #0 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f Tokenize with padding (num_proc=16): 76%|███████▌ | 57000/75399 [00:08<00:01, 12586.88 examples/s] Tokenize with padding (num_proc=16): 79%|███████▉ | 59713/75399 [00:09<00:02, 6145.87 examples/s] Tokenize with padding (num_proc=16): 83%|████████▎ | 62426/75399 [00:09<00:01, 7732.48 examples/s] Tokenize with padding (num_proc=16): 86%|████████▌ | 64564/75399 [00:10<00:01, 8820.68 examples/s] Tokenize with padding (num_proc=16): 90%|█████████ | 68126/75399 [00:10<00:00, 11581.86 examples/s] Tokenize with padding (num_proc=16): 94%|█████████▍| 70839/75399 [00:11<00:00, 6095.72 examples/s] Tokenize with padding (num_proc=16): 97%|█████████▋| 72975/75399 [00:12<00:00, 4382.05 examples/s] Tokenize with padding (num_proc=16): 99%|█████████▉| 74687/75399 [00:13<00:00, 2939.91 examples/s]c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00002_of_00016.arrow 11/16/2023 11:41:22 - INFO - datasets.arrow_dataset - Process #2 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00002_of_00016.arrow Process #3 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00003_of_00016.arrow 11/16/2023 11:41:22 - INFO - datasets.arrow_dataset - Process #3 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00003_of_00016.arrow Process #4 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_0[INFO|tokenization_utils_base.py:907] 2023-11-16 11:41:24,498 >> Assigning to the eos_token key of the tokenizer [INFO|tokenization_utils_base.py:907] 2023-11-16 11:41:24,499 >> Assigning to the bos_token key of the tokenizer [INFO|tokenization_utils_base.py:907] 2023-11-16 11:41:24,499 >> Assigning to the unk_token key of the tokenizer Process #0 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00000_of_00016.arrow 11/16/2023 11:41:24 - INFO - datasets.arrow_dataset - Process #0 will write at /apdcephfs/share_733 Tokenize with padding (num_proc=16): 80%|████████ | 60426/75399 [00:09<00:02, 6755.24 examples/s] Tokenize with padding (num_proc=16): 84%|████████▎ | 63139/75399 [00:09<00:01, 8496.45 examples/s] Tokenize with padding (num_proc=16): 88%|████████▊ | 66702/75399 [00:09<00:00, 11852.14 examples/s] Tokenize with padding (num_proc=16): 93%|█████████▎| 69839/75399 [00:10<00:00, 10523.70 examples/s] Tokenize with padding (num_proc=16): 96%|█████████▌| 72263/75399 [00:11<00:00, 5436.55 examples/s] ba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00001_of_00016.arrow Process #2 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00002_of_00016.arrow 11/16/2023 11:41:24 - INFO - datasets.arrow_dataset - Process #2 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00002_of_00016.arrow Process #3 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00003_of_00016.arrow 11/16/2023 11:41:24 - INFO - datasets.arrow_dataset - Process #3 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00003_of_00016.arrow Process #4 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00004_of_00016.arrow 11/16/2023 11:41:24 - INFO - datasets.arrow_dataset - Process #4 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00004_of_00016.arrow Process #5 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00005_of_00016.arrow 11/16/2023 11:41:24 - INFO - datasets.arrow_dataset - Process #5 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00005_of_00016.arrow Process #6 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00006_of_00016.arrow 11/16/2023 11:41:24 - INFO - datasets.arrow_dataset - Process #6 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00006_of_00016.arrow Process #7 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00007_of_00016.arrow 11/16/2023 11:41:24 - INFO - datasets.arrow_dataset - Process #7 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00007_of_00016.arrow Process #8 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00008_of_00016.arrow 11/16/2023 11:41:24 - INFO - datasets.arrow_dataset - Process #8 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00008_of_00016.arrow Process #9 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00009_of_00016.arrow 11/16/2023 11:41:24 - INFO - datasets.arrow_dataset - Process #9 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00009_of_00016.arrow Process #10 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-398076e6bdcd1f27_00010_of_00016.arrow 11/16/2023 11:41:24 - INFO - datasets.arrow_dataset - Process #10 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-3980Spawning 16 processes 11/16/2023 11:41:27 - INFO - datasets.arrow_dataset - Spawning 16 processes Tokenize with padding (num_proc=16): 0%| | 0/75399 [00:00 ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54968:54968 [7] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5ets-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:74474 [4] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:74475 [5] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:74474 [4] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:74473 [3] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:74476 [6] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:74477 [7] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:74473 [3] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:74472 [2] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:74476 [6] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:74477 [7] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:74471 [1] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:74472 [2] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:74471 [1] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:74474 [4] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:74473 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:74476 [6] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75622 [ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54965:56069 [4] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.57.242<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54965:56069 [4] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54963:56070 [2] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.57.242<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54963:56070 [2] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54967:56071 [6] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.57.242<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54967:56071 [6] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54964:56074 [3] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.57.242<0> ts-cbba87c5e7504a249f51271ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75624 [3] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75624 [3] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75625 [6] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75625 [6] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:75629 [1] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:75629 [1] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75627 [7] NCCL Tokenize with padding (num_proc=16): 31%|███ | 23560/75399 [00:14<00:41, 1235.47 examples/s] Tokenize with padding (num_proc=16): 31%|███ | 23560/75399 [00:14<00:32, 1577.17 examples/s] 5e7504a249f5127103d9ce40f-worker-0:74477:75627 [7] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75628 [2] NCCL INFO Using network IB rrow_dataset - Caching processed dataset at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-84cf8a8bd09bf187_00008_of_00016.arrow Caching processed dataset at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-84cf8a8bd09bf187_00010_of_00016.arrow 11/16/2023 11:41:38 - INFO - datasets.arrow_dataset - Caching processed dataset at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-84cf8a8bd09bf187_00010_of_00016.arrow Caching processed dataset at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-84cf8a8bd09bf187_00013_of_00016.arrow 11/16/2023 11:41:38 - INFO - datasets.arrow_dataset - Caching processed dataset at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-84cf8a8bd09bf187_00013_of_00016.arrow Caching processed dataset at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-84cf8a8bd09bf187_00011_of_00016.arrow 11/16/2023 11:41:38 - INFO - datasets.arrow_dataset - Caching processed dataset at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-84cf8a8bd09bf187_00011_of_00016.arrow Tokenize with padding (num_proc=16): 69%|██████▉ | 1364/1984 [00:00<00:00, 2150.59 examples/s]Caching processed dataset at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-84cf8a8bd09bf187_00012_of_00016.arrow 11/16/2023 11:41:38 - INFO - datasets.arrow_dataset - Caching processed dataset at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-84cf8a8bd09bf187_00012_of_00016.arrow Caching processed dataset at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-84cf8a8bd09bf187_00015_of_00016.arrow Caching processed dataset at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-84cf8a8bd09bf187_00014_of_00016.arrow 11/16/2023 11:41:38 - INFO - datasets.arrow_dataset - Caching processed dataset at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-84cf8a8bd09bf187_00014_of_00016.arrow 11/16/2023 11:41:38 - INFO - datasets.arrow_dataset - Caching processed dataset at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache/datasets/json/default-757af2c3ccba6a1a/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-84cf8a8bd09bf187_00015_of_00016.arrow Tokenize with padding (num_proc=16): 100%|██████████| 1984/1984 [00:01<00:00, 1351.08 examples/s] Concatenating 16 shards 11/16/2023 11:41:39 - INFO - datasets.arrow_dataset - Concatenating 16 shards ts-cbba87c5e7504a249f512710ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82589:82589 [7] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82585:82585 [3] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82589:82589 [7] NCCL INFO Bootstrap : Using eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82586:82586 [4] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82585:82585 [3] NCCL INFO Bootstrap : Using eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82583:82583 [1] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82586:82586 [4] NCCL INFO Bootstrap : Using eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82583:82583 [1] NCCL INFO Bootstrap : Using eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82584:82584 [2] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82582:82582 [0] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82584:82584 [2] NCCL INFO Bootstrap : Using eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82582:82582 [0] NCCL INFO Bootstrap : Using eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82587:82587 [5] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82587:82587 [5] NCCL INFO Bootstrap : Using eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82588:82588 [6] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82588:82588 [6] NCCL INFO Bootstrap : Using eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82589:82589 [7] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82586:82586 [4] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82584:82584 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82582:82582 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82585:82585 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82583:82583 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82587:82587 [5] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82588:82588 [6] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82586:83808 [4] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82586:83808 [4] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82589:83807 [7] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82589:83807 [7] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82584:83809 [2] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82585:83811 [3] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82585:83811 [3] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82584:83809 [2] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82582:83810 [0] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82582:83810 [0] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82583:83812 [1] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82583:83812 [1] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82587:83813 [5] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82587:83813 [5] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82588:83814 [6] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82588:83814 [6] NCCL INFO Using network IB 7103d9ce40f-launcher:149856:151277 [7] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149853:151276 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffff0000,00000000,ffffffff,ffff0000,00000000 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149856:151277 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffff0000,00000000,ffffffff,ffff0000,00000000 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149855:151275 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffff0000,00000000,ffffffff,ffff0000,00000000 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149849:151270 [0] NCCL INFO Setting affinity for GPU 0 to ffff,ffffffff,00000000,0000ffff,ffffffff ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149852:151272 [3] NCCL INFO Setting affinity for GPU 3 to ffff,ffffffff,00000000,0000ffff,ffffffff ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149851:151278 [2] NCCL INFO Setting affinity for GPU 2 to ffff,ffffffff,00000000,0000ffff,ffffffff ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149850:151274 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ffffffff,00000000,0000ffff,ffffffff ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149854:151273 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffff0000,00000000,ffffffff,ffff0000,00000000 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149850:151274 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149849:151270 [0] NCCL INFO Channel 00/02 : 0 3 10 15 14 13 12 9 8 11 18 23 22 21 20 17 16 19 26 31 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149856:151277 [7] NCCL INFO Trees [0] 0/-1/-1->7->6 [1] 0/-1/-1->7->6 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149849:151270 [0] NCCL INFO Channel 01/02 : 0 3 10 15 14 13 12 9 8 11 18 23 22 21 20 17 16 19 26 31 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149855:151275 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149854:151273 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149851:151278 [2] NCCL INFO Trees [0] 3/18/-1->2->-1 [1] 3/-1/-1->2->10 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149849:151270 [0] NCCL INFO Trees [0] 1/-1/-1->0->7 [1] 1/-1/-1->0->7 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149852:151272 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149853:151276 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149853:151276 [4] NCCL INFO Channel 00/0 : 4[93000] -> 1[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149849:151270 [0] NCCL INFO Channel 00/0 : 0[e000] -> 3[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149851:151278 [2] NCCL INFO Channel 00/0 : 2[4b000] -> 7[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149853:151276 [4] NCCL INFO Channel 01/0 : 4[93000] -> 1[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149849:151270 [0] NCCL INFO Channel 01/0 : 0[e000] -> 3[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149854:151273 [5] NCCL INFO Channel 00/0 : 5[99000] -> 4[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149851:151278 [2] NCCL INFO Channel 01/0 : 2[4b000] -> 7[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149855:151275 [6] NCCL INFO Channel 00/0 : 6[cb000] -> 5[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149854:151273 [5] NCCL INFO Channel 01/0 : 5[99000] -> 4[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149855:151275 [6] NCCL INFO Channel 01/0 : 6[cb000] -> 5[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149850:151274 [1] NCCL INFO Channel 00/0 : 1[13000] -> 0[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149856:151277 [7] NCCL INFO Channel 00/0 : 7[d0000] -> 6[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149852:151272 [3] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149850:151274 [1] NCCL INFO Channel 01/0 : 1[13000] -> 0[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149856:151277 [7] NCCL INFO Channel 01/0 : 7[d0000] -> 6[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149849:151270 [0] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149849:151270 [0] NCCL INFO Channel 00/0 : 0[e000] -> 1[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149853:151276 [4] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149850:151274 [1] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149856:151277 [7] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149849:151270 [0] NCCL INFO Channel 01/0 : 0[e000] -> 1[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149855:151275 [6] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149854:151273 [5] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149853:151276 [4] NCCL INFO Channel 00/0 : 4[93000] -> 5[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149853:151276 [4] NCCL INFO Channel 01/0 : 4[93000] -> 5[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149849:151270 [0] NCCL INFO Channel 00/0 : 0[e000] -> 7[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149850:151274 [1] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149850:151274 [1] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149850:151274 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149855:151275 [6] NCCL INFO Channel 00/0 : 6[cb000] -> 7[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149854:151273 [5] NCCL INFO Channel 00/0 : 5[99000] -> 6[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149849:151270 [0] NCCL INFO Channel 01/0 : 0[e000] -> 7[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149855:151275 [6] NCCL INFO Channel 01/0 : 6[cb000] -> 7[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149854:151273 [5] NCCL INFO Channel 01/0 : 5[99000] -> 6[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149854:151273 [5] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149854:151273 [5] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149854:151273 [5] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149855:151275 [6] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149855:151275 [6] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149855:151275 [6] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149856:151277 [7] NCCL INFO Channel 00/0 : 7[d0000] -> 0[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149856:151277 [7] NCCL INFO Channel 01/0 : 7[d0000] -> 0[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149849:151270 [0] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149849:151270 [0] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149849:151270 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149856:151277 [7] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149856:151277 [7] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149856:151277 [7] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149851:151278 [2] NCCL INFO Channel 00/0 : 27[51000] -> 2[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149852:151272 [3] NCCL INFO Channel 00/0 : 3[51000] -> 10[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149850:151274 [1] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149854:151273 [5] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149855:151275 [6] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149856:151277 [7] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149849:151270 [0] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149851:151278 [2] NCCL INFO Channel 01/0 : 27[51000] -> 2[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149852:151272 [3] NCCL INFO Channel 01/0 : 3[51000] -> 10[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149851:151310 [2] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149851:151310 [2] NCCL INFO NCCL_IB_SL set by environment to 3. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149852:151309 [3] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149852:151272 [3] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149852:151272 [3] NCCL INFO Channel 00/0 : 3[51000] -> 4[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149852:151272 [3] NCCL INFO Channel 01/0 : 3[51000] -> 4[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149851:151278 [2] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149851:151278 [2] NCCL INFO Channel 00/0 : 2[4b000] -> 3[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149851:151278 [2] NCCL INFO Channel 01/0 : 2[4b000] -> 3[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149853:151276 [4] NCCL INFO Channel 00/0 : 4[93000] -> 3[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149851:151278 [2] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149853:151276 [4] NCCL INFO Channel 01/0 : 4[93000] -> 3[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149852:151272 [3] NCCL INFO Channel 00/0 : 3[51000] -> 2[4b000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149852:151272 [3] NCCL INFO Channel 01/0 : 3[51000] -> 2[4b000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149853:151276 [4] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149853:151276 [4] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149853:151276 [4] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149851:151278 [2] NCCL INFO Channel 01/0 : 2[4b000] -> 10[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149853:151276 [4] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ->20->19 [1] 21/-1/-1->20->19 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82585:83811 [3] NCCL INFO Trees [0] 20/10/-1->19->18 [1] 20/-1/-1->19->18 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82584:83809 [2] NCCL INFO Trees [0] 19/26/-1->18->2 [1] 19/-1/-1->18->11 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82583:83812 [1] NCCL INFO Trees [0] -1/-1/-1->17->16 [1] -1/-1/-1->17->16 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82582:83810 [0] NCCL INFO Trees [0] 17/-1/-1->16->23 [1] 17/-1/-ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75623 [4] NCCL INFO Channel 00/0 : 12[93000] -> 9[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75639 [0] NCCL INFO Channel 00/0 : 8[e000] -> 11[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75628 [2] NCCL INFO Channel 00/0 : 10[4b000] -> 15[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75623 [4] NCCL INFO Channel 01/0 : 12[93000] -> 9[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75639 [0] NCCL INFO Channel 01/0 : 8[e000] -> 11[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75628 [2] NCCL INFO Channel 01/0 : 10[4b000] -> 15[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75622 [5] NCCL INFO Channel 00/0 : 13[99000] -> 12[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75625 [6] NCCL INFO Channel 00/0 : 14[cb000] -> 13[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75622 [5] NCCL INFO Channel 01/0 : 13[99000] -> 12[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75625 [6] NCCL INFO Channel 01/0 : 14[cb000] -> 13[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:75629 [1] NCCL INFO Channel 00/0 : 9[13000] -> 8[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75624 [3] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75627 [7] NCCL INFO Channel 00/0 : 15[d0000] -> 14[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:75629 [1] NCCL INFO Channel 01/0 : 9[13000] -> 8[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75627 [7] NCCL INFO Channel 01/0 : 15[d0000] -> 14[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75623 [4] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75639 [0] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75639 [0] NCCL INFO Channel 00/0 : 8[e000] -> 9[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:75629 [1] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75627 [7] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75639 [0] NCCL INFO Channel 01/0 : 8[e000] -> 9[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75622 [5] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75625 [6] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75623 [4] NCCL INFO Channel 00/0 : 12[93000] -> 13[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75623 [4] NCCL INFO Channel 01/0 : 12[93000] -> 13[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75622 [5] NCCL INFO Channel 00/0 : 13[99000] -> 14[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75639 [0] NCCL INFO Channel 00/0 : 8[e000] -> 15[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:75629 [1] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:75629 [1] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:75629 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75625 [6] NCCL INFO Channel 00/0 : 14[cb000] -> 15[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75622 [5] NCCL INFO Channel 01/0 : 13[99000] -> 14[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75639 [0] NCCL INFO Channel 01/0 : 8[e000] -> 15[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75625 [6] NCCL INFO Channel 01/0 : 14[cb000] -> 15[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75622 [5] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75622 [5] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75622 [5] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75625 [6] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75625 [6] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75625 [6] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75627 [7] NCCL INFO Channel 00/0 : 15[d0000] -> 8[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75627 [7] NCCL INFO Channel 01/0 : 15[d0000] -> 8[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75624 [3] NCCL INFO Channel 00/0 : 11[51000] -> 18[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75628 [2] NCCL INFO Channel 00/0 : 3[51000] -> 10[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75627 [7] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75627 [7] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75627 [7] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75639 [0] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75639 [0] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75639 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:75629 [1] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75622 [5] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75625 [6] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75627 [7] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75624 [3] NCCL INFO Channel 01/0 : 11[51000] -> 18[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75639 [0] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75628 [2] NCCL INFO Channel 01/0 : 3[51000] -> 10[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75647 [3] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75648 [2] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75648 [2] NCCL INFO NCCL_IB_SL set by environment to 3. ts-cbba87c5e7504a249f512710tts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82585:83811 [3] NCts-cbba87c5e7504a249f512710ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54964:56074 [3] NCCL INFO Channel 00/0 : 27[51000] -> ts-cbba87c5e7504a249f512710ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54964:56074 [3] NCCL INFO Channel 01/0 : 27[51000] -> 28[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54963:56070 [2] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54963:56070 [2] NCCL INFO Channel 00/0 : 26[4b000] -> ts-cbba87c5e7504a249f512710ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54963:56070 [2] NCCL INFO Channel 01/0 : 26[4b000] -> ts-cbba87c5e7504a249f512710ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54965:56069 [4] NCCL INFO Channel 00/0 : 28[93000] -> 27[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54965:56069 [4] NCCL INFO Channel 01/0 : 28[93000] -> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75628 [2] NCCL INFO Channel 01/0 : 2[4b000] -> 10[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75624 [3] NCCL INFO Channel 01/0 : 18[4b000] -> 11[51000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75628 [2] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75628 [2] NCCL INFO Channel 00/0 : 10[4b000] -> 19[51000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75628 [2] NCCL INFO Channel 01/0 : 26[4b000] -> 10[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75628 [2] NCCL INFO Channel 01/0 : 10[4b000] -> 26[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75628 [2] NCCL INFO Channel 00/0 : 19[51000] -> 10[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75628 [2] NCCL INFO Channel 01/0 : 10[4b000] -> 2[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75647 [3] NCCL INFO NCCL_IB_SL set by environment to 3. ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75624 [3] NCCL INFO Channel 00/0 : 11[51000] -> 10[4b000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75624 [3] NCCL INFO Channel 01/0 : 11[51000] -> 10[4b000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75623 [4] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75623 [4] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75623 [4] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75628 [2] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75628 [2] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 5ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149849:151270 [0] NCCL INFO comm 0x49482970 rank 0 nranks 32 cudaDev 0 busId e000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149850:151274 [1] NCCL INFO comm 0x4aae3420 rank 1 nranks 32 cudaDev 1 busId 13000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149852:151272 [3] NCCL INFO comm 0x49838ad0 rank 3 nranks 32 cudaDev 3 busId 51000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149851:151278 [2] NCCL INFO comm 0x485a1a60 rank 2 nranks 32 cudaDev 2 busId 4b000 - Init COMPLETE ts-cbba87c5e7ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75623 [4] NCCL INFO comm 0x46989410 rank 12 nranks 32 cudaDev 4 busId 93000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75627 [7] NCCL INFO comm 0x480dc140 rank 15 nranks 32 cudaDev 7 busId d0000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75639 [0] NCCL INFO comm 0x48b2f8b0 rank 8 nranks 32 cudaDev 0 busId e000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75625 [6] NCCL INFO comm 0x46d6d5b0 rank 14 nranks 32 cudaDev 6 busId cb000 - Init CO11/16/2023 11:41:47 - INFO - __main__ - xxx: Showcase the tokenized training samples. {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933{'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]} {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]} {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29{'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374,{'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]} {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]}{'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]}{'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]} {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]} {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]} {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]} {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]} {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]} {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]}{'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]} {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]} {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]} {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]} {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]} {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]} {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]} {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]} {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]} {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2]} {'input_ids': [1, 13866, 338, 385, 15278, 393, 16612, 263, 3414, 29889, 14350, 263, 2933, 393, 7128, 2486, 1614, 2167, 278, 2009, 29889, 13, 13, 2277, 29937, 2799, 4080, 29901, 13, 29954, 573, 2211, 25562, 363, 7952, 292, 9045, 29891, 29889, 13, 13, 2277, 29937, 13291, 29901, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 8709, 1269, 4646, 29889, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 29896, 29889, 382, 271, 263, 6411, 8362, 322, 18254, 768, 2738, 652, 300, 29901, 8561, 1854, 596, 592, 1338, 526, 20978, 573, 310, 263, 12875, 310, 285, 21211, 322, 18655, 1849, 29892, 20793, 26823, 29892, 3353, 2646, 1144, 29892, 322, 9045, 29891, 285, 1446, 29889, 910, 6911, 304, 3867, 596, 3573, 411, 278, 18853, 18254, 374, 1237, 304, 740, 472, 967, 1900, 322, 508, 1371, 5557, 17168, 293, 10267, 2129, 29889, 13, 13, 29906, 29889, 2201, 482, 297, 4943, 9128, 6354, 29901, 1222, 6269, 895, 338, 7618, 1455, 363, 7344, 292, 4549, 289, 2873, 29892, 2301, 7799, 29892, 322, 5881, 29875, 586, 6151, 1070, 9045, 29889, 319, 326, 363, 472, 3203, 29871, 29896, 29945, 29900, 6233, 310, 17768, 403, 14911, 711, 293, 15058, 470, 29871, 29955, 29945, 6233, 310, 14877, 20657, 15058, 1269, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29[INFO|trainer.py:606] 2023-11-16 11:41:48,693 >> Using cuda_amp half precision backend [2023-11-16 11:41:48,703] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.11.1, git-hash=unknown, git-branch=unknown 9, 4723, 29889, 13, 13, 29941, 29889, 3617, 3307, 8709, 29901, 24162, 3307, 11029, 8709, 338, 7618, 1455, 363, 9128, 322, 19119, 1532, 29899, 915, 292, 29889, 739, 6911, 304, 1072, 5987, 286, 2092, 29892, 11157, 25323, 3321, 740, 29892, 322, 11286, 9045, 29891, 14321, 322, 5198, 1540, 740, 29889, 319, 326, 363, 29871, 29955, 29899, 29929, 6199, 310, 870[INFO|trainer.py:606] 2023-[INFO|trainer.py:606] 2023-11-16 11:41:48,681 >> Using cuda_amp half precision backend ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54965:56130 [4] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54961:56132 [0] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54964:56131 [3] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54967:56133 [6] NCCL INFO Using network IB ts-cbba8ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149851:151368 [2] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149853:151369 [4] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149852:151370 [3] NCCL INFO Usits-cbba87c5e75ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149855:151371 [6] NCCL INFO Uts-cbba87c5e7504ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75683 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffff0000,00000000,ffffffff,ffff0000,00000000 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75688 [2] NCCL INFO Setting affinity for GPU 2 to ffff,ffffffff,00000000,0000ffff,ffffffff ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75687 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffff0000,00000000,ffffffff,ffff0000,00000000 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75685 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffff0000,00000000,ffffffff,ffff0000,00000000 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75684 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffff0000,00000000,ffffffff,ffff0000,00000000 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75690 [0] NCCL INFO Setting affinity for GPU 0 to ffff,ffffffff,00000000,0000ffff,ffffffff ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:75689 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ffffffff,00000000,0000ffff,ffffffff ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75686 [3] NCCL INFO Setting affinity for GPU 3 to ffff,ffffffff,00000000,00ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82582:83887 [0] NCCL INFO Trees [0] 17/-1/-1->16->23 [1] 17/-1/-1->16->23 ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54968:56136 [7] NCCL INFO Trees [0] 24/-1/-1->31->30 [1] 24/-1/-1->31->30 ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54967:56133 [6] NCCL INFO Trees [0] 31/-1/-1->30->29 [1] 31/-1/-1->30->29ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82588:83881 [6] NCCL INFO Trees [0] 23/-1/-1->22->21 [1] 23/-1/-1->22->21 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82589:83888 [7] NCCL INFO Trees [0] 16/-1/-1->23->22 [1] 16/-1/-1->23->22 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82585:83884 [3] NCCL INFO Trees [0] 20/10/-1->19->18 [1] 20/-1/-1->19->18 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82587:83883 [5] NCCL INFO Trees [0] 22/-1/-1->21->20 [1] 22/-1/-1->21->20 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82586:83882 [4] NCCL INFO Trees [0] 21/-1/-1->20->19 [1] 21/-1/-1->20->19 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82582:83887 [0] NCCL INFO Channel 00/0 : 16[e000] -> 19[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82586:83882 [4] NCCL INFO Channel 00/0 : 20[93000] -> 17[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82584:83885 [2] NCCL INFO Channel 00/0 : 18[4b000] -> 23[d0000] via P2P/IPC/read tts-ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75683 [4] NCCL INFO Channel 01/0 : 12[93000] -> 9[13000] via P2P/IPC/rets-ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75688 [2] NCCL INFO Channel 01/0 : 10[4b000] -> 15[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75690 [0] NCCL INFO Channel 01/0 : 8[e000] -> 11[51000] via P2P/IPC/reattts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75685 [5] NCCL INFO Channel 00/0 : 13[99000] -> 12[93000] via P2P/IPC/readtts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75684 [6] NCCL INFO Channel 00/0 : 14[cb000] -> 13[99000] via P2P/IPC/reattts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75684 [6] NCCL INFO Channel 01/0 : 14[cb000] -> 13[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75685 [5] NCCL INFO Channel 01/0 : 13[99000] -> 12[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75687 [7] NCCL INFO Channel 00/0 : 15[d0000] -> 14[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:75689 [1] NCCL INFO Channel 00/0 : 9[13000] -> 8[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75687 [7] NCCL INFO Channel 01/0 : 15[d0000] -> 14[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:75689 [1] NCCL INFO Channel 01/0 : 9[13000] -> 8[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75683 [4] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75687 [7] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:75689 [1] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75690 [0] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75690 [0] NCCL INFO Channel 00/0 : 8[e000] -> 9[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75690 [0] NCCL INFO Channel 01/0 : 8[e000] -> 9[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75685 [5] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75684 [6] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75683 [4] NCCL INFO Channel 00/0 : 12[93000] -> 13[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75683 [4] NCCL INFO Channel 01/0 : 12[93000] -> 13[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75685 [5] NCCL INFO Channel 00/0 : 13[99000] -> 14[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75684 [6] NCCL INFO Channel 00/0 : 14[cb000] -> 15[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:75689 [1] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:75689 [1] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:75689 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75690 [0] NCCL INFO Channel 00/0 : 8[e000] -> 15[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75685 [5] NCCL INFO Channel 01/0 : 13[99000] -> 14[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75684 [6] NCCL INFO Channel 01/0 : 14[cb000] -> 15[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75690 [0] NCCL INFO Channel 01/0 : 8[e000] -> 15[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75685 [5] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75685 [5] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75685 [5] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75684 [6] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75684 [6] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75684 [6] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75688 [2] NCCL INFO Channel 00/0 : 3[51000] -> 10[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75686 [3] NCCL INFO Channel 00/0 : 11[51000] -> 18[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75687 [7] NCCL INFO Channel 00/0 : 15[d0000] -> 8[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75687 [7] NCCL INFO Channel 01/0 : 15[d0000] -> 8[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75687 [7] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75687 [7] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75687 [7] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75690 [0] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75690 [0] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75690 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75686 [3] NCCL INFO Channel 01/0 : 11[51000] -> 18[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e75ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54964:56131 [3] NCCL INFO Channel 01/0 : 27[51000] -> 2[4b000] [send] via NET/Its-cbba87c5ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54964:56131 [3] NCCL INFO Connectedts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75686 [3] NCCL INFO Channel 00/0 : 11[51000] -> 12[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75686 [3] NCCL INFO Channel 01/0 : 11[51000] -> 12[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75688 [2] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75688 [2] NCCL INFO Channel 00/0 : 10[4b000] -> 11[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75688 [2] NCCL INFO Channel 01/0 : 10[4b000] -> 11[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75683 [4] NCCL INFO Channel 00/0 : 12[93000] -> 11[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75683 [4] NCCL INFO Channel 01/0 : 12[93000] -> 11[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75688 [2] NCCL INFO Channel 01/0 : 2[4b000] -> 10[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75686 [3] NCCL INFO Channel 01/0 : 18[4b000] -> 11[51000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75688 [2] NCCL INFO Channel 00/0 : 10[4b000] -> 19[51000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75688 [2] NCCL INFO Channel 01/0 : 26[4b000] -> 10[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75688 [2] NCCL INFO Channel 01/0 : 10[4b000] -> 26[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75688 [2] NCCL INFO Channel 00/0 : 19[51000] -> 10[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba8ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82585:83884 [3] NCCL INFO Channel 00/0 : 19[51000] -> 18[4b000] via P2P/IPC/read tts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75686 [3] NCCL INFO Channel 00/0 : 11[51000] -> 10[4b000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75686 [3] NCCL INFO Channel 01/0 : 11[51000] -> 10[4b000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75688 [2] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75688 [2] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75688 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75683 [4] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75683 [4] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75683 [4] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75686 [3] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75686 [3] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75686 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74474:75683 [4] NCCL INFO comm 0xb30b140 rank 12 nranks 32 cudaDev 4 busId 93000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75690 [0] NCCL INFO comm 0xbb13e80 rank 8 nranks 32 cudaDev 0 busId e000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74476:75684 [6] NCCL INFO comm 0xaa48900 rank 14 nranks 32 cudaDev 6 busId cb000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74477:75687 [7] NCCL INFO comm 0xce47680 rank 15 nranks 32 cudaDev 7 busId d0000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74473:75686 [3] NCCL INFO comm 0xb0b1b80 rank 11 nranks 32 cudaDev 3 busId 51000 - Init COMPLETE ts-cbba87c5e7504a249f51271ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54964:56151 [3] NCCL INFO NCCL_IB_SL set by environment to 3. COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74475:75685 [5] NCCL INFO comm 0x97e5d00 rank 13 nranks 32 cudaDev 5 busId 99000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:75689 [1] NCCL INFO comm 0xb5a92c0 rank 9 nranks 32 cudaDev 1 busId 13000 - Init COMPLETE t COMPLETE els, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149852:151370 [3] NCCL INFO comm 0xce01180 rank 3 nranks 32 cudaDev 3 busId 51000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149849:151364 [0] NCCL INFO comm 0xbfe1760 rank 0 nranks 32 cudaDev 0 busId e000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149850:151365 [1] NCCL INFO comm 0xccacb80 rank 1 nranks 32 cudaDev 1 busId 13000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149851:151368 [2] NCCL INFO comm 0xb893aa0 rank 2 nranks 32 cudaDev 2 busId 4b000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149854:151366 [5] NCCL INFO comm 0xb546140 rank 5 nranks 32 cudaDev 5 busId 99000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149856:151367 [7] NCCL INFO comm 0xaea7e00 rank 7 nranks 32 cudaDev 7 busId d0000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149853:151369 [4] NCCL INFO comm 0xc254540 rank 4 nranks 32 cudaDev 4 busId 93000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149855:151371 [6] NCCL INFO comm 0xb16cbc0 rank 6 nranks 32 cudaDev 6 busId cb000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149852:151393 [3] NCCL INFO NCCL_IB_SL set by environment to 3. [2023-11-16 11:42:04,592] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Detected CUDA files, patching ldflags Emitting ninja build file /root/.cache/torch_extensions/py38_cu117/cpu_adam/build.ninja... Building extension module cpu_adam... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... ninja: no work to do. Loading extension module cpu_adam... Loading extension module cpu_adam... Time to load cpu_adam op: 1.1647510528564453 seconds Time to load cpu_adam op: 0.7984278202056885 seconds Loading extension module cpu_adam... Time to load cpu_adam op: 0.9247500896453857 seconds Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Detected CUDA files, patching ldflags Emitting ninja build file /root/.cache/torch_extensions/py38_cu117/cpu_adam/build.ninja... Building extension module cpu_adam... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module cpu_adam... Time to load cpu_adam op: 1.1291096210479736 seconds Loading extension module cpu_adam... Time to load cpu_adam op: 1.1177616119384766 seconds Loading extension module cpu_adam... Loading extension module cpu_adam... Time to load cpu_adam op: 1.092052698135376 seconds Time to load cpu_adam op: 1.0916635990142822 seconds Loading extension module cpu_adam... Time to load cpu_adam op: 1.196195363998413 seconds Adam Optimizer #0 is created with AVX2 arithmetic capability. Config: alpha=0.000020, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_w=1 y:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam [2023-11-16 11:42:11,075] [INFO] [utils.py:56:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type= [2023-11-16 11:42:11,075] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 2 optimizer [2023-11-16 11:42:11,075] [INFO] [stage_1_and_2.py:146:__init__] Reduce bucket size 200000000 [2023-11-16 11:42:11,075] [INFO] [stage_1_and_2.py:147:__init__] Allgather bucket size 200000000 [2023-11-16 11:42:11,076] [INFO] [stage_1_and_2.py:148:__init__] CPU Offload: True [2023-11-16 11:42:11,076] [INFO] [stage_1_and_2.py:149:__init__] Round robin gradient partitioning: False [WARNING|logging.py:295] 2023-11-16 11:42:27,018 >> `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... [WARNING|logging.py:295] 2023-11-16 11:42:27,799 >> `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... [WARNING|logging.py:295] 2023-11-16 11:42:28,776 >> `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... [WARNING|logging.py:295] 2023-11-16 11:42:28,945 >> `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... [INFO|trainer.py:1755] 2023-11-16 11:42:29,376 >> ***** Running training ***** [INFO|trainer.py:1756] 2023-11-16 11:42:29,377 >> Num examples = 75399 [INFO|trainer.py:1757] 2023-11-16 11:42:29,377 >> Num Epochs = 3 [INFO|trainer.py:1758] 2023-11-16 11:42:29,377 >> Instantaneous batch size per device = 48 [INFO|trainer.py:1759] 2023-11-16 11:42:29,377 >> Total train batch size (w. parallel, distributed & accumulation) = 1536 [INFO|trainer.py:1760] 2023-11-16 11:42:29,377 >> Gradient Accumulation steps = 1 [INFO|trainer.py:1761] 2023-11-16 11:42:29,377 >> Total optimization steps = 150 [INFO|trainer.py:1762] 2023-11-16 11:42:29,378 >> Number of trainable parameters = 6738423808 0%| | 0/150 [00:00> `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... [WARNING|logging.py:295] 2023-11-16 11:42:29,886 >> `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... [WARNING|logging.py:295] 2023-11-16 11:42:29,920 >> `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... [WARNING|logging.py:295] 2023-11-16 11:42:30,284 >> `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... "contiguous_memory_optimization": false, "cpu_checkpointing": false, "number_checkpoints": null, "synchronize_checkpoint_boundary": false, "profile": false } [2023-11-16 11:42:27,899] [INFO] [config.py:972:print] aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True} [2023-11-16 11:42:27,899] [INFO] [config.py:972:print] amp_enabled .................. False [2023-11-16 11:42:27,899] [INFO] [config.py:972:print] amp_params ................... False [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] autotuning_config ............ { "enabled": false, "start_step": null, "end_step": null, "metric_path": null, "arg_mappings": null, "metric": "throughput", "model_info": null, "results_dir": "autotuning_results", "exps_dir": "autotuning_exps", "overwrite": true, "fast": true, "start_profile_step": 3, "end_profile_step": 5, "tuner_type": "gridsearch", "tuner_early_stopping": 5, "tuner_num_trials": 50, "model_info_path": null, "mp_size": 1, "max_train_batch_size": null, "min_train_batch_size": 1, "max_train_micro_batch_size_per_gpu": 1.024000e+03, "min_train_micro_batch_size_per_gpu": 1, "num_tuning_micro_batch_sizes": 3 } [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] bfloat16_enabled ............. True [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] checkpoint_parallel_write_pipeline False [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] checkpoint_tag_validation_enabled True [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] checkpoint_tag_validation_fail False [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] comms_config ................. [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] communication_data_type ...... None [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}} [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] curriculum_enabled_legacy .... False [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] curriculum_params_legacy ..... False [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}} [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] data_efficiency_enabled ...... False [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] dataloader_drop_last ......... False [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] disable_allgather ............ False [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] dump_state ................... False [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] dynamic_loss_scale_args ...... None [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] eigenvalue_enabled ........... False [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] eigenvalue_gas_boundary_resolution 1 [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] eigenvalue_layer_name ........ bert.encoder.layer [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] eigenvalue_layer_num ......... 0 [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] eigenvalue_max_iter .......... 100 [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] eigenvalue_stability ......... 1e-06 [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] eigenvalue_tol ............... 0.01 [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] eigenvalue_verbose ........... False [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] elasticity_enabled ........... False [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] flops_profiler_config ........ { "enabled": false, "recompute_fwd_factor": 0.0, "profile_step": 1, "module_depth": -1, "top_modules": 1, "detailed": true, "output_file": null } [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] fp16_auto_cast ............... None [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] fp16_enabled ................. False [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] fp16_master_weights_and_gradients False [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] global_rank .................. 0 [2023-11-16 11:42:27,900] [INFO] [config.py:972:print] grad_accum_dtype ............. None [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] gradient_accumulation_steps .. 1 [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] gradient_clipping ............ 1.0 [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] gradient_predivide_factor .... 1.0 [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8 [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] initial_dynamic_scale ........ 1 [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] load_universal_checkpoint .... False [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] loss_scale ................... 1.0 [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] memory_breakdown ............. False [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] mics_hierarchial_params_gather False [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] mics_shard_size .............. -1 [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] nebula_config ................ { "enabled": false, "persistent_storage_path": null, "persistent_time_interval": 100, "num_of_version_in_retention": 2, "enable_nebula_load": true, "load_path": null } [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] optimizer_legacy_fusion ...... False [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] optimizer_name ............... adam [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] optimizer_params ............. {'lr': 2e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.0} [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0} [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] pld_enabled .................. False [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] pld_params ................... False [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] prescale_gradients ........... False [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] scheduler_name ............... None [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] scheduler_params ............. None [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] sparse_attention ............. None [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] sparse_gradients_enabled ..... False [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] steps_per_print .............. 1000 [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] train_batch_size ............. 1536 [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] train_micro_batch_size_per_gpu 48 [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] use_node_local_storage ....... False [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] wall_clock_breakdown ......... False [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] weight_quantization_config ... None [2023-11-16 11:42:27,901] [INFO] [config.py:972:print] world_size ................... 32 [2023-11-16 11:42:27,902] [INFO] [config.py:972:print] zero_allow_untested_optimizer False [2023-11-16 11:42:27,902] [INFO] [config.py:972:print] zero_config .................. stage=2 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=200000000 allgather_partitions=True allgather_bucket_size=200000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='cpu', nvme_path=None, buffer_count=4, pin_memory=True, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False) sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True [2023-11-16 11:42:27,902] [INFO] [config.py:972:print] zero_enabled ................. True [2023-11-16 11:42:27,902] [INFO] [config.py:972:print] zero_force_ds_cpu_optimizer .. True [2023-11-16 11:42:27,902] [INFO] [config.py:972:print] zero_optimization_stage ...... 2 [2023-11-16 11:42:27,902] [INFO] [config.py:958:print_user_config] json = { "optimizer": { "type": "Adam", "params": { "lr": 2e-05, "betas": [0.9, 0.999], "eps": 1e-08, "weight_decay": 0.0 } }, "bf16": { "enabled": true }, "fp16": { "enabled": false, "loss_scale": 0, "loss_scale_window": 1000, "initial_scale_power": 16, "hysteresis": 2, "min_loss_scale": 1 }, "zero_optimization": { "stage": 2, "offload_optimizer": { "device": "cpu", "pin_memory": true }, "allgather_partitions": true, "allgather_bucket_size": 2.000000e+08, "overlap_comm": true, "reduce_scatter": true, "reduce_bucket_size": 2.000000e+08, "contiguous_gradients": true }, "gradient_accumulation_steps": 1, "gradient_clipping": 1.0, "steps_per_print": 1000, "train_batch_size": 1.536000e+03, "train_micro_batch_size_per_gpu": 48, "wall_clock_breakdown": false } [INFO|trainer.py:1755] 2023-11-16 11:42:27,903 >> ***** Running training ***** [INFO|trainer.py:1756] 2023-11-16 11:42:27,903 >> Num examples = 75399 [INFO|trainer.py:1757] 2023-11-16 11:42:27,903 >> Num Epochs = 3 [INFO|trainer.py:1758] 2023-11-16 11:42:27,903 >> Instantaneous batch size per device = 48 [INFO|trainer.py:1759] 2023-11-16 11:42:27,903 >> Total train batch size (w. parallel, distributed & accumulation) = 1536 [INFO|trainer.py:1760] 2023-11-16 11:42:27,903 >> Gradient Accumulation steps = 1 [INFO|trainer.py:1761] 2023-11-16 11:42:27,903 >> Total optimization steps = 150 [INFO|trainer.py:1762] 2023-11-16 11:42:27,905 >> Number of trainable parameters = 6738423808 0%| | 0/150 [00:00> `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... [WARNING|logging.py:295] 2023-11-16 11:42:28,666 >> `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... [WARNING|logging.py:295] 2023-11-16 11:42:28,810 >> `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... [WARNING|logging.py:295] 2023-11-16 11:42:28,918 >> `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... [WARNING|logging.py:295] 2023-11-16 11:42:28,919 >> `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... [WARNING|logging.py:295] 2023-11-16 11:42:28,928 >> `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... [WARNING|logging.py:295] 2023-11-16 11:42:28,953 >> `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... [WARNING|logging.py:295] 2023-11-16 11:42:29,038 >> `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... 1%| | 1/150 [00:17<43:28, 17.51s/it] 1%|▏ | 2/150 [00:36<44:50, 18.18s/it] 2%|▏ | 3/150 [00:54<43:37, 17.81s/it] 3%|▎ | 4/150 [01:11<42:55, 17.64s/it] 3%|▎ | 5/150 [01:28<42:01, 17.39s/it] 4%|▍ | 6/150 [01:44<40:45, 16.98s/it] 5%|▍ | 7/150 [01:59<40:12, 16.87s/it] 5%|▌ | 8/150 [02:18<40:09, 16.97s/it] 6%|▌ | 9/150 [02:35<39:47, 16.94s/it] 7%|▋ | 10/150 [02:51<39:23, 16.88s/it] {'loss': 1.0685, 'learning_rate': 1.9941379571543597e-05, 'epoch': 0.2} 7%|▋ | 10/150 [02:51<39:23, 16.88s/it] 7%|▋ | 11/150 [03:09<39:27, 17.03s/it] 8%|▊ | 12/150 [03:26<39:07, 17.01s/it] 9%|▊ | 13/150 [03:44<39:11, 17.17s/it] 9%|▉ | 14/150 [04:01<38:58, 17.19s/it] 10%|█ | 15/150 [04:17<38:12, 16.98s/it] 11%|█ | 16/150 [04:35<38:02, 17.03s/it] 11%|█▏ | 17/150 [04:51<37:36, 16.97s/it] 12%|█▏ | 18/150 [05:08<37:01, 16.83s/it] 13%|█▎ | 19/150 [05:25<36:49, 16.87s/it] 13%|█▎ | 20/150 [05:42<36:36, 16.90s/it] {'loss': 0.9694, 'learning_rate': 1.9476531711828027e-05, 'epoch': 0.4} 13%|█▎ | 20/150 [05:41<36:36, 16.90s/it] 14%|█▍ | 21/150 [05:59<36:24, 16.94s/it] 15%|█▍ | 22/150 [06:14<36:41, 17.20s/it] 15%|█▌ | 23/150 [06:34<36:16, 17.14s/it] 16%|█▌ | 24/150 [06:50<35:46, 17.04s/it] 17%|█▋ | 25/150 [07:08<35:31, 17.06s/it] 17%|█▋ | 26/150 [07:24<35:02, 16.95s/it] 18%|█▊ | 27/150 [07:41<34:29, 16.82s/it] 19%|█▊ | 28/150 [07:58<34:16, 16.86s/it] 19%|█▉ | 29/150 [08:15<34:17, 17.00s/it] 20%|██ | 30/150 [08:32<34:14, 17.12s/it] {'loss': 0.9469, 'learning_rate': 1.8568571761675893e-05, 'epoch': 0.6} 20%|██ | 30/150 [08:31<34:14, 17.12s/it] 21%|██ | 31/150 [08:50<34:00, 17.15s/it] 21%|██▏ | 32/150 [09:05<32:49, 16.69s/it] 22%|██▏ | 33/150 [09:22<32:31, 16.68s/it] 23%|██▎ | 34/150 [09:38<32:07, 16.61s/it] 23%|██▎ | 35/150 [09:55<31:43, 16.55s/it] 24%|██▍ | 36/150 [10:12<31:43, 16.70s/it] 25%|██▍ | 37/150 [10:27<31:44, 16.85s/it] 25%|██▌ | 38/150 [10:46<31:27, 16.85s/it] 26%|██▌ | 39/150 [11:02<30:45, 16.63s/it] 27%|██▋ | 40/150 [11:19<30:30, 16.64s/it] {'loss': 0.9373, 'learning_rate': 1.725995491923131e-05, 'epoch': 0.8} 27%|██▋ | 40/150 [11:18<30:29, 16.63s/it] 27%|██▋ | 41/150 [11:36<30:23, 16.73s/it] 28%|██▊ | 42/150 [11:52<30:18, 16.84s/it] 29%|██▊ | 43/150 [12:09<30:09, 16.91s/it] 29%|██▉ | 44/150 [12:27<30:14, 17.12s/it] 30%|███ | 45/150 [12:44<29:57, 17.12s/it] 31%|███ | 46/150 [12:58<29:20, 16.93s/it] 31%|███▏ | 47/150 [13:18<29:03, 16.93s/it] 32%|███▏ | 48/150 [13:35<28:50, 16.97s/it] 33%|███▎ | 49/150 [13:52<28:26, 16.89s/it] 33%|███▎ | 50/150 [13:57<23:39, 14.20s/it] {'loss': 0.9309, 'learning_rate': 1.5611870653623826e-05, 'epoch': 1.0} 33%|███▎ | 50/150 [13:58<23:39, 14.19s/it] 34%|███▍ | 51/150 [14:16<25:58, 15.75s/it] 35%|███▍ | 52/150 [14:34<26:25, 16.18s/it] 35%|███▌ | 53/150 [14:50<26:27, 16.36s/it] 36%|███▌ | 54/150 [15:07<26:13, 16.39s/it] 37%|███▋ | 55/150 [15:24<26:08, 16.51s/it] 37%|███▋ | 56/150 [15:40<25:57, 16.57s/it] 38%|███▊ | 57/150 [15:59<25:52, 16.69s/it] 39%|███▊ | 58/150 [16:14<25:43, 16.77s/it] 39%|███▉ | 59/150 [16:32<25:41, 16.94s/it] 40%|████ | 60/150 [16:49<25:23, 16.93s/it] {'loss': 0.9124, 'learning_rate': 1.3701381553399147e-05, 'epoch': 1.2} 40%|████ | 60/150 [16:50<25:23, 16.93s/it] 41%|████ | 61/150 [17:06<24:51, 16.76s/it] 41%|████▏ | 62/150 [17:23<24:35, 16.77s/it] 42%|████▏ | 63/150 [17:38<24:16, 16.75s/it] 43%|████▎ | 64/150 [17:56<24:18, 16.96s/it] 43%|████▎ | 65/150 [18:13<23:59, 16.93s/it] 44%|████▍ | 66/150 [18:31<23:32, 16.81s/it] 45%|████▍ | 67/150 [18:46<23:17, 16.84s/it] 45%|████▌ | 68/150 [19:04<23:02, 16.85s/it] 46%|████▌ | 69/150 [19:20<22:37, 16.76s/it] 47%|████▋ | 70/150 [19:36<22:18, 16.73s/it] {'loss': 0.9115, 'learning_rate': 1.161781996552765e-05, 'epoch': 1.4} 47%|████▋ | 70/150 [19:38<22:18, 16.73s/it] 47%|████▋ | 71/150 [19:56<22:17, 16.93s/it] 48%|████▊ | 72/150 [20:13<22:00, 16.93s/it] 49%|████▊ | 73/150 [20:29<21:36, 16.84s/it] 49%|████▉ | 74/150 [20:46<21:22, 16.87s/it] 50%|█████ | 75/150 [21:02<21:00, 16.80s/it] 51%|█████ | 76/150 [21:17<20:38, 16.74s/it] 51%|█████▏ | 77/150 [21:34<20:23, 16.76s/it] 52%|█████▏ | 78/150 [21:53<20:13, 16.85s/it] 53%|█████▎ | 79/150 [22:11<19:59, 16.90s/it] 53%|█████▎ | 80/150 [22:27<19:54, 17.06s/it] {'loss': 0.906, 'learning_rate': 9.458610914145826e-06, 'epoch': 1.6} 53%|█████▎ | 80/150 [22:27<19:54, 17.06s/it] 54%|█████▍ | 81/150 [22:44<19:36, 17.05s/it] 55%|█████▍ | 82/150 [23:02<19:11, 16.93s/it] 55%|█████▌ | 83/150 [23:16<18:55, 16.95s/it] 56%|█████▌ | 84/150 [23:35<18:38, 16.95s/it] 57%|█████▋ | 85/150 [23:50<18:15, 16.85s/it] 57%|█████▋ | 86/150 [24:07<17:58, 16.85s/it] 58%|█████▊ | 87/150 [24:26<17:52, 17.02s/it] 59%|█████▊ | 88/150 [24:43<17:33, 16.99s/it] 59%|█████▉ | 89/150 [24:59<17:04, 16.79s/it] 60%|██████ | 90/150 [25:16<16:52, 16.88s/it] {'loss': 0.9042, 'learning_rate': 7.324716614707794e-06, 'epoch': 1.8} 60%|██████ | 90/150 [25:16<16:52, 16.88s/it] 61%|██████ | 91/150 [25:31<16:37, 16.90s/it] 61%|██████▏ | 92/150 [25:50<16:18, 16.87s/it] 62%|██████▏ | 93/150 [26:07<16:01, 16.87s/it] 63%|██████▎ | 94/150 [26:25<15:51, 16.99s/it] 63%|██████▎ | 95/150 [26:41<15:32, 16.96s/it] 64%|██████▍ | 96/150 [26:58<15:05, 16.76s/it] 65%|██████▍ | 97/150 [27:15<14:49, 16.78s/it] 65%|██████▌ | 98/150 [27:29<14:36, 16.85s/it] 66%|██████▌ | 99/150 [27:48<14:18, 16.83s/it] 67%|██████▋ | 100/150 [27:55<11:42, 14.05s/it] {'loss': 0.9025, 'learning_rate': 5.3159155930021e-06, 'epoch': 2.0} 67%|██████▋ | 100/150 [27:55<11:42, 14.05s/it] 67%|██████▋ | 101/150 [28:16<12:50, 15.71s/it] 68%|██████▊ | 102/150 [28:31<12:41, 15.87s/it] 69%|██████▊ | 103/150 [28:47<12:32, 16.00s/it] 69%|██████▉ | 104/150 [29:03<12:30, 16.32s/it] 70%|███████ | 105/150 [29:20<12:22, 16.50s/it] 71%|███████ | 106/150 [29:36<12:07, 16.54s/it] 71%|███████▏ | 107/150 [29:53<11:55, 16.64s/it] 72%|███████▏ | 108/150 [30:11<11:48, 16.87s/it] 73%|███████▎ | 109/150 [30:30<11:30, 16.85s/it] 73%|███████▎ | 110/150 [30:46<11:09, 16.75s/it] {'loss': 0.8989, 'learning_rate': 3.5261371521817247e-06, 'epoch': 2.2} 73%|███████▎ | 110/150 [30:46<11:09, 16.75s/it] 74%|███████▍ | 111/150 [31:01<10:55, 16.81s/it] 75%|███████▍ | 112/150 [31:20<10:38, 16.80s/it] 75%|███████▌ | 113/150 [31:36<10:18, 16.72s/it] 76%|███████▌ | 114/150 [31:52<10:00, 16.69s/it] 77%|███████▋ | 115/150 [32:11<09:52, 16.93s/it] 77%|███████▋ | 116/150 [32:25<09:36, 16.95s/it] 78%|███████▊ | 117/150 [32:42<09:18, 16.92s/it] 79%|███████▊ | 118/150 [32:59<09:01, 16.91s/it] 79%|███████▉ | 119/150 [33:18<08:41, 16.83s/it] 80%|████████ | 120/150 [33:32<08:22, 16.74s/it] {'loss': 0.893, 'learning_rate': 2.0390693429435626e-06, 'epoch': 2.4} 80%|████████ | 120/150 [33:34<08:22, 16.74s/it] 81%|████████ | 121/150 [33:49<08:05, 16.73s/it] 81%|████████▏ | 122/150 [34:07<07:54, 16.96s/it] 82%|████████▏ | 123/150 [34:25<07:38, 16.97s/it] 83%|████████▎ | 124/150 [34:41<07:20, 16.96s/it] 83%|████████▎ | 125/150 [34:57<07:03, 16.95s/it] 84%|████████▍ | 126/150 [35:15<06:42, 16.79s/it] 85%|████████▍ | 127/150 [35:32<06:22, 16.64s/it] 85%|████████▌ | 128/150 [35:49<06:08, 16.77s/it] 86%|████████▌ | 129/150 [36:06<05:53, 16.83s/it] 87%|████████▋ | 130/150 [36:23<05:37, 16.89s/it] {'loss': 0.8912, 'learning_rate': 9.242458032904311e-07, 'epoch': 2.6} 87%|████████▋ | 130/150 [36:23<05:37, 16.89s/it] 87%|████████▋ | 131/150 [36:39<05:25, 17.12s/it] 88%|████████▊ | 132/150 [36:57<05:07, 17.07s/it] 89%|████████▊ | 133/150 [37:14<04:46, 16.85s/it] 89%|████████▉ | 134/150 [37:30<04:29, 16.87s/it] 90%|█████████ | 135/150 [37:48<04:14, 16.93s/it] 91%|█████████ | 136/150 [38:05<03:57, 16.93s/it] 91%|█████████▏| 137/150 [38:21<03:39, 16.91s/it] 92%|█████████▏| 138/150 [38:37<03:25, 17.08s/it] 93%|█████████▎| 139/150 [38:55<03:07, 17.08s/it] 93%|█████████▎| 140/150 [39:11<02:49, 16.99s/it] {'loss': 0.8924, 'learning_rate': 2.3379444289913344e-07, 'epoch': 2.8} 93%|█████████▎| 140/150 [39:13<02:49, 16.99s/it] 94%|█████████▍| 141/150 [39:30<02:33, 17.02s/it] 95%|█████████▍| 142/150 [39:46<02:14, 16.86s/it] 95%|█████████▌| 143/150 [40:02<01:57, 16.84s/it] 96%|█████████▌| 144/150 [40:21<01:40, 16.79s/it] 97%|█████████▋| 145/150 [40:38<01:24, 16.97s/it] 97%|█████████▋| 146/150 [40:55<01:08, 17.01s/it] 98%|█████████▊| 147/150 [41:11<00:50, 16.75s/it] 99%|█████████▊| 148/150 [41:28<00:33, 16.66s/it] 99%|█████████▉| 149/150 [41:45<00:16, 16.79s/it] 100%|██████████| 150/150 [41:53<00:00, 14.18s/it] {'loss': 0.8895, 'learning_rate': 0.0, 'epoch': 3.0} 100%|██████████| 150/150 [41:52<00:00, 14.18s/it][INFO|trainer.py:2025] 2023-11-16 12:24:21,300 >> Training completed. Do not forget to share your model on huggingface.co/models =) {'train_runtime': 2513.3699, 'train_samples_per_second': 89.997, 'train_steps_per_second': 0.06, 'train_loss': 0.9236494954427084, 'epoch': 3.0} 100%|██████████| 150/150 [41:53<00:00, 14.18s/it] 100%|██████████| 150/150 [41:53<00:00, 16.76s/it]ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74472:75648 [2] NCCL INFO [Service thread] Connection closed by localRank 7 tts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82588:83837 [6] NCCL INFO [Service thread] Connection closed by localRank 6tts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82588:82588 [6] NCCL INFO comm 0x464afa20 rank 22 nranks 32 cudaDev 6 busId cb000 - Abort COMPLETEtts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82584:82584 [2] NCCL INFO comm 0x4780fe70 rank 18 nranks 32 cudaDev 2 busIdts-cbba87c5e7504a249f512ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82589:83836 [7] NCCL INFO [Service thread] Connectiots-cbba87c5e7504a249f512ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82589:82589 [7] NCCL INFO comm 0x486d57f0 rank 23 nranks 32 cudaDev 7 busIdts-cbba87c5e7504a249f512ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82585:83833 [3] NCCL INFO [Service thread] Connection closed by localRank 3tts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82585:82585 [3] NCCL INFO comm 0x473d79c0 rank 19 nranks 32 cudaDev 3 busIdts-cbba87c5e7504a249f512ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82582:83840 [0] NCCL INFO [Service thread] Connection closed by localRank 0ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74470:75641 [0] NCCL INFO [Service thread] Connection closed by localRank 0 ts-cbba87c5e7504a249f51ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82586:83835 [4] NCCL INFO [Service thread] Connection closed by localRank 4 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82586:82586 [4] NCCL INFO comm 0x488eed80 rank 20 nranks 32 cudaDev 4 busId 93000 - Abort COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82587:83834 [5] NCCL INFO [Service thread] Connection closed by localRank tsts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82587:82587 [5] NCCL INFO comm 0x4860f740 rank 21 nranks 32 cudaDev 5 busIts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:75643 [1] NCCL INFO [Service thread] Connection closed by localRank 1 ts-cbba87c5e7504a249f5127ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:82583:82583 [1] NCCL INFO comm 0x4900cfe0 rank 17 nranks 32 cudaDev 1 busIts-cbba87c5e7504a249f5127103d9ce40f-worker-0:74471:74471 [1] NCCL INFO comm 0x4799dfc0 rank 9 nranks 32 cudaDev 1 busId 13000 - Abort COMPLETE 03d9ce40f-launcher:149854:151312 [5] NCCL INFO [Service thread] Connection closed by localRank 5 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149850:149850 [1] NCCL INFO comm 0x4aae3420 rank 1 nranks 32 cudaDev 1 busId 13000 - Abort COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149854:149854 [5] NCCL INFO comm 0x472ea430 rank 5 nranks 32 cudaDev 5 busId 99000 - Abort COMPLETE k 27 nranks 32 cudaDev 3 busId 51000 - Abort COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54963:56089 [2] NCCL INFO [Service thread] Connection closed by localRank 2 ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54961:56090 [0] NCCL INFO [Service thread] Connection closed by localRank 0 ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54961:54961 [0] NCCL INFO comm 0x481aeb60 rank 24 nranks 32 cudaDev 0 busId e000 - Abort COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54963:54963 [2] NCCL INFO comm 0x48ad2e50 rank 26 nranks 32 cudaDev 2 busId 4b000 - Abort COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54968:56092 [7] NCCL INFO [Service thread] Connection closed by localRank 7 ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:54968:54968 [7] NCCL INFO comm 0x496b48d0 rank 31 nranks 32 cudaDev 7 busId d0000 - Abort COMPLETE [INFO|modeling_utils.py:1767] 2023-11-16 12:25:02,210 >> The model is bigger than the maximum size per checkpoint (10GB) and is going to be split in 2 checkpoint shards. You can find where each parameters has been saved in the index located at /apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20/pytorch_model.bin.index.json. [INFO|tokenization_utils_base.py:2164] 2023-11-16 12:25:02,218 >> tokenizer config file saved in /apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20/tokenizer_config.json [INFO|tokenization_utils_base.py:2171] 2023-11-16 12:25:02,222 >> Special tokens file saved in /apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20/special_tokens_map.json [INFO|tokenization_utils_base.py:2221] 2023-11-16 12:25:02,225 >> added tokens file saved in /apdcephfs/share_733425/vinnylywang/jianhuipang/LLMs4MT/model/newptmodel-llms4mt-zh2en-32a100/llama2-sfton-10000-bitexts-and-alpacagpt4-and-newstests17to20/added_tokens.json ***** train metrics ***** epoch = 3.0 train_loss = 0.9236 train_runtime = 0:41:53.36 train_samples = 75399 train_samples_per_second = 89.997 train_steps_per_second = 0.06 [INFO|modelcard.py:451] 2023-11-16 12:25:02,256 >> Dropping the following result as it does not have all the necessary fields: {'task': {'name': 'Causal Language Modeling', 'type': 'text-generation'}} ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149849:151308 [0] NCCL INFO [Service thread] Connection closed by localRank 0 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:149849:149849 [0] NCCL INFO comm 0x49482970 rank 0 nranks 32 cudaDev 0 busId e000 - Abort COMPLETE