File size: 9,981 Bytes
872c781 735daf5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 |
2021-07-09 14:14:45,372 INFO MainThread:245719 [wandb_setup.py:_flush():69] Unhandled environment var: WANDB_LOG_MODEL
2021-07-09 14:14:45,372 INFO MainThread:245719 [wandb_setup.py:_flush():69] setting env: {'project': 'hf-flax-gpt2-indonesian', 'entity': 'wandb'}
2021-07-09 14:14:45,372 INFO MainThread:245719 [wandb_setup.py:_flush():69] setting login settings: {}
2021-07-09 14:14:45,372 INFO MainThread:245719 [wandb_init.py:_log_setup():337] Logging user logs to /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/logs/debug.log
2021-07-09 14:14:45,372 INFO MainThread:245719 [wandb_init.py:_log_setup():338] Logging internal logs to /home/cahya/Work/flax-community/gpt2-medium-indonesian/wandb/run-20210709_141445-2k8cnty2/logs/debug-internal.log
2021-07-09 14:14:45,372 INFO MainThread:245719 [wandb_init.py:init():370] calling init triggers
2021-07-09 14:14:45,372 INFO MainThread:245719 [wandb_init.py:init():375] wandb.init called with sweep_config: {}
config: {}
2021-07-09 14:14:45,373 INFO MainThread:245719 [wandb_init.py:init():419] starting backend
2021-07-09 14:14:45,373 INFO MainThread:245719 [backend.py:_multiprocessing_setup():70] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2021-07-09 14:14:45,413 INFO MainThread:245719 [backend.py:ensure_launched():135] starting backend process...
2021-07-09 14:14:45,453 INFO MainThread:245719 [backend.py:ensure_launched():139] started backend process with pid: 246776
2021-07-09 14:14:45,455 INFO MainThread:245719 [wandb_init.py:init():424] backend started and connected
2021-07-09 14:14:45,458 INFO MainThread:245719 [wandb_init.py:init():472] updated telemetry
2021-07-09 14:14:45,458 INFO MainThread:245719 [wandb_init.py:init():491] communicating current version
2021-07-09 14:14:46,105 INFO MainThread:245719 [wandb_init.py:init():496] got version response
2021-07-09 14:14:46,105 INFO MainThread:245719 [wandb_init.py:init():504] communicating run to backend with 30 second timeout
2021-07-09 14:14:46,291 INFO MainThread:245719 [wandb_init.py:init():529] starting run threads in backend
2021-07-09 14:14:49,944 INFO MainThread:245719 [wandb_run.py:_console_start():1623] atexit reg
2021-07-09 14:14:49,944 INFO MainThread:245719 [wandb_run.py:_redirect():1497] redirect: SettingsConsole.REDIRECT
2021-07-09 14:14:49,945 INFO MainThread:245719 [wandb_run.py:_redirect():1502] Redirecting console.
2021-07-09 14:14:49,946 INFO MainThread:245719 [wandb_run.py:_redirect():1558] Redirects installed.
2021-07-09 14:14:49,947 INFO MainThread:245719 [wandb_init.py:init():554] run started, returning control to user process
2021-07-09 14:14:49,956 INFO MainThread:245719 [wandb_run.py:_config_callback():872] config_cb None None {'output_dir': '/home/cahya/Work/flax-community/gpt2-medium-indonesian', 'overwrite_output_dir': True, 'do_train': True, 'do_eval': True, 'do_predict': False, 'evaluation_strategy': 'IntervalStrategy.NO', 'prediction_loss_only': False, 'per_device_train_batch_size': 24, 'per_device_eval_batch_size': 24, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 1, 'eval_accumulation_steps': None, 'learning_rate': 0.0024, 'weight_decay': 0.01, 'adam_beta1': 0.9, 'adam_beta2': 0.98, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 20.0, 'max_steps': -1, 'lr_scheduler_type': 'SchedulerType.LINEAR', 'warmup_ratio': 0.0, 'warmup_steps': 1000, 'log_level': -1, 'log_level_replica': -1, 'log_on_each_node': True, 'logging_dir': '/home/cahya/Work/flax-community/gpt2-medium-indonesian/runs/Jul09_14-14-49_t1v-n-528d9406-w-0', 'logging_strategy': 'IntervalStrategy.STEPS', 'logging_first_step': False, 'logging_steps': 500, 'save_strategy': 'IntervalStrategy.STEPS', 'save_steps': 10, 'save_total_limit': None, 'save_on_each_node': False, 'no_cuda': False, 'seed': 42, 'fp16': False, 'fp16_opt_level': 'O1', 'fp16_backend': 'auto', 'fp16_full_eval': False, 'local_rank': -1, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': 10, 'dataloader_num_workers': 64, 'past_index': -1, 'run_name': '/home/cahya/Work/flax-community/gpt2-medium-indonesian', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'sharded_ddp': [], 'deepspeed': None, 'label_smoothing_factor': 0.0, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['tensorboard', 'wandb'], 'ddp_find_unused_parameters': None, 'dataloader_pin_memory': True, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': True, 'resume_from_checkpoint': None, 'push_to_hub_model_id': 'gpt2-medium-indonesian', 'push_to_hub_organization': None, 'push_to_hub_token': None, 'mp_parameters': '', '_n_gpu': 0, '__cached__setup_devices': 'cpu'}
2021-07-09 14:14:49,957 INFO MainThread:245719 [wandb_run.py:_config_callback():872] config_cb None None {'model_name_or_path': None, 'model_type': 'gpt2', 'config_name': '/home/cahya/Work/flax-community/gpt2-medium-indonesian', 'tokenizer_name': '/home/cahya/Work/flax-community/gpt2-medium-indonesian', 'cache_dir': None, 'use_fast_tokenizer': True, 'dtype': 'float32'}
2021-07-09 14:14:49,958 INFO MainThread:245719 [wandb_run.py:_config_callback():872] config_cb None None {'dataset_name': 'oscar', 'dataset_config_name': 'unshuffled_deduplicated_id', 'train_file': None, 'validation_file': None, 'max_train_samples': 10000, 'max_eval_samples': 1000, 'overwrite_cache': False, 'validation_split_percentage': 5, 'block_size': 512, 'preprocessing_num_workers': 64}
2021-07-09 14:14:49,958 INFO MainThread:245719 [wandb_config.py:__setitem__():141] config set test_log = 12345 - <bound method Run._config_callback of <wandb.sdk.wandb_run.Run object at 0x7fab4d59abb0>>
2021-07-09 14:14:49,958 INFO MainThread:245719 [wandb_run.py:_config_callback():872] config_cb test_log 12345 None
2021-07-09 14:15:27,424 INFO MainThread:245719 [wandb_run.py:_tensorboard_callback():943] tensorboard callback: /home/cahya/Work/flax-community/gpt2-medium-indonesian, None
2021-07-09 14:30:34,276 INFO MainThread:245719 [wandb_run.py:_atexit_cleanup():1593] got exitcode: 255
2021-07-09 14:30:34,276 INFO MainThread:245719 [wandb_run.py:_restore():1565] restore
2021-07-09 14:30:36,624 INFO MainThread:245719 [wandb_run.py:_wait_for_finish():1715] got exit ret: file_counts {
wandb_count: 1
other_count: 1
}
pusher_stats {
uploaded_bytes: 3022
total_bytes: 3022
}
2021-07-09 14:30:41,726 INFO MainThread:245719 [wandb_run.py:_wait_for_finish():1715] got exit ret: None
2021-07-09 14:30:42,963 INFO MainThread:245719 [wandb_run.py:_wait_for_finish():1715] got exit ret: file_counts {
wandb_count: 1
other_count: 1
}
pusher_stats {
uploaded_bytes: 3022
total_bytes: 3022
}
2021-07-09 14:30:43,171 INFO MainThread:245719 [wandb_run.py:_wait_for_finish():1715] got exit ret: file_counts {
wandb_count: 1
other_count: 1
}
pusher_stats {
uploaded_bytes: 3022
total_bytes: 3022
}
2021-07-09 14:30:43,697 INFO MainThread:245719 [wandb_run.py:_wait_for_finish():1715] got exit ret: file_counts {
wandb_count: 5
other_count: 1
}
pusher_stats {
uploaded_bytes: 3022
total_bytes: 100895
}
2021-07-09 14:30:43,799 INFO MainThread:245719 [wandb_run.py:_wait_for_finish():1715] got exit ret: file_counts {
wandb_count: 6
other_count: 1
}
pusher_stats {
uploaded_bytes: 1722
total_bytes: 101169
}
2021-07-09 14:30:43,901 INFO MainThread:245719 [wandb_run.py:_wait_for_finish():1715] got exit ret: file_counts {
wandb_count: 6
other_count: 1
}
pusher_stats {
uploaded_bytes: 101169
total_bytes: 101169
}
2021-07-09 14:30:44,003 INFO MainThread:245719 [wandb_run.py:_wait_for_finish():1715] got exit ret: file_counts {
wandb_count: 6
other_count: 1
}
pusher_stats {
uploaded_bytes: 101169
total_bytes: 101169
}
2021-07-09 14:30:44,104 INFO MainThread:245719 [wandb_run.py:_wait_for_finish():1715] got exit ret: file_counts {
wandb_count: 6
other_count: 1
}
pusher_stats {
uploaded_bytes: 101169
total_bytes: 101169
}
2021-07-09 14:30:44,206 INFO MainThread:245719 [wandb_run.py:_wait_for_finish():1715] got exit ret: file_counts {
wandb_count: 6
other_count: 1
}
pusher_stats {
uploaded_bytes: 101169
total_bytes: 101169
}
2021-07-09 14:30:44,308 INFO MainThread:245719 [wandb_run.py:_wait_for_finish():1715] got exit ret: file_counts {
wandb_count: 6
other_count: 1
}
pusher_stats {
uploaded_bytes: 101169
total_bytes: 101169
}
2021-07-09 14:30:44,410 INFO MainThread:245719 [wandb_run.py:_wait_for_finish():1715] got exit ret: file_counts {
wandb_count: 6
other_count: 1
}
pusher_stats {
uploaded_bytes: 101169
total_bytes: 101169
}
2021-07-09 14:30:44,511 INFO MainThread:245719 [wandb_run.py:_wait_for_finish():1715] got exit ret: file_counts {
wandb_count: 6
other_count: 1
}
pusher_stats {
uploaded_bytes: 101169
total_bytes: 101169
}
2021-07-09 14:30:44,832 INFO MainThread:245719 [wandb_run.py:_wait_for_finish():1715] got exit ret: file_counts {
wandb_count: 6
other_count: 1
}
pusher_stats {
uploaded_bytes: 101169
total_bytes: 101169
}
2021-07-09 14:30:44,934 INFO MainThread:245719 [wandb_run.py:_wait_for_finish():1715] got exit ret: done: true
exit_result {
}
file_counts {
wandb_count: 6
other_count: 1
}
pusher_stats {
uploaded_bytes: 101169
total_bytes: 101169
}
2021-07-09 14:30:46,219 INFO MainThread:245719 [wandb_run.py:_show_summary():1870] rendering summary
2021-07-09 14:30:46,220 INFO MainThread:245719 [wandb_run.py:_show_history():1908] rendering history
2021-07-09 14:30:46,220 INFO MainThread:245719 [wandb_run.py:_show_files():1937] logging synced files
|