versae's picture
Saving weights and logs of step 1000
9c3de9e
2022-01-14 21:28:55,408 INFO MainThread:5000 [wandb_setup.py:_flush():71] setting env: {}
2022-01-14 21:28:55,408 INFO MainThread:5000 [wandb_setup.py:_flush():71] setting login settings: {}
2022-01-14 21:28:55,408 INFO MainThread:5000 [wandb_init.py:_log_setup():371] Logging user logs to /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/logs/debug.log
2022-01-14 21:28:55,408 INFO MainThread:5000 [wandb_init.py:_log_setup():372] Logging internal logs to /data/roberta-base-ncc/wandb/run-20220114_212855-32qdb4k5/logs/debug-internal.log
2022-01-14 21:28:55,408 INFO MainThread:5000 [wandb_init.py:init():404] calling init triggers
2022-01-14 21:28:55,408 INFO MainThread:5000 [wandb_init.py:init():409] wandb.init called with sweep_config: {}
config: {}
2022-01-14 21:28:55,409 INFO MainThread:5000 [wandb_init.py:init():460] starting backend
2022-01-14 21:28:55,409 INFO MainThread:5000 [backend.py:_multiprocessing_setup():99] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2022-01-14 21:28:55,461 INFO MainThread:5000 [backend.py:ensure_launched():216] starting backend process...
2022-01-14 21:28:55,488 INFO MainThread:5000 [backend.py:ensure_launched():221] started backend process with pid: 8253
2022-01-14 21:28:55,490 INFO MainThread:5000 [wandb_init.py:init():469] backend started and connected
2022-01-14 21:28:55,501 INFO MainThread:5000 [wandb_init.py:init():533] updated telemetry
2022-01-14 21:28:55,563 INFO MainThread:5000 [wandb_init.py:init():563] communicating current version
2022-01-14 21:28:56,351 INFO MainThread:5000 [wandb_init.py:init():568] got version response
2022-01-14 21:28:56,351 INFO MainThread:5000 [wandb_init.py:init():578] communicating run to backend with 30 second timeout
2022-01-14 21:28:56,515 INFO MainThread:5000 [wandb_init.py:init():606] starting run threads in backend
2022-01-14 21:29:01,520 INFO MainThread:5000 [wandb_run.py:_console_start():1810] atexit reg
2022-01-14 21:29:01,520 INFO MainThread:5000 [wandb_run.py:_redirect():1684] redirect: SettingsConsole.REDIRECT
2022-01-14 21:29:01,520 INFO MainThread:5000 [wandb_run.py:_redirect():1689] Redirecting console.
2022-01-14 21:29:01,523 INFO MainThread:5000 [wandb_run.py:_redirect():1745] Redirects installed.
2022-01-14 21:29:01,523 INFO MainThread:5000 [wandb_init.py:init():633] run started, returning control to user process
2022-01-14 21:29:01,523 INFO MainThread:5000 [wandb_run.py:_config_callback():956] config_cb None None {'output_dir': './', 'overwrite_output_dir': True, 'do_train': True, 'do_eval': True, 'per_device_train_batch_size': 250, 'per_device_eval_batch_size': 250, 'learning_rate': 0.0006, 'weight_decay': 0.01, 'adam_beta1': 0.9, 'adam_beta2': 0.98, 'adam_epsilon': 1e-06, 'adafactor': False, 'num_train_epochs': 3.0, 'warmup_steps': 10000, 'logging_steps': 1000, 'save_steps': 1000, 'eval_steps': 1000, 'seed': 42, 'push_to_hub': True, 'hub_model_id': None, 'hub_token': None}
2022-01-14 21:29:01,524 INFO MainThread:5000 [wandb_run.py:_config_callback():956] config_cb None None {'model_name_or_path': None, 'model_type': 'roberta', 'config_name': 'roberta-base', 'tokenizer_name': 'NbAiLab/nb-roberta-base', 'cache_dir': None, 'use_fast_tokenizer': True, 'dtype': 'bfloat16'}
2022-01-14 21:29:01,524 INFO MainThread:5000 [wandb_run.py:_config_callback():956] config_cb None None {'dataset_name': 'NbAiLab/NCC', 'dataset_config_name': None, 'train_file': None, 'validation_file': None, 'train_ref_file': None, 'validation_ref_file': None, 'overwrite_cache': False, 'validation_split_percentage': 5, 'max_seq_length': 128, 'preprocessing_num_workers': None, 'mlm_probability': 0.15, 'pad_to_max_length': True, 'line_by_line': False}
2022-01-14 21:32:14,189 INFO MainThread:5000 [wandb_run.py:_atexit_cleanup():1780] got exitcode: 1
2022-01-14 21:32:14,192 INFO MainThread:5000 [wandb_run.py:_restore():1752] restore
2022-01-14 21:32:16,895 INFO MainThread:5000 [wandb_run.py:_wait_for_finish():1912] got exit ret: file_counts {
wandb_count: 1
other_count: 1
}
pusher_stats {
uploaded_bytes: 37446
total_bytes: 37446
}
2022-01-14 21:32:17,686 INFO MainThread:5000 [wandb_run.py:_wait_for_finish():1912] got exit ret: file_counts {
wandb_count: 1
other_count: 1
}
pusher_stats {
uploaded_bytes: 37446
total_bytes: 37446
}
2022-01-14 21:32:18,657 INFO MainThread:5000 [wandb_run.py:_wait_for_finish():1912] got exit ret: file_counts {
wandb_count: 5
other_count: 1
}
pusher_stats {
uploaded_bytes: 37446
total_bytes: 45931
}
2022-01-14 21:32:18,759 INFO MainThread:5000 [wandb_run.py:_wait_for_finish():1912] got exit ret: file_counts {
wandb_count: 5
other_count: 1
}
pusher_stats {
uploaded_bytes: 37446
total_bytes: 45931
}
2022-01-14 21:32:18,861 INFO MainThread:5000 [wandb_run.py:_wait_for_finish():1912] got exit ret: file_counts {
wandb_count: 5
other_count: 1
}
pusher_stats {
uploaded_bytes: 45903
total_bytes: 45931
}
2022-01-14 21:32:18,962 INFO MainThread:5000 [wandb_run.py:_wait_for_finish():1912] got exit ret: file_counts {
wandb_count: 5
other_count: 1
}
pusher_stats {
uploaded_bytes: 45903
total_bytes: 45931
}
2022-01-14 21:32:19,064 INFO MainThread:5000 [wandb_run.py:_wait_for_finish():1912] got exit ret: file_counts {
wandb_count: 5
other_count: 1
}
pusher_stats {
uploaded_bytes: 45931
total_bytes: 45931
}
2022-01-14 21:32:19,166 INFO MainThread:5000 [wandb_run.py:_wait_for_finish():1912] got exit ret: file_counts {
wandb_count: 5
other_count: 1
}
pusher_stats {
uploaded_bytes: 45931
total_bytes: 45931
}
2022-01-14 21:32:19,269 INFO MainThread:5000 [wandb_run.py:_wait_for_finish():1912] got exit ret: file_counts {
wandb_count: 5
other_count: 1
}
pusher_stats {
uploaded_bytes: 45931
total_bytes: 45931
}
2022-01-14 21:32:19,371 INFO MainThread:5000 [wandb_run.py:_wait_for_finish():1912] got exit ret: file_counts {
wandb_count: 5
other_count: 1
}
pusher_stats {
uploaded_bytes: 45931
total_bytes: 45931
}
2022-01-14 21:32:19,473 INFO MainThread:5000 [wandb_run.py:_wait_for_finish():1912] got exit ret: file_counts {
wandb_count: 5
other_count: 1
}
pusher_stats {
uploaded_bytes: 45931
total_bytes: 45931
}
2022-01-14 21:32:19,917 INFO MainThread:5000 [wandb_run.py:_wait_for_finish():1912] got exit ret: file_counts {
wandb_count: 5
other_count: 1
}
pusher_stats {
uploaded_bytes: 45931
total_bytes: 45931
}
2022-01-14 21:32:20,277 INFO MainThread:5000 [wandb_run.py:_wait_for_finish():1912] got exit ret: done: true
exit_result {
}
file_counts {
wandb_count: 5
other_count: 1
}
pusher_stats {
uploaded_bytes: 45931
total_bytes: 45931
}
local_info {
}
2022-01-14 21:32:23,445 INFO MainThread:5000 [wandb_run.py:_append_files():2180] logging synced files