File size: 15,103 Bytes
da5860c 9ab0742 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 |
2022-03-02 00:03:00,882 INFO MainThread:249774 [wandb_setup.py:_flush():75] Loading settings from /home/sanchit_huggingface_co/.config/wandb/settings
2022-03-02 00:03:00,883 INFO MainThread:249774 [wandb_setup.py:_flush():75] Loading settings from /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/settings
2022-03-02 00:03:00,883 INFO MainThread:249774 [wandb_setup.py:_flush():75] Loading settings from environment variables: {}
2022-03-02 00:03:00,883 INFO MainThread:249774 [wandb_setup.py:_flush():75] Inferring run settings from compute environment: {'program_relpath': 'run_speech_recognition_seq2seq.py', 'program': '/home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/run_speech_recognition_seq2seq.py'}
2022-03-02 00:03:00,883 INFO MainThread:249774 [wandb_init.py:_log_setup():386] Logging user logs to /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_000300-2ebwk6gp/logs/debug.log
2022-03-02 00:03:00,883 INFO MainThread:249774 [wandb_init.py:_log_setup():387] Logging internal logs to /home/sanchit_huggingface_co/wav2vec2-gpt2-wandb-grid-search/wandb/run-20220302_000300-2ebwk6gp/logs/debug-internal.log
2022-03-02 00:03:00,883 INFO MainThread:249774 [wandb_init.py:init():420] calling init triggers
2022-03-02 00:03:00,883 INFO MainThread:249774 [wandb_init.py:init():425] wandb.init called with sweep_config: {}
config: {}
2022-03-02 00:03:00,883 INFO MainThread:249774 [wandb_init.py:init():471] starting backend
2022-03-02 00:03:00,883 INFO MainThread:249774 [backend.py:_multiprocessing_setup():99] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2022-03-02 00:03:00,940 INFO MainThread:249774 [backend.py:ensure_launched():219] starting backend process...
2022-03-02 00:03:00,997 INFO MainThread:249774 [backend.py:ensure_launched():224] started backend process with pid: 249873
2022-03-02 00:03:00,999 INFO MainThread:249774 [wandb_init.py:init():480] backend started and connected
2022-03-02 00:03:01,009 INFO MainThread:249774 [wandb_init.py:init():550] updated telemetry
2022-03-02 00:03:01,146 INFO MainThread:249774 [wandb_init.py:init():581] communicating current version
2022-03-02 00:03:01,853 INFO MainThread:249774 [wandb_init.py:init():586] got version response upgrade_message: "wandb version 0.12.11 is available! To upgrade, please run:\n $ pip install wandb --upgrade"
2022-03-02 00:03:01,853 INFO MainThread:249774 [wandb_init.py:init():596] communicating run to backend with 30 second timeout
2022-03-02 00:03:02,050 INFO MainThread:249774 [wandb_init.py:init():624] starting run threads in backend
2022-03-02 00:03:02,159 INFO MainThread:249774 [wandb_run.py:_console_start():1827] atexit reg
2022-03-02 00:03:02,159 INFO MainThread:249774 [wandb_run.py:_redirect():1701] redirect: SettingsConsole.REDIRECT
2022-03-02 00:03:02,160 INFO MainThread:249774 [wandb_run.py:_redirect():1706] Redirecting console.
2022-03-02 00:03:02,161 INFO MainThread:249774 [wandb_run.py:_redirect():1762] Redirects installed.
2022-03-02 00:03:02,161 INFO MainThread:249774 [wandb_init.py:init():651] run started, returning control to user process
2022-03-02 00:03:02,165 INFO MainThread:249774 [wandb_run.py:_config_callback():966] config_cb None None {'return_dict': True, 'output_hidden_states': False, 'output_attentions': False, 'torchscript': False, 'torch_dtype': 'torch.float32', 'use_bfloat16': False, 'pruned_heads': {}, 'tie_word_embeddings': False, 'is_encoder_decoder': True, 'is_decoder': False, 'cross_attention_hidden_size': None, 'add_cross_attention': False, 'tie_encoder_decoder': False, 'max_length': 50, 'min_length': 0, 'do_sample': False, 'early_stopping': False, 'num_beams': 1, 'num_beam_groups': 1, 'diversity_penalty': 0.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0, 'repetition_penalty': 1.0, 'length_penalty': 1.0, 'no_repeat_ngram_size': 0, 'encoder_no_repeat_ngram_size': 0, 'bad_words_ids': None, 'num_return_sequences': 1, 'chunk_size_feed_forward': 0, 'output_scores': False, 'return_dict_in_generate': False, 'forced_bos_token_id': None, 'forced_eos_token_id': None, 'remove_invalid_values': False, 'architectures': ['SpeechEncoderDecoderModel'], 'finetuning_task': None, 'id2label': {0: 'LABEL_0', 1: 'LABEL_1'}, 'label2id': {'LABEL_0': 0, 'LABEL_1': 1}, 'tokenizer_class': None, 'prefix': None, 'bos_token_id': None, 'pad_token_id': 50256, 'eos_token_id': 50256, 'sep_token_id': None, 'decoder_start_token_id': 50256, 'task_specific_params': None, 'problem_type': None, '_name_or_path': './', 'transformers_version': None, 'decoder': {'vocab_size': 50257, 'n_positions': 1024, 'n_embd': 1024, 'n_layer': 24, 'n_head': 16, 'n_inner': None, 'activation_function': 'gelu_new', 'resid_pdrop': 0.0, 'embd_pdrop': 0.0, 'attn_pdrop': 0.0, 'layer_norm_epsilon': 1e-05, 'initializer_range': 0.02, 'summary_type': 'cls_index', 'summary_use_proj': True, 'summary_activation': None, 'summary_first_dropout': 0.0, 'summary_proj_to_labels': True, 'scale_attn_weights': True, 'use_cache': False, 'scale_attn_by_inverse_layer_idx': False, 'reorder_and_upcast_attn': False, 'bos_token_id': 50256, 'eos_token_id': 50256, 'return_dict': True, 'output_hidden_states': False, 'output_attentions': False, 'torchscript': False, 'torch_dtype': None, 'use_bfloat16': False, 'pruned_heads': {}, 'tie_word_embeddings': True, 'is_encoder_decoder': False, 'is_decoder': True, 'cross_attention_hidden_size': None, 'add_cross_attention': True, 'tie_encoder_decoder': False, 'max_length': 20, 'min_length': 0, 'do_sample': False, 'early_stopping': False, 'num_beams': 1, 'num_beam_groups': 1, 'diversity_penalty': 0.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0, 'repetition_penalty': 1.0, 'length_penalty': 1.0, 'no_repeat_ngram_size': 0, 'encoder_no_repeat_ngram_size': 0, 'bad_words_ids': None, 'num_return_sequences': 1, 'chunk_size_feed_forward': 0, 'output_scores': False, 'return_dict_in_generate': False, 'forced_bos_token_id': None, 'forced_eos_token_id': None, 'remove_invalid_values': False, 'architectures': ['GPT2LMHeadModel'], 'finetuning_task': None, 'id2label': {0: 'LABEL_0', 1: 'LABEL_1'}, 'label2id': {'LABEL_0': 0, 'LABEL_1': 1}, 'tokenizer_class': None, 'prefix': None, 'pad_token_id': None, 'sep_token_id': None, 'decoder_start_token_id': None, 'task_specific_params': {'text-generation': {'do_sample': True, 'max_length': 50}}, 'problem_type': None, '_name_or_path': 'gpt2-medium', 'transformers_version': '4.17.0.dev0', 'n_ctx': 1024, 'n_special': 0, 'predict_special_tokens': True, 'model_type': 'gpt2'}, 'encoder': {'return_dict': True, 'output_hidden_states': False, 'output_attentions': False, 'torchscript': False, 'torch_dtype': None, 'use_bfloat16': False, 'pruned_heads': {}, 'tie_word_embeddings': True, 'is_encoder_decoder': False, 'is_decoder': False, 'cross_attention_hidden_size': None, 'add_cross_attention': False, 'tie_encoder_decoder': False, 'max_length': 20, 'min_length': 0, 'do_sample': False, 'early_stopping': False, 'num_beams': 1, 'num_beam_groups': 1, 'diversity_penalty': 0.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0, 'repetition_penalty': 1.0, 'length_penalty': 1.0, 'no_repeat_ngram_size': 0, 'encoder_no_repeat_ngram_size': 0, 'bad_words_ids': None, 'num_return_sequences': 1, 'chunk_size_feed_forward': 0, 'output_scores': False, 'return_dict_in_generate': False, 'forced_bos_token_id': None, 'forced_eos_token_id': None, 'remove_invalid_values': False, 'architectures': ['Wav2Vec2ForPreTraining'], 'finetuning_task': None, 'id2label': {0: 'LABEL_0', 1: 'LABEL_1'}, 'label2id': {'LABEL_0': 0, 'LABEL_1': 1}, 'tokenizer_class': None, 'prefix': None, 'bos_token_id': 1, 'pad_token_id': 0, 'eos_token_id': 2, 'sep_token_id': None, 'decoder_start_token_id': None, 'task_specific_params': None, 'problem_type': None, '_name_or_path': 'facebook/wav2vec2-large-lv60', 'transformers_version': '4.17.0.dev0', 'feat_extract_dropout': 0.0, 'gradient_checkpointing': False, 'hidden_dropout_prob': 0.0, 'num_feat_extract_layers': 7, 'hidden_size': 1024, 'feat_extract_norm': 'layer', 'feat_extract_activation': 'gelu', 'conv_dim': [512, 512, 512, 512, 512, 512, 512], 'conv_stride': [5, 2, 2, 2, 2, 2, 2], 'conv_kernel': [10, 3, 3, 3, 3, 2, 2], 'conv_bias': True, 'num_conv_pos_embeddings': 128, 'num_conv_pos_embedding_groups': 16, 'num_hidden_layers': 24, 'intermediate_size': 4096, 'hidden_act': 'gelu', 'num_attention_heads': 16, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'activation_dropout': 0.0, 'feat_proj_dropout': 0.0, 'final_dropout': 0.0, 'layerdrop': 0.0, 'layer_norm_eps': 1e-05, 'initializer_range': 0.02, 'vocab_size': 32, 'do_stable_layer_norm': True, 'use_weighted_layer_sum': False, 'apply_spec_augment': False, 'mask_time_prob': 0.0, 'mask_time_length': 10, 'mask_time_min_masks': 2, 'mask_feature_prob': 0.0, 'mask_feature_length': 10, 'mask_feature_min_masks': 0, 'num_codevectors_per_group': 320, 'num_codevector_groups': 2, 'contrastive_logits_temperature': 0.1, 'feat_quantizer_dropout': 0.0, 'num_negatives': 100, 'codevector_dim': 768, 'proj_codevector_dim': 768, 'diversity_loss_weight': 0.1, 'ctc_loss_reduction': 'sum', 'ctc_zero_infinity': False, 'add_adapter': True, 'adapter_kernel_size': 3, 'adapter_stride': 2, 'num_adapter_layers': 3, 'output_hidden_size': 1024, 'classifier_proj_size': 256, 'tdnn_dim': [512, 512, 512, 512, 1500], 'tdnn_kernel': [5, 3, 3, 1, 1], 'tdnn_dilation': [1, 2, 3, 1, 1], 'xvector_output_dim': 512, 'model_type': 'wav2vec2'}, 'model_type': 'speech-encoder-decoder', 'processor_class': 'Wav2Vec2Processor', 'use_cache': False, 'output_dir': './', 'overwrite_output_dir': True, 'do_train': True, 'do_eval': True, 'do_predict': False, 'evaluation_strategy': 'steps', 'prediction_loss_only': False, 'per_device_train_batch_size': 12, 'per_device_eval_batch_size': 12, 'per_gpu_train_batch_size': 'None', 'per_gpu_eval_batch_size': 'None', 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': 'None', 'learning_rate': 0.0001, 'weight_decay': 0.0, 'adam_beta1': 0.9, 'adam_beta2': 0.999, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 1.0, 'max_steps': -1, 'lr_scheduler_type': 'linear', 'warmup_ratio': 0.0, 'warmup_steps': 500, 'log_level': -1, 'log_level_replica': -1, 'log_on_each_node': True, 'logging_dir': './runs/Mar02_00-02-19_sanchit--v100', 'logging_strategy': 'steps', 'logging_first_step': False, 'logging_steps': 1, 'logging_nan_inf_filter': True, 'save_strategy': 'steps', 'save_steps': 500, 'save_total_limit': 1, 'save_on_each_node': False, 'no_cuda': False, 'seed': 42, 'bf16': False, 'fp16': True, 'fp16_opt_level': 'O1', 'half_precision_backend': 'amp', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': 'None', 'local_rank': -1, 'xpu_backend': 'None', 'tpu_num_cores': 'None', 'tpu_metrics_debug': False, 'debug': '[]', 'dataloader_drop_last': False, 'eval_steps': 500, 'dataloader_num_workers': 0, 'past_index': -1, 'run_name': './', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': 'None', 'load_best_model_at_end': False, 'metric_for_best_model': 'None', 'greater_is_better': 'None', 'ignore_data_skip': False, 'sharded_ddp': '[]', 'deepspeed': 'None', 'label_smoothing_factor': 0.0, 'optim': 'adamw_hf', 'adafactor': False, 'group_by_length': True, 'length_column_name': 'input_length', 'report_to': "['wandb']", 'ddp_find_unused_parameters': 'None', 'ddp_bucket_cap_mb': 'None', 'dataloader_pin_memory': True, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': True, 'resume_from_checkpoint': 'None', 'hub_model_id': 'None', 'hub_strategy': 'every_save', 'hub_token': '<HUB_TOKEN>', 'gradient_checkpointing': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': 'None', 'push_to_hub_organization': 'None', 'push_to_hub_token': '<PUSH_TO_HUB_TOKEN>', '_n_gpu': 1, 'mp_parameters': '', 'sortish_sampler': False, 'predict_with_generate': True, 'generation_max_length': 40, 'generation_num_beams': 1, 'train_batch_size': 12, 'eval_batch_size': 12}
2022-03-02 00:03:02,170 INFO MainThread:249774 [wandb_watch.py:watch():43] Watching
2022-03-02 02:14:56,039 INFO MainThread:249774 [wandb_run.py:_atexit_cleanup():1797] got exitcode: 1
2022-03-02 02:14:56,042 INFO MainThread:249774 [wandb_run.py:_restore():1769] restore
2022-03-02 02:14:58,465 INFO MainThread:249774 [wandb_run.py:_wait_for_finish():1929] got exit ret: file_counts {
wandb_count: 1
}
pusher_stats {
uploaded_bytes: 2095
total_bytes: 2095
}
2022-03-02 02:14:58,644 INFO MainThread:249774 [wandb_run.py:_wait_for_finish():1929] got exit ret: file_counts {
wandb_count: 1
}
pusher_stats {
uploaded_bytes: 2095
total_bytes: 2095
}
2022-03-02 02:14:59,473 INFO MainThread:249774 [wandb_run.py:_wait_for_finish():1929] got exit ret: file_counts {
wandb_count: 1
}
pusher_stats {
uploaded_bytes: 2095
total_bytes: 2095
}
2022-03-02 02:14:59,768 INFO MainThread:249774 [wandb_run.py:_wait_for_finish():1929] got exit ret: file_counts {
wandb_count: 3
}
pusher_stats {
uploaded_bytes: 2095
total_bytes: 3106931
}
2022-03-02 02:14:59,870 INFO MainThread:249774 [wandb_run.py:_wait_for_finish():1929] got exit ret: file_counts {
wandb_count: 5
}
pusher_stats {
uploaded_bytes: 1455316
total_bytes: 3408335
}
2022-03-02 02:14:59,971 INFO MainThread:249774 [wandb_run.py:_wait_for_finish():1929] got exit ret: file_counts {
wandb_count: 5
}
pusher_stats {
uploaded_bytes: 1950305
total_bytes: 3408335
}
2022-03-02 02:15:00,073 INFO MainThread:249774 [wandb_run.py:_wait_for_finish():1929] got exit ret: file_counts {
wandb_count: 5
}
pusher_stats {
uploaded_bytes: 3408335
total_bytes: 3408335
}
2022-03-02 02:15:00,175 INFO MainThread:249774 [wandb_run.py:_wait_for_finish():1929] got exit ret: file_counts {
wandb_count: 5
}
pusher_stats {
uploaded_bytes: 3408335
total_bytes: 3408335
}
2022-03-02 02:15:00,276 INFO MainThread:249774 [wandb_run.py:_wait_for_finish():1929] got exit ret: file_counts {
wandb_count: 5
}
pusher_stats {
uploaded_bytes: 3408335
total_bytes: 3408335
}
2022-03-02 02:15:00,378 INFO MainThread:249774 [wandb_run.py:_wait_for_finish():1929] got exit ret: file_counts {
wandb_count: 5
}
pusher_stats {
uploaded_bytes: 3408335
total_bytes: 3408335
}
2022-03-02 02:15:01,873 INFO MainThread:249774 [wandb_run.py:_wait_for_finish():1929] got exit ret: file_counts {
wandb_count: 5
}
pusher_stats {
uploaded_bytes: 3408335
total_bytes: 3408335
}
2022-03-02 02:15:02,026 INFO MainThread:249774 [wandb_run.py:_wait_for_finish():1929] got exit ret: done: true
exit_result {
}
file_counts {
wandb_count: 5
}
pusher_stats {
uploaded_bytes: 3408335
total_bytes: 3408335
}
local_info {
}
2022-03-02 02:15:03,170 INFO MainThread:249774 [wandb_run.py:_append_history():2144] rendering history
2022-03-02 02:15:03,171 INFO MainThread:249774 [wandb_run.py:_append_summary():2102] rendering summary
2022-03-02 02:15:03,172 INFO MainThread:249774 [wandb_run.py:_append_files():2194] logging synced files
|