| wandb: Currently logged in as: sanchit-gandhi (use `wandb login --relogin` to force relogin) | |
| wandb: wandb version 0.12.17 is available! To upgrade, please run: | |
| wandb: $ pip install wandb --upgrade | |
| wandb: Tracking run with wandb version 0.12.15 | |
| wandb: Run data is saved locally in /home/sanchitgandhi/flax-wav2vec2-2-bart-large-ls-960h-feature-encoder/wandb/run-20220530_175928-2nzi0chc | |
| wandb: Run `wandb offline` to turn off syncing. | |
| wandb: Syncing run flax-wav2vec2-2-bart-large-ls-960h-feature-encoder | |
| wandb: ⭐️ View project at https://wandb.ai/sanchit-gandhi/librispeech_960h | |
| wandb: 🚀 View run at https://wandb.ai/sanchit-gandhi/librispeech_960h/runs/2nzi0chc | |
| 05/30/2022 17:59:29 - INFO - __main__ - Training/evaluation parameters FlaxSeq2SeqTrainingArguments( | |
| _n_gpu=-1, | |
| adafactor=False, | |
| adam_beta1=0.9, | |
| adam_beta2=0.999, | |
| adam_epsilon=1e-08, | |
| bf16=False, | |
| bf16_full_eval=False, | |
| data_seed=None, | |
| dataloader_drop_last=False, | |
| dataloader_num_workers=0, | |
| dataloader_pin_memory=True, | |
| ddp_bucket_cap_mb=None, | |
| ddp_find_unused_parameters=None, | |
| debug=, | |
| deepspeed=None, | |
| disable_tqdm=None, | |
| do_eval=True, | |
| do_predict=True, | |
| do_train=True, | |
| eval_accumulation_steps=None, | |
| eval_delay=0, | |
| eval_steps=10000, | |
| evaluation_strategy=no, | |
| final_generation_max_length=200, | |
| final_generation_num_beams=5, | |
| fp16=False, | |
| fp16_backend=auto, | |
| fp16_full_eval=False, | |
| fp16_opt_level=O1, | |
| generation_length_penalty=1.2, | |
| generation_max_length=40, | |
| generation_num_beams=1, | |
| gradient_accumulation_steps=2, | |
| gradient_checkpointing=True, | |
| greater_is_better=None, | |
| group_by_length=False, | |
| half_precision_backend=auto, | |
| hub_model_id=None, | |
| hub_strategy=every_save, | |
| hub_token=<HUB_TOKEN>, | |
| ignore_data_skip=False, | |
| label_names=None, | |
| label_smoothing_factor=0.0, | |
| learning_rate=0.0001, | |
| length_column_name=input_length, | |
| load_best_model_at_end=False, | |
| local_rank=-1, | |
| log_level=passive, | |
| log_level_replica=passive, | |
| log_on_each_node=True, | |
| logging_dir=None, | |
| logging_first_step=False, | |
| logging_nan_inf_filter=True, | |
| logging_steps=25, | |
| logging_strategy=steps, | |
| lr_scheduler_type=linear, | |
| matmul_precision=default, | |
| max_grad_norm=1.0, | |
| max_steps=50000, | |
| metric_for_best_model=None, | |
| mp_parameters=, | |
| no_cuda=False, | |
| num_train_epochs=3.0, | |
| optim=adamw_hf, | |
| output_dir=./, | |
| overwrite_output_dir=True, | |
| past_index=-1, | |
| per_device_eval_batch_size=4, | |
| per_device_train_batch_size=4, | |
| precision=full, | |
| predict_with_generate=True, | |
| prediction_loss_only=False, | |
| push_to_hub=True, | |
| push_to_hub_model_id=None, | |
| push_to_hub_organization=None, | |
| push_to_hub_token=<PUSH_TO_HUB_TOKEN>, | |
| remove_unused_columns=True, | |
| report_to=None, | |
| resume_from_checkpoint=None, | |
| run_name=None, | |
| save_on_each_node=False, | |
| save_steps=10000, | |
| save_strategy=steps, | |
| save_total_limit=1, | |
| seed=42, | |
| sharded_ddp=, | |
| skip_memory_metrics=True, | |
| sortish_sampler=False, | |
| tf32=None, | |
| tpu_metrics_debug=False, | |
| tpu_num_cores=None, | |
| use_legacy_prediction_loop=False, | |
| warmup_ratio=0.0, | |
| warmup_steps=500, | |
| weight_decay=0.0, | |
| xpu_backend=None, | |
| ) | |
| 05/30/2022 17:59:29 - INFO - __main__ - JAX devices: 8, matmul precision: default | |
| 05/30/2022 17:59:30 - WARNING - datasets.builder - Reusing dataset librispeech_asr (/home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb) | |
| 05/30/2022 17:59:30 - WARNING - datasets.builder - Reusing dataset librispeech_asr (/home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb) | |
| 05/30/2022 17:59:30 - WARNING - datasets.builder - Reusing dataset librispeech_asr (/home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb) | |
| 05/30/2022 17:59:31 - WARNING - datasets.builder - Reusing dataset librispeech_asr (/home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb) | |
| 05/30/2022 17:59:31 - WARNING - datasets.builder - Reusing dataset librispeech_asr (/home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb) | |
| loading configuration file https://huggingface.co/sanchit-gandhi/flax-wav2vec2-2-bart-large-scan/resolve/main/config.json from cache at /home/sanchitgandhi/.cache/huggingface/transformers/e6d3af8a2b6624d8adf8fc289717c121400164223b3e51d49b639aa34d1d3048.c9a58c9120361b7b034a0136cc74d5dce009e745c4cc111c255d5f3d0a9e2fd9 | |
| /home/sanchitgandhi/transformers/src/transformers/configuration_utils.py:358: UserWarning: Passing `gradient_checkpointing` to a config initialization is deprecated and will be removed in v5 Transformers. Using `model.gradient_checkpointing_enable()` instead, or if you are using the `Trainer` API, pass `gradient_checkpointing=True` in your `TrainingArguments`. | |
| warnings.warn( | |
| Model config SpeechEncoderDecoderConfig { | |
| "_name_or_path": "sanchit-gandhi/flax-wav2vec2-2-bart-large-scan", | |
| "architectures": [ | |
| "SpeechEncoderDecoderModel" | |
| ], | |
| "decoder": { | |
| "_name_or_path": "", | |
| "activation_dropout": 0.1, | |
| "activation_function": "gelu", | |
| "add_bias_logits": false, | |
| "add_cross_attention": true, | |
| "add_final_layer_norm": false, | |
| "architectures": [ | |
| "BartModel" | |
| ], | |
| "attention_dropout": 0.1, | |
| "bad_words_ids": null, | |
| "bos_token_id": 0, | |
| "chunk_size_feed_forward": 0, | |
| "classif_dropout": 0.1, | |
| "classifier_dropout": 0.0, | |
| "cross_attention_hidden_size": null, | |
| "d_model": 1024, | |
| "decoder_attention_heads": 16, | |
| "decoder_ffn_dim": 4096, | |
| "decoder_layerdrop": 0.0, | |
| "decoder_layers": 12, | |
| "decoder_start_token_id": 2, | |
| "diversity_penalty": 0.0, | |
| "do_sample": false, | |
| "dropout": 0.1, | |
| "early_stopping": true, | |
| "encoder_attention_heads": 16, | |
| "encoder_ffn_dim": 4096, | |
| "encoder_layerdrop": 0.0, | |
| "encoder_layers": 12, | |
| "encoder_no_repeat_ngram_size": 0, | |
| "eos_token_id": 2, | |
| "exponential_decay_length_penalty": null, | |
| "finetuning_task": null, | |
| "forced_bos_token_id": 0, | |
| "forced_eos_token_id": 2, | |
| "fuse_matmuls": false, | |
| "gradient_checkpointing": true, | |
| "id2label": { | |
| "0": "LABEL_0", | |
| "1": "LABEL_1", | |
| "2": "LABEL_2" | |
| }, | |
| "init_std": 0.02, | |
| "is_decoder": true, | |
| "is_encoder_decoder": false, | |
| "label2id": { | |
| "LABEL_0": 0, | |
| "LABEL_1": 1, | |
| "LABEL_2": 2 | |
| }, | |
| "length_penalty": 1.0, | |
| "max_length": 20, | |
| "max_position_embeddings": 1024, | |
| "min_length": 0, | |
| "model_type": "bart", | |
| "no_repeat_ngram_size": 3, | |
| "normalize_before": false, | |
| "num_beam_groups": 1, | |
| "num_beams": 4, | |
| "num_hidden_layers": 12, | |
| "num_return_sequences": 1, | |
| "output_attentions": false, | |
| "output_hidden_states": false, | |
| "output_scores": false, | |
| "pad_token_id": 1, | |
| "prefix": null, | |
| "problem_type": null, | |
| "pruned_heads": {}, | |
| "remove_invalid_values": false, | |
| "repetition_penalty": 1.0, | |
| "return_dict": true, | |
| "return_dict_in_generate": false, | |
| "scale_embedding": false, | |
| "sep_token_id": null, | |
| "task_specific_params": { | |
| "summarization": { | |
| "length_penalty": 1.0, | |
| "max_length": 128, | |
| "min_length": 12, | |
| "num_beams": 4 | |
| }, | |
| "summarization_cnn": { | |
| "length_penalty": 2.0, | |
| "max_length": 142, | |
| "min_length": 56, | |
| "num_beams": 4 | |
| }, | |
| "summarization_xsum": { | |
| "length_penalty": 1.0, | |
| "max_length": 62, | |
| "min_length": 11, | |
| "num_beams": 6 | |
| } | |
| }, | |
| "temperature": 1.0, | |
| "tie_encoder_decoder": false, | |
| "tie_word_embeddings": true, | |
| "tokenizer_class": null, | |
| "top_k": 50, | |
| "top_p": 1.0, | |
| "torch_dtype": "float32", | |
| "torchscript": false, | |
| "transformers_version": "4.18.0.dev0", | |
| "typical_p": 1.0, | |
| "use_bfloat16": false, | |
| "use_cache": true, | |
| "use_scan": true, | |
| "vocab_size": 50265 | |
| }, | |
| "decoder_start_token_id": 0, | |
| "encoder": { | |
| "_name_or_path": "", | |
| "activation_dropout": 0.1, | |
| "adapter_kernel_size": 3, | |
| "adapter_stride": 2, | |
| "add_adapter": true, | |
| "add_cross_attention": false, | |
| "apply_spec_augment": true, | |
| "architectures": [ | |
| "Wav2Vec2ForPreTraining" | |
| ], | |
| "attention_dropout": 0.1, | |
| "bad_words_ids": null, | |
| "bos_token_id": 1, | |
| "chunk_size_feed_forward": 0, | |
| "classifier_proj_size": 256, | |
| "codevector_dim": 768, | |
| "contrastive_logits_temperature": 0.1, | |
| "conv_bias": true, | |
| "conv_dim": [ | |
| 512, | |
| 512, | |
| 512, | |
| 512, | |
| 512, | |
| 512, | |
| 512 | |
| ], | |
| "conv_kernel": [ | |
| 10, | |
| 3, | |
| 3, | |
| 3, | |
| 3, | |
| 2, | |
| 2 | |
| ], | |
| "conv_stride": [ | |
| 5, | |
| 2, | |
| 2, | |
| 2, | |
| 2, | |
| 2, | |
| 2 | |
| ], | |
| "cross_attention_hidden_size": null, | |
| "ctc_loss_reduction": "sum", | |
| "ctc_zero_infinity": false, | |
| "decoder_start_token_id": null, | |
| "diversity_loss_weight": 0.1, | |
| "diversity_penalty": 0.0, | |
| "do_sample": false, | |
| "do_stable_layer_norm": true, | |
| "early_stopping": false, | |
| "encoder_no_repeat_ngram_size": 0, | |
| "eos_token_id": 2, | |
| "exponential_decay_length_penalty": null, | |
| "feat_extract_activation": "gelu", | |
| "feat_extract_dropout": 0.0, | |
| "feat_extract_norm": "layer", | |
| "feat_proj_dropout": 0.0, | |
| "feat_quantizer_dropout": 0.0, | |
| "final_dropout": 0.0, | |
| "finetuning_task": null, | |
| "forced_bos_token_id": null, | |
| "forced_eos_token_id": null, | |
| "fuse_matmuls": false, | |
| "gradient_checkpointing": true, | |
| "hidden_act": "gelu", | |
| "hidden_dropout": 0.1, | |
| "hidden_dropout_prob": 0.1, | |
| "hidden_size": 1024, | |
| "id2label": { | |
| "0": "LABEL_0", | |
| "1": "LABEL_1" | |
| }, | |
| "initializer_range": 0.02, | |
| "intermediate_size": 4096, | |
| "is_decoder": false, | |
| "is_encoder_decoder": false, | |
| "label2id": { | |
| "LABEL_0": 0, | |
| "LABEL_1": 1 | |
| }, | |
| "layer_norm_eps": 1e-05, | |
| "layerdrop": 0.0, | |
| "length_penalty": 1.0, | |
| "mask_feature_length": 10, | |
| "mask_feature_min_masks": 0, | |
| "mask_feature_prob": 0.0, | |
| "mask_time_length": 10, | |
| "mask_time_min_masks": 2, | |
| "mask_time_prob": 0.1, | |
| "max_length": 20, | |
| "min_length": 0, | |
| "model_type": "wav2vec2", | |
| "no_repeat_ngram_size": 0, | |
| "num_adapter_layers": 3, | |
| "num_attention_heads": 16, | |
| "num_beam_groups": 1, | |
| "num_beams": 1, | |
| "num_codevector_groups": 2, | |
| "num_codevectors_per_group": 320, | |
| "num_conv_pos_embedding_groups": 16, | |
| "num_conv_pos_embeddings": 128, | |
| "num_feat_extract_layers": 7, | |
| "num_hidden_layers": 24, | |
| "num_negatives": 100, | |
| "num_return_sequences": 1, | |
| "output_attentions": false, | |
| "output_hidden_size": 1024, | |
| "output_hidden_states": false, | |
| "output_scores": false, | |
| "pad_token_id": 0, | |
| "prefix": null, | |
| "problem_type": null, | |
| "proj_codevector_dim": 768, | |
| "pruned_heads": {}, | |
| "remove_invalid_values": false, | |
| "repetition_penalty": 1.0, | |
| "return_dict": true, | |
| "return_dict_in_generate": false, | |
| "sep_token_id": null, | |
| "task_specific_params": null, | |
| "tdnn_dilation": [ | |
| 1, | |
| 2, | |
| 3, | |
| 1, | |
| 1 | |
| ], | |
| "tdnn_dim": [ | |
| 512, | |
| 512, | |
| 512, | |
| 512, | |
| 1500 | |
| ], | |
| "tdnn_kernel": [ | |
| 5, | |
| 3, | |
| 3, | |
| 1, | |
| 1 | |
| ], | |
| "temperature": 1.0, | |
| "tie_encoder_decoder": false, | |
| "tie_word_embeddings": true, | |
| "tokenizer_class": null, | |
| "top_k": 50, | |
| "top_p": 1.0, | |
| "torch_dtype": null, | |
| "torchscript": false, | |
| "transformers_version": "4.18.0.dev0", | |
| "typical_p": 1.0, | |
| "use_bfloat16": false, | |
| "use_scan": true, | |
| "use_weighted_layer_sum": false, | |
| "vocab_size": 32, | |
| "xvector_output_dim": 512 | |
| }, | |
| "eos_token_id": 2, | |
| "is_encoder_decoder": true, | |
| "max_length": 40, | |
| "model_type": "speech-encoder-decoder", | |
| "pad_token_id": 1, | |
| "processor_class": "Wav2Vec2Processor", | |
| "tie_word_embeddings": false, | |
| "transformers_version": null, | |
| "use_cache": false | |
| } | |
| loading feature extractor configuration file https://huggingface.co/sanchit-gandhi/flax-wav2vec2-2-bart-large-scan/resolve/main/preprocessor_config.json from cache at /home/sanchitgandhi/.cache/huggingface/transformers/bc2232c616201c7d3d66ba3f6a7d1186306134838dfb19786149f0e16122787d.bbc1eb890a39c82e710a893223b8452ac5b78e8b57083b2f893aa7dc59d4ed69 | |
| Feature extractor Wav2Vec2FeatureExtractor { | |
| "do_normalize": true, | |
| "feature_extractor_type": "Wav2Vec2FeatureExtractor", | |
| "feature_size": 1, | |
| "padding_side": "right", | |
| "padding_value": 0.0, | |
| "return_attention_mask": true, | |
| "sampling_rate": 16000 | |
| } | |
| loading file https://huggingface.co/sanchit-gandhi/flax-wav2vec2-2-bart-large-scan/resolve/main/vocab.json from cache at /home/sanchitgandhi/.cache/huggingface/transformers/86c0de13925d1534934e540ff4c9dd778f49761b4eaf59dae3335a4f6690a814.bfdcc444ff249bca1a95ca170ec350b442f81804d7df3a95a2252217574121d7 | |
| loading file https://huggingface.co/sanchit-gandhi/flax-wav2vec2-2-bart-large-scan/resolve/main/merges.txt from cache at /home/sanchitgandhi/.cache/huggingface/transformers/7cf4fc91891684e1177d1c519689e4c310ebdec965e00d6e45134bb9227ab01b.f5b91da9e34259b8f4d88dbc97c740667a0e8430b96314460cdb04e86d4fc435 | |
| loading file https://huggingface.co/sanchit-gandhi/flax-wav2vec2-2-bart-large-scan/resolve/main/tokenizer.json from cache at /home/sanchitgandhi/.cache/huggingface/transformers/c02f3f3009bfacaa24cfead1d0f7fbf4fc2fb5f8092f68703449f02aa3a28e03.393fa6a095aa312a3cce4d5263e471bd94ec0215e6c63448a6464d59ff900814 | |
| loading file https://huggingface.co/sanchit-gandhi/flax-wav2vec2-2-bart-large-scan/resolve/main/added_tokens.json from cache at None | |
| loading file https://huggingface.co/sanchit-gandhi/flax-wav2vec2-2-bart-large-scan/resolve/main/special_tokens_map.json from cache at /home/sanchitgandhi/.cache/huggingface/transformers/505d61b8f6e05764b5aec1483bfdd13a310681a5af54957263604323be3bbabf.a11ebb04664c067c8fe5ef8f8068b0f721263414a26058692f7b2e4ba2a1b342 | |
| loading file https://huggingface.co/sanchit-gandhi/flax-wav2vec2-2-bart-large-scan/resolve/main/tokenizer_config.json from cache at /home/sanchitgandhi/.cache/huggingface/transformers/ff79c23164eac352d7f9651f3c3774a962ce80f81460d9e17d689235fa34ee80.0e8b2b497f91e23302894a5c1f19ced6334b0abd450a7bce75a67bf0f9ee5c54 | |
| loading weights file https://huggingface.co/sanchit-gandhi/flax-wav2vec2-2-bart-large-scan/resolve/main/flax_model.msgpack from cache at /home/sanchitgandhi/.cache/huggingface/transformers/1279dc21f7dd9ed546f166e7e445e068b2672ddfa5386b2e3a3a973b8d668365.8e03496bb6919447aeb468483249e7b65dfb59c42989be9787af0aa6aa9b3f50 | |
| tcmalloc: large alloc 2353618944 bytes == 0xa2612000 @ 0x7fb1de660680 0x7fb1de681824 0x5f8a01 0x648cf1 0x5c4676 0x4f290e 0x64f718 0x5048b3 0x56b1da 0x56939a 0x50aaa0 0x56c28c 0x56939a 0x5f6a13 0x56b0ae 0x56939a 0x68d047 0x67e351 0x67e3cf 0x67e471 0x67e817 0x6b6fe2 0x6b736d 0x7fb1de4730b3 0x5fa5ce | |
| /home/sanchitgandhi/hf/lib/python3.8/site-packages/jax/_src/tree_util.py:188: FutureWarning: jax.tree_util.tree_multimap() is deprecated. Please use jax.tree_util.tree_map() instead as a drop-in replacement. | |
| warnings.warn('jax.tree_util.tree_multimap() is deprecated. Please use jax.tree_util.tree_map() ' | |
| All model checkpoint weights were used when initializing FlaxSpeechEncoderDecoderModel. | |
| All the weights of FlaxSpeechEncoderDecoderModel were initialized from the model checkpoint at sanchit-gandhi/flax-wav2vec2-2-bart-large-scan. | |
| If your task is similar to the task the model of the checkpoint was trained on, you can already use FlaxSpeechEncoderDecoderModel for predictions without further training. | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-5bcd522a0f1116c8.arrow | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-148286b6e8c7c55a.arrow | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-6afe8daa3edc3979.arrow | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-fb71416814467da1.arrow | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-c4d90e72e7945373.arrow | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-1441a1a362f45f74.arrow | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-86716240f0d5e9ce.arrow | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-567649d38b4eb4b9.arrow | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-a0253d3135dc5a4b.arrow | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-90f3995b6106b9f2.arrow | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-c1105482a786c706.arrow | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-2c5d0ae89484568c.arrow | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-e43260f16a4667dc.arrow | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-d1485c81a0cbb06c.arrow | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-a791cf68d9739c69.arrow | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-94bc2d5136fdbae7.arrow | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-61413d2b04061247.arrow | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-5b531145a5727e3e.arrow | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-eef5cb2925240576.arrow | |
| 05/30/2022 18:00:06 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/sanchitgandhi/cache/huggingface/datasets/librispeech_asr/all/2.1.0/14c8bffddb861b4b3a4fcdff648a56980dbb808f3fc56f5a3d56b18ee88458eb/cache-dc7710d750f9cc97.arrow | |
| Feature extractor saved in ./preprocessor_config.json | |
| tokenizer config file saved in ./tokenizer_config.json | |
| Special tokens file saved in ./special_tokens_map.json | |
| Configuration saved in ./config.json | |
| loading feature extractor configuration file ./preprocessor_config.json | |
| loading configuration file ./config.json | |
| Model config SpeechEncoderDecoderConfig { | |
| "_name_or_path": "./", | |
| "architectures": [ | |
| "SpeechEncoderDecoderModel" | |
| ], | |
| "decoder": { | |
| "_name_or_path": "", | |
| "activation_dropout": 0.1, | |
| "activation_function": "gelu", | |
| "add_bias_logits": false, | |
| "add_cross_attention": true, | |
| "add_final_layer_norm": false, | |
| "architectures": [ | |
| "BartModel" | |
| ], | |
| "attention_dropout": 0.1, | |
| "bad_words_ids": null, | |
| "bos_token_id": 0, | |
| "chunk_size_feed_forward": 0, | |
| "classif_dropout": 0.1, | |
| "classifier_dropout": 0.0, | |
| "cross_attention_hidden_size": null, | |
| "d_model": 1024, | |
| "decoder_attention_heads": 16, | |
| "decoder_ffn_dim": 4096, | |
| "decoder_layerdrop": 0.0, | |
| "decoder_layers": 12, | |
| "decoder_start_token_id": 2, | |
| "diversity_penalty": 0.0, | |
| "do_sample": false, | |
| "dropout": 0.1, | |
| "early_stopping": true, | |
| "encoder_attention_heads": 16, | |
| "encoder_ffn_dim": 4096, | |
| "encoder_layerdrop": 0.0, | |
| "encoder_layers": 12, | |
| "encoder_no_repeat_ngram_size": 0, | |
| "eos_token_id": 2, | |
| "exponential_decay_length_penalty": null, | |
| "finetuning_task": null, | |
| "forced_bos_token_id": 0, | |
| "forced_eos_token_id": 2, | |
| "fuse_matmuls": false, | |
| "gradient_checkpointing": true, | |
| "id2label": { | |
| "0": "LABEL_0", | |
| "1": "LABEL_1", | |
| "2": "LABEL_2" | |
| }, | |
| "init_std": 0.02, | |
| "is_decoder": true, | |
| "is_encoder_decoder": false, | |
| "label2id": { | |
| "LABEL_0": 0, | |
| "LABEL_1": 1, | |
| "LABEL_2": 2 | |
| }, | |
| "length_penalty": 1.0, | |
| "max_length": 20, | |
| "max_position_embeddings": 1024, | |
| "min_length": 0, | |
| "model_type": "bart", | |
| "no_repeat_ngram_size": 3, | |
| "normalize_before": false, | |
| "num_beam_groups": 1, | |
| "num_beams": 4, | |
| "num_hidden_layers": 12, | |
| "num_return_sequences": 1, | |
| "output_attentions": false, | |
| "output_hidden_states": false, | |
| "output_scores": false, | |
| "pad_token_id": 1, | |
| "prefix": null, | |
| "problem_type": null, | |
| "pruned_heads": {}, | |
| "remove_invalid_values": false, | |
| "repetition_penalty": 1.0, | |
| "return_dict": true, | |
| "return_dict_in_generate": false, | |
| "scale_embedding": false, | |
| "sep_token_id": null, | |
| "task_specific_params": { | |
| "summarization": { | |
| "length_penalty": 1.0, | |
| "max_length": 128, | |
| "min_length": 12, | |
| "num_beams": 4 | |
| }, | |
| "summarization_cnn": { | |
| "length_penalty": 2.0, | |
| "max_length": 142, | |
| "min_length": 56, | |
| "num_beams": 4 | |
| }, | |
| "summarization_xsum": { | |
| "length_penalty": 1.0, | |
| "max_length": 62, | |
| "min_length": 11, | |
| "num_beams": 6 | |
| } | |
| }, | |
| "temperature": 1.0, | |
| "tie_encoder_decoder": false, | |
| "tie_word_embeddings": true, | |
| "tokenizer_class": null, | |
| "top_k": 50, | |
| "top_p": 1.0, | |
| "torch_dtype": "float32", | |
| "torchscript": false, | |
| "transformers_version": "4.18.0.dev0", | |
| "typical_p": 1.0, | |
| "use_bfloat16": false, | |
| "use_cache": true, | |
| "use_scan": true, | |
| "vocab_size": 50265 | |
| }, | |
| "decoder_start_token_id": 0, | |
| "encoder": { | |
| "_name_or_path": "", | |
| "activation_dropout": 0.1, | |
| "adapter_kernel_size": 3, | |
| "adapter_stride": 2, | |
| "add_adapter": true, | |
| "add_cross_attention": false, | |
| "apply_spec_augment": true, | |
| "architectures": [ | |
| "Wav2Vec2ForPreTraining" | |
| ], | |
| "attention_dropout": 0.1, | |
| "bad_words_ids": null, | |
| "bos_token_id": 1, | |
| "chunk_size_feed_forward": 0, | |
| "classifier_proj_size": 256, | |
| "codevector_dim": 768, | |
| "contrastive_logits_temperature": 0.1, | |
| "conv_bias": true, | |
| "conv_dim": [ | |
| 512, | |
| 512, | |
| 512, | |
| 512, | |
| 512, | |
| 512, | |
| 512 | |
| ], | |
| "conv_kernel": [ | |
| 10, | |
| 3, | |
| 3, | |
| 3, | |
| 3, | |
| 2, | |
| 2 | |
| ], | |
| "conv_stride": [ | |
| 5, | |
| 2, | |
| 2, | |
| 2, | |
| 2, | |
| 2, | |
| 2 | |
| ], | |
| "cross_attention_hidden_size": null, | |
| "ctc_loss_reduction": "sum", | |
| "ctc_zero_infinity": false, | |
| "decoder_start_token_id": null, | |
| "diversity_loss_weight": 0.1, | |
| "diversity_penalty": 0.0, | |
| "do_sample": false, | |
| "do_stable_layer_norm": true, | |
| "early_stopping": false, | |
| "encoder_no_repeat_ngram_size": 0, | |
| "eos_token_id": 2, | |
| "exponential_decay_length_penalty": null, | |
| "feat_extract_activation": "gelu", | |
| "feat_extract_dropout": 0.0, | |
| "feat_extract_norm": "layer", | |
| "feat_proj_dropout": 0.0, | |
| "feat_quantizer_dropout": 0.0, | |
| "final_dropout": 0.0, | |
| "finetuning_task": null, | |
| "forced_bos_token_id": null, | |
| "forced_eos_token_id": null, | |
| "fuse_matmuls": false, | |
| "gradient_checkpointing": true, | |
| "hidden_act": "gelu", | |
| "hidden_dropout": 0.1, | |
| "hidden_dropout_prob": 0.1, | |
| "hidden_size": 1024, | |
| "id2label": { | |
| "0": "LABEL_0", | |
| "1": "LABEL_1" | |
| }, | |
| "initializer_range": 0.02, | |
| "intermediate_size": 4096, | |
| "is_decoder": false, | |
| "is_encoder_decoder": false, | |
| "label2id": { | |
| "LABEL_0": 0, | |
| "LABEL_1": 1 | |
| }, | |
| "layer_norm_eps": 1e-05, | |
| "layerdrop": 0.0, | |
| "length_penalty": 1.0, | |
| "mask_feature_length": 10, | |
| "mask_feature_min_masks": 0, | |
| "mask_feature_prob": 0.0, | |
| "mask_time_length": 10, | |
| "mask_time_min_masks": 2, | |
| "mask_time_prob": 0.1, | |
| "max_length": 20, | |
| "min_length": 0, | |
| "model_type": "wav2vec2", | |
| "no_repeat_ngram_size": 0, | |
| "num_adapter_layers": 3, | |
| "num_attention_heads": 16, | |
| "num_beam_groups": 1, | |
| "num_beams": 1, | |
| "num_codevector_groups": 2, | |
| "num_codevectors_per_group": 320, | |
| "num_conv_pos_embedding_groups": 16, | |
| "num_conv_pos_embeddings": 128, | |
| "num_feat_extract_layers": 7, | |
| "num_hidden_layers": 24, | |
| "num_negatives": 100, | |
| "num_return_sequences": 1, | |
| "output_attentions": false, | |
| "output_hidden_size": 1024, | |
| "output_hidden_states": false, | |
| "output_scores": false, | |
| "pad_token_id": 0, | |
| "prefix": null, | |
| "problem_type": null, | |
| "proj_codevector_dim": 768, | |
| "pruned_heads": {}, | |
| "remove_invalid_values": false, | |
| "repetition_penalty": 1.0, | |
| "return_dict": true, | |
| "return_dict_in_generate": false, | |
| "sep_token_id": null, | |
| "task_specific_params": null, | |
| "tdnn_dilation": [ | |
| 1, | |
| 2, | |
| 3, | |
| 1, | |
| 1 | |
| ], | |
| "tdnn_dim": [ | |
| 512, | |
| 512, | |
| 512, | |
| 512, | |
| 1500 | |
| ], | |
| "tdnn_kernel": [ | |
| 5, | |
| 3, | |
| 3, | |
| 1, | |
| 1 | |
| ], | |
| "temperature": 1.0, | |
| "tie_encoder_decoder": false, | |
| "tie_word_embeddings": true, | |
| "tokenizer_class": null, | |
| "top_k": 50, | |
| "top_p": 1.0, | |
| "torch_dtype": null, | |
| "torchscript": false, | |
| "transformers_version": "4.18.0.dev0", | |
| "typical_p": 1.0, | |
| "use_bfloat16": false, | |
| "use_scan": true, | |
| "use_weighted_layer_sum": false, | |
| "vocab_size": 32, | |
| "xvector_output_dim": 512 | |
| }, | |
| "eos_token_id": 2, | |
| "is_encoder_decoder": true, | |
| "max_length": 40, | |
| "model_type": "speech-encoder-decoder", | |
| "pad_token_id": 1, | |
| "processor_class": "Wav2Vec2Processor", | |
| "tie_word_embeddings": false, | |
| "transformers_version": null, | |
| "use_cache": false | |
| } | |
| loading feature extractor configuration file ./preprocessor_config.json | |
| Feature extractor Wav2Vec2FeatureExtractor { | |
| "do_normalize": true, | |
| "feature_extractor_type": "Wav2Vec2FeatureExtractor", | |
| "feature_size": 1, | |
| "padding_side": "right", | |
| "padding_value": 0.0, | |
| "return_attention_mask": true, | |
| "sampling_rate": 16000 | |
| } | |
| Didn't find file ./added_tokens.json. We won't load it. | |
| loading file ./vocab.json | |
| loading file ./merges.txt | |
| loading file ./tokenizer.json | |
| loading file None | |
| loading file ./special_tokens_map.json | |
| loading file ./tokenizer_config.json | |
| 05/30/2022 18:00:07 - WARNING - __main__ - Unable to display metrics through TensorBoard because the package is not installed: Please run `pip install tensorboard` to enable. | |
| /home/sanchitgandhi/flax-wav2vec2-2-bart-large-ls-960h-feature-encoder/./ is already a clone of https://huggingface.co/sanchit-gandhi/flax-wav2vec2-2-bart-large-ls-960h-feature-encoder. Make sure you pull the latest changes with `repo.git_pull()`. | |
| 05/30/2022 18:00:11 - WARNING - huggingface_hub.repository - /home/sanchitgandhi/flax-wav2vec2-2-bart-large-ls-960h-feature-encoder/./ is already a clone of https://huggingface.co/sanchit-gandhi/flax-wav2vec2-2-bart-large-ls-960h-feature-encoder. Make sure you pull the latest changes with `repo.git_pull()`. | |
| 05/30/2022 18:00:14 - INFO - __main__ - ***** Running training ***** | |
| 05/30/2022 18:00:14 - INFO - __main__ - Num examples = 281186 | |
| 05/30/2022 18:00:14 - INFO - __main__ - Num Epochs = 12 | |
| 05/30/2022 18:00:14 - INFO - __main__ - Instantaneous batch size per device = 4 | |
| 05/30/2022 18:00:14 - INFO - __main__ - Num gradient accumulation steps = 2 | |
| 05/30/2022 18:00:14 - INFO - __main__ - Total train batch size (w. parallel & distributed) = 64 | |
| 05/30/2022 18:00:14 - INFO - __main__ - Total optimization steps = 50000 | |
| 05/30/2022 18:00:14 - INFO - __main__ - Gradient checkpointing: True | |
| 05/30/2022 18:00:14 - INFO - __main__ - Use scan: True | |
| 05/30/2022 18:00:14 - INFO - __main__ - Fuse matmuls: False | |
| Epoch ... (1/12): 0% 0/12 [00:00<?, ?it/s] | |
| Training...: 0% 0/4393 [00:00<?, ?it/s][A Training...: 0% 0/4393 [00:07<?, ?it/s] | |
| Epoch ... (1/12): 0% 0/12 [00:15<?, ?it/s] | |
| Traceback (most recent call last): | |
| File "run_flax_speech_recognition_seq2seq.py", line 1507, in <module> | |
| main() | |
| File "run_flax_speech_recognition_seq2seq.py", line 1400, in main | |
| state, train_metric = p_train_step(state, batch) | |
| File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/jax/_src/traceback_util.py", line 162, in reraise_with_filtered_traceback | |
| return fun(*args, **kwargs) | |
| File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/jax/_src/api.py", line 2013, in cache_miss | |
| out_tree, out_flat = f_pmapped_(*args, **kwargs) | |
| File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/jax/_src/api.py", line 1889, in pmap_f | |
| out = pxla.xla_pmap( | |
| File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/jax/core.py", line 1863, in bind | |
| return map_bind(self, fun, *args, **params) | |
| File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/jax/core.py", line 1895, in map_bind | |
| outs = primitive.process(top_trace, fun, tracers, params) | |
| File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/jax/core.py", line 1866, in process | |
| return trace.process_map(self, fun, tracers, params) | |
| File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/jax/core.py", line 660, in process_call | |
| return primitive.impl(f, *tracers, **params) | |
| File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/jax/interpreters/pxla.py", line 801, in xla_pmap_impl | |
| compiled_fun, fingerprint = parallel_callable( | |
| File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/jax/linear_util.py", line 285, in memoized_fun | |
| ans = call(fun, *args) | |
| File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/jax/interpreters/pxla.py", line 829, in parallel_callable | |
| pmap_computation = lower_parallel_callable( | |
| File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/jax/_src/profiler.py", line 206, in wrapper | |
| return func(*args, **kwargs) | |
| File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/jax/interpreters/pxla.py", line 1002, in lower_parallel_callable | |
| jaxpr, consts, replicas, parts, shards = stage_parallel_callable( | |
| File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/jax/interpreters/pxla.py", line 909, in stage_parallel_callable | |
| jaxpr, out_sharded_avals, consts = pe.trace_to_jaxpr_final( | |
| File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/jax/_src/profiler.py", line 206, in wrapper | |
| return func(*args, **kwargs) | |
| File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/jax/interpreters/partial_eval.py", line 1844, in trace_to_jaxpr_final | |
| jaxpr, out_avals, consts = trace_to_subjaxpr_dynamic( | |
| File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/jax/interpreters/partial_eval.py", line 1816, in trace_to_subjaxpr_dynamic | |
| ans = fun.call_wrapped(*in_tracers_) | |
| File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/jax/linear_util.py", line 168, in call_wrapped | |
| ans = self.f(*args, **dict(self.params, **kwargs)) | |
| File "run_flax_speech_recognition_seq2seq.py", line 1174, in train_step | |
| if gradient_accumulation_steps == 1 or training_args.multisteps: | |
| jax._src.traceback_util.UnfilteredStackTrace: AttributeError: 'FlaxSeq2SeqTrainingArguments' object has no attribute 'multisteps' | |
| The stack trace below excludes JAX-internal frames. | |
| The preceding is the original exception that occurred, unmodified. | |
| -------------------- | |
| The above exception was the direct cause of the following exception: | |
| Traceback (most recent call last): | |
| File "run_flax_speech_recognition_seq2seq.py", line 1507, in <module> | |
| main() | |
| File "run_flax_speech_recognition_seq2seq.py", line 1400, in main | |
| state, train_metric = p_train_step(state, batch) | |
| File "run_flax_speech_recognition_seq2seq.py", line 1174, in train_step | |
| if gradient_accumulation_steps == 1 or training_args.multisteps: | |
| AttributeError: 'FlaxSeq2SeqTrainingArguments' object has no attribute 'multisteps' | |
| wandb: Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing. | |
| wandb: - 0.002 MB of 0.002 MB uploaded (0.000 MB deduped) wandb: \ 0.002 MB of 0.002 MB uploaded (0.000 MB deduped) wandb: | 0.002 MB of 0.034 MB uploaded (0.000 MB deduped) wandb: / 0.002 MB of 0.034 MB uploaded (0.000 MB deduped) wandb: - 0.034 MB of 0.034 MB uploaded (0.000 MB deduped) wandb: \ 0.034 MB of 0.034 MB uploaded (0.000 MB deduped) wandb: | 0.034 MB of 0.034 MB uploaded (0.000 MB deduped) wandb: / 0.034 MB of 0.034 MB uploaded (0.000 MB deduped) wandb: - 0.034 MB of 0.034 MB uploaded (0.000 MB deduped) wandb: \ 0.034 MB of 0.034 MB uploaded (0.000 MB deduped) wandb: | 0.034 MB of 0.034 MB uploaded (0.000 MB deduped) wandb: / 0.034 MB of 0.034 MB uploaded (0.000 MB deduped) wandb: | |
| wandb: Synced flax-wav2vec2-2-bart-large-ls-960h-feature-encoder: https://wandb.ai/sanchit-gandhi/librispeech_960h/runs/2nzi0chc | |
| wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) | |
| wandb: Find logs at: ./wandb/run-20220530_175928-2nzi0chc/logs | |