| # python3 -m espnet2.bin.asr_train --use_preprocessor true --bpemodel data/cy_token_list/bpe_unigram150/bpe.model --token_type bpe --token_list data/cy_token_list/bpe_unigram150/tokens.txt --non_linguistic_symbols none --cleaner none --g2p none --valid_data_path_and_name_and_type dump/raw/dev_cy/wav.scp,speech,sound --valid_data_path_and_name_and_type dump/raw/dev_cy/text,text,text --valid_shape_file exp/asr_stats_raw_cy_bpe150_sp/valid/speech_shape --valid_shape_file exp/asr_stats_raw_cy_bpe150_sp/valid/text_shape.bpe --resume true --init_param --ignore_init_mismatch false --fold_length 80000 --fold_length 150 --output_dir exp/asr_oxford_frontend_raw_cy_bpe150_sp --config conf/tuning/oxford_frontend.yaml --frontend_conf fs=16k --train_data_path_and_name_and_type dump/raw/train_cy_sp/wav.scp,speech,sound --train_data_path_and_name_and_type dump/raw/train_cy_sp/text,text,text --train_shape_file exp/asr_stats_raw_cy_bpe150_sp/train/speech_shape --train_shape_file exp/asr_stats_raw_cy_bpe150_sp/train/text_shape.bpe --ngpu 3 --multiprocessing_distributed True | |
| # Started at Mon Jun 6 21:31:07 EDT 2022 | |
| # | |
| /usr/bin/python3 /project/ocean/junweih/espnet/espnet2/bin/asr_train.py --use_preprocessor true --bpemodel data/cy_token_list/bpe_unigram150/bpe.model --token_type bpe --token_list data/cy_token_list/bpe_unigram150/tokens.txt --non_linguistic_symbols none --cleaner none --g2p none --valid_data_path_and_name_and_type dump/raw/dev_cy/wav.scp,speech,sound --valid_data_path_and_name_and_type dump/raw/dev_cy/text,text,text --valid_shape_file exp/asr_stats_raw_cy_bpe150_sp/valid/speech_shape --valid_shape_file exp/asr_stats_raw_cy_bpe150_sp/valid/text_shape.bpe --resume true --init_param --ignore_init_mismatch false --fold_length 80000 --fold_length 150 --output_dir exp/asr_oxford_frontend_raw_cy_bpe150_sp --config conf/tuning/oxford_frontend.yaml --frontend_conf fs=16k --train_data_path_and_name_and_type dump/raw/train_cy_sp/wav.scp,speech,sound --train_data_path_and_name_and_type dump/raw/train_cy_sp/text,text,text --train_shape_file exp/asr_stats_raw_cy_bpe150_sp/train/speech_shape --train_shape_file exp/asr_stats_raw_cy_bpe150_sp/train/text_shape.bpe --ngpu 3 --multiprocessing_distributed True | |
| [islpc50:0/3] 2022-06-06 21:31:22,179 (distributed_c10d:217) INFO: Added key: store_based_barrier_key:1 to store for rank: 0 | |
| [islpc50:0/3] 2022-06-06 21:31:22,180 (distributed_c10d:251) INFO: Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes. | |
| [islpc50:0/3] 2022-06-06 21:31:22,249 (asr:411) INFO: Vocabulary size: 150 | |
| [islpc50:0/3] 2022-06-06 21:31:22,968 (filelock:274) INFO: Lock 140511308283184 acquired on ./hub/s3prl_cache/1c76d6e88090f01736036b28dc995fef583f47f42662d55286332557f957609f.lock | |
| [islpc50:0/3] 2022-06-06 21:31:22,969 (filelock:318) INFO: Lock 140511308283184 released on ./hub/s3prl_cache/1c76d6e88090f01736036b28dc995fef583f47f42662d55286332557f957609f.lock | |
| [Featurizer] - The selected feature last_hidden_state's downsample rate is 320 | |
| [islpc50:0/3] 2022-06-06 21:31:34,900 (s3prl:159) INFO: Pretrained S3PRL frontend model parameters reloaded! | |
| [islpc50:0/3] 2022-06-06 21:31:38,773 (abs_task:1157) INFO: pytorch.version=1.10.1+cu111, cuda.available=True, cudnn.version=8005, cudnn.benchmark=False, cudnn.deterministic=True | |
| [islpc50:0/3] 2022-06-06 21:31:38,779 (abs_task:1158) INFO: Model structure: | |
| ESPnetASRModel( | |
| (frontend): S3prlFrontend( | |
| (upstream): UpstreamExpert( | |
| (model): Wav2Vec2Model( | |
| (feature_extractor): ConvFeatureExtractionModel( | |
| (conv_layers): ModuleList( | |
| (0): Sequential( | |
| (0): Conv1d(1, 512, kernel_size=(10,), stride=(5,)) | |
| (1): Dropout(p=0.0, inplace=False) | |
| (2): Sequential( | |
| (0): TransposeLast() | |
| (1): Fp32LayerNorm((512,), eps=1e-05, elementwise_affine=True) | |
| (2): TransposeLast() | |
| ) | |
| (3): GELU() | |
| ) | |
| (1): Sequential( | |
| (0): Conv1d(512, 512, kernel_size=(3,), stride=(2,)) | |
| (1): Dropout(p=0.0, inplace=False) | |
| (2): Sequential( | |
| (0): TransposeLast() | |
| (1): Fp32LayerNorm((512,), eps=1e-05, elementwise_affine=True) | |
| (2): TransposeLast() | |
| ) | |
| (3): GELU() | |
| ) | |
| (2): Sequential( | |
| (0): Conv1d(512, 512, kernel_size=(3,), stride=(2,)) | |
| (1): Dropout(p=0.0, inplace=False) | |
| (2): Sequential( | |
| (0): TransposeLast() | |
| (1): Fp32LayerNorm((512,), eps=1e-05, elementwise_affine=True) | |
| (2): TransposeLast() | |
| ) | |
| (3): GELU() | |
| ) | |
| (3): Sequential( | |
| (0): Conv1d(512, 512, kernel_size=(3,), stride=(2,)) | |
| (1): Dropout(p=0.0, inplace=False) | |
| (2): Sequential( | |
| (0): TransposeLast() | |
| (1): Fp32LayerNorm((512,), eps=1e-05, elementwise_affine=True) | |
| (2): TransposeLast() | |
| ) | |
| (3): GELU() | |
| ) | |
| (4): Sequential( | |
| (0): Conv1d(512, 512, kernel_size=(3,), stride=(2,)) | |
| (1): Dropout(p=0.0, inplace=False) | |
| (2): Sequential( | |
| (0): TransposeLast() | |
| (1): Fp32LayerNorm((512,), eps=1e-05, elementwise_affine=True) | |
| (2): TransposeLast() | |
| ) | |
| (3): GELU() | |
| ) | |
| (5): Sequential( | |
| (0): Conv1d(512, 512, kernel_size=(2,), stride=(2,)) | |
| (1): Dropout(p=0.0, inplace=False) | |
| (2): Sequential( | |
| (0): TransposeLast() | |
| (1): Fp32LayerNorm((512,), eps=1e-05, elementwise_affine=True) | |
| (2): TransposeLast() | |
| ) | |
| (3): GELU() | |
| ) | |
| (6): Sequential( | |
| (0): Conv1d(512, 512, kernel_size=(2,), stride=(2,)) | |
| (1): Dropout(p=0.0, inplace=False) | |
| (2): Sequential( | |
| (0): TransposeLast() | |
| (1): Fp32LayerNorm((512,), eps=1e-05, elementwise_affine=True) | |
| (2): TransposeLast() | |
| ) | |
| (3): GELU() | |
| ) | |
| ) | |
| ) | |
| (post_extract_proj): Linear(in_features=512, out_features=1024, bias=True) | |
| (dropout_input): Dropout(p=0.1, inplace=False) | |
| (dropout_features): Dropout(p=0.1, inplace=False) | |
| (quantizer): GumbelVectorQuantizer( | |
| (weight_proj): Linear(in_features=512, out_features=640, bias=True) | |
| ) | |
| (project_q): Linear(in_features=768, out_features=768, bias=True) | |
| (encoder): TransformerEncoder( | |
| (pos_conv): Sequential( | |
| (0): Conv1d(1024, 1024, kernel_size=(128,), stride=(1,), padding=(64,), groups=16) | |
| (1): SamePad() | |
| (2): GELU() | |
| ) | |
| (layers): ModuleList( | |
| (0): AdapterTransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (adapter1): Adapter( | |
| (down_projection): Linear(in_features=1024, out_features=192, bias=True) | |
| (up_projection): Linear(in_features=192, out_features=1024, bias=True) | |
| ) | |
| (adapter2): Adapter( | |
| (down_projection): Linear(in_features=1024, out_features=192, bias=True) | |
| (up_projection): Linear(in_features=192, out_features=1024, bias=True) | |
| ) | |
| ) | |
| (1): AdapterTransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (adapter1): Adapter( | |
| (down_projection): Linear(in_features=1024, out_features=192, bias=True) | |
| (up_projection): Linear(in_features=192, out_features=1024, bias=True) | |
| ) | |
| (adapter2): Adapter( | |
| (down_projection): Linear(in_features=1024, out_features=192, bias=True) | |
| (up_projection): Linear(in_features=192, out_features=1024, bias=True) | |
| ) | |
| ) | |
| (2): AdapterTransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (adapter1): Adapter( | |
| (down_projection): Linear(in_features=1024, out_features=192, bias=True) | |
| (up_projection): Linear(in_features=192, out_features=1024, bias=True) | |
| ) | |
| (adapter2): Adapter( | |
| (down_projection): Linear(in_features=1024, out_features=192, bias=True) | |
| (up_projection): Linear(in_features=192, out_features=1024, bias=True) | |
| ) | |
| ) | |
| (3): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (4): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (5): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (6): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (7): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (8): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (9): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (10): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (11): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (12): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (13): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (14): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (15): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (16): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (17): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (18): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (19): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (20): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (21): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (22): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (23): TransformerSentenceEncoderLayer( | |
| (self_attn): MultiheadAttention( | |
| (dropout_module): FairseqDropout() | |
| (k_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (v_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (q_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| (out_proj): Linear(in_features=1024, out_features=1024, bias=True) | |
| ) | |
| (dropout1): Dropout(p=0.0, inplace=False) | |
| (dropout2): Dropout(p=0.0, inplace=False) | |
| (dropout3): Dropout(p=0.0, inplace=False) | |
| (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| (fc1): Linear(in_features=1024, out_features=4096, bias=True) | |
| (fc2): Linear(in_features=4096, out_features=1024, bias=True) | |
| (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| ) | |
| (layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) | |
| ) | |
| (layer_norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) | |
| (final_proj): Linear(in_features=1024, out_features=768, bias=True) | |
| ) | |
| ) | |
| (featurizer): Featurizer() | |
| ) | |
| (normalize): UtteranceMVN(norm_means=True, norm_vars=False) | |
| (encoder): RNNEncoder( | |
| (enc): ModuleList( | |
| (0): RNNP( | |
| (birnn0): LSTM(1024, 320, batch_first=True, bidirectional=True) | |
| (bt0): Linear(in_features=640, out_features=320, bias=True) | |
| (birnn1): LSTM(320, 320, batch_first=True, bidirectional=True) | |
| (bt1): Linear(in_features=640, out_features=320, bias=True) | |
| (birnn2): LSTM(320, 320, batch_first=True, bidirectional=True) | |
| (bt2): Linear(in_features=640, out_features=320, bias=True) | |
| (birnn3): LSTM(320, 320, batch_first=True, bidirectional=True) | |
| (bt3): Linear(in_features=640, out_features=320, bias=True) | |
| ) | |
| ) | |
| ) | |
| (criterion_att): LabelSmoothingLoss( | |
| (criterion): KLDivLoss() | |
| ) | |
| (ctc): CTC( | |
| (ctc_lo): Linear(in_features=320, out_features=150, bias=True) | |
| (ctc_loss): CTCLoss() | |
| ) | |
| ) | |
| Model summary: | |
| Class Name: ESPnetASRModel | |
| Total Number of model parameters: 329.00 M | |
| Number of trainable parameters: 11.61 M (3.5%) | |
| Size: 46.45 MB | |
| Type: torch.float32 | |
| [islpc50:0/3] 2022-06-06 21:31:38,779 (abs_task:1161) INFO: Optimizer: | |
| Adam ( | |
| Parameter Group 0 | |
| amsgrad: False | |
| betas: (0.9, 0.999) | |
| eps: 1e-08 | |
| initial_lr: 0.00027 | |
| lr: 6.749999999999999e-09 | |
| weight_decay: 0 | |
| ) | |
| [islpc50:0/3] 2022-06-06 21:31:38,779 (abs_task:1162) INFO: Scheduler: WarmupLR(warmup_steps=40000) | |
| [islpc50:0/3] 2022-06-06 21:31:38,792 (abs_task:1171) INFO: Saving the configuration in exp/asr_oxford_frontend_raw_cy_bpe150_sp/config.yaml | |
| [islpc50:0/3] 2022-06-06 21:31:41,115 (abs_task:1525) INFO: [train] dataset: | |
| ESPnetDataset( | |
| speech: {"path": "dump/raw/train_cy_sp/wav.scp", "type": "sound"} | |
| text: {"path": "dump/raw/train_cy_sp/text", "type": "text"} | |
| preprocess: <espnet2.train.preprocessor.CommonPreprocessor object at 0x7fca96680220>) | |
| [islpc50:0/3] 2022-06-06 21:31:41,115 (abs_task:1526) INFO: [train] Batch sampler: FoldedBatchSampler(N-batch=24097, batch_size=10, shape_files=['exp/asr_stats_raw_cy_bpe150_sp/train/speech_shape', 'exp/asr_stats_raw_cy_bpe150_sp/train/text_shape.bpe'], sort_in_batch=descending, sort_batch=descending) | |
| [islpc50:0/3] 2022-06-06 21:31:41,118 (abs_task:1527) INFO: [train] mini-batch sizes summary: N-batch=24097, mean=7.3, min=3, max=10 | |
| [islpc50:0/3] 2022-06-06 21:31:41,217 (abs_task:1525) INFO: [valid] dataset: | |
| ESPnetDataset( | |
| speech: {"path": "dump/raw/dev_cy/wav.scp", "type": "sound"} | |
| text: {"path": "dump/raw/dev_cy/text", "type": "text"} | |
| preprocess: <espnet2.train.preprocessor.CommonPreprocessor object at 0x7fca6854fb50>) | |
| [islpc50:0/3] 2022-06-06 21:31:41,217 (abs_task:1526) INFO: [valid] Batch sampler: FoldedBatchSampler(N-batch=464, batch_size=10, shape_files=['exp/asr_stats_raw_cy_bpe150_sp/valid/speech_shape', 'exp/asr_stats_raw_cy_bpe150_sp/valid/text_shape.bpe'], sort_in_batch=descending, sort_batch=descending) | |
| [islpc50:0/3] 2022-06-06 21:31:41,217 (abs_task:1527) INFO: [valid] mini-batch sizes summary: N-batch=464, mean=6.3, min=3, max=10 | |
| [islpc50:0/3] 2022-06-06 21:31:41,255 (abs_task:1525) INFO: [plot_att] dataset: | |
| ESPnetDataset( | |
| speech: {"path": "dump/raw/dev_cy/wav.scp", "type": "sound"} | |
| text: {"path": "dump/raw/dev_cy/text", "type": "text"} | |
| preprocess: <espnet2.train.preprocessor.CommonPreprocessor object at 0x7fca96ccfd60>) | |
| [islpc50:0/3] 2022-06-06 21:31:41,255 (abs_task:1526) INFO: [plot_att] Batch sampler: UnsortedBatchSampler(N-batch=2933, batch_size=1, key_file=exp/asr_stats_raw_cy_bpe150_sp/valid/speech_shape, | |
| [islpc50:0/3] 2022-06-06 21:31:41,255 (abs_task:1527) INFO: [plot_att] mini-batch sizes summary: N-batch=3, mean=1.0, min=1, max=1 | |
| [islpc50:0/3] 2022-06-06 21:31:59,752 (trainer:155) INFO: The training was resumed using exp/asr_oxford_frontend_raw_cy_bpe150_sp/checkpoint.pth | |
| islpc50:1048747:1048747 [0] NCCL INFO Bootstrap : Using bond0:128.2.205.9<0> | |
| islpc50:1048747:1048747 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation | |
| islpc50:1048747:1048747 [0] NCCL INFO NET/IB : No device found. | |
| islpc50:1048747:1048747 [0] NCCL INFO NET/Socket : Using [0]bond0:128.2.205.9<0> | |
| islpc50:1048747:1048747 [0] NCCL INFO Using network Socket | |
| NCCL version 2.10.3+cuda11.1 | |
| islpc50:1048748:1048748 [1] NCCL INFO Bootstrap : Using bond0:128.2.205.9<0> | |
| islpc50:1048748:1048748 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation | |
| islpc50:1048749:1048749 [2] NCCL INFO Bootstrap : Using bond0:128.2.205.9<0> | |
| islpc50:1048749:1048749 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation | |
| islpc50:1048748:1048748 [1] NCCL INFO NET/IB : No device found. | |
| islpc50:1048748:1048748 [1] NCCL INFO NET/Socket : Using [0]bond0:128.2.205.9<0> | |
| islpc50:1048748:1048748 [1] NCCL INFO Using network Socket | |
| islpc50:1048749:1048749 [2] NCCL INFO NET/IB : No device found. | |
| islpc50:1048749:1048749 [2] NCCL INFO NET/Socket : Using [0]bond0:128.2.205.9<0> | |
| islpc50:1048749:1048749 [2] NCCL INFO Using network Socket | |
| islpc50:1048747:1048816 [0] NCCL INFO Could not enable P2P between dev 1(=1a000) and dev 0(=19000) | |
| islpc50:1048747:1048816 [0] NCCL INFO Could not enable P2P between dev 0(=19000) and dev 1(=1a000) | |
| islpc50:1048747:1048816 [0] NCCL INFO Could not enable P2P between dev 1(=1a000) and dev 0(=19000) | |
| islpc50:1048747:1048816 [0] NCCL INFO Could not enable P2P between dev 0(=19000) and dev 1(=1a000) | |
| islpc50:1048748:1048817 [1] NCCL INFO Could not enable P2P between dev 1(=1a000) and dev 0(=19000) | |
| islpc50:1048748:1048817 [1] NCCL INFO Could not enable P2P between dev 0(=19000) and dev 1(=1a000) | |
| islpc50:1048748:1048817 [1] NCCL INFO Could not enable P2P between dev 1(=1a000) and dev 0(=19000) | |
| islpc50:1048749:1048818 [2] NCCL INFO Could not enable P2P between dev 1(=1a000) and dev 0(=19000) | |
| islpc50:1048748:1048817 [1] NCCL INFO Could not enable P2P between dev 0(=19000) and dev 1(=1a000) | |
| islpc50:1048749:1048818 [2] NCCL INFO Could not enable P2P between dev 0(=19000) and dev 1(=1a000) | |
| islpc50:1048749:1048818 [2] NCCL INFO Could not enable P2P between dev 1(=1a000) and dev 0(=19000) | |
| islpc50:1048749:1048818 [2] NCCL INFO Could not enable P2P between dev 0(=19000) and dev 1(=1a000) | |
| islpc50:1048749:1048818 [2] NCCL INFO Trees [0] -1/-1/-1->2->1 [1] -1/-1/-1->2->1 | |
| islpc50:1048748:1048817 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 | |
| islpc50:1048749:1048818 [2] NCCL INFO Setting affinity for GPU 2 to ffffff | |
| islpc50:1048748:1048817 [1] NCCL INFO Setting affinity for GPU 1 to ffffff | |
| islpc50:1048747:1048816 [0] NCCL INFO Channel 00/02 : 0 1 2 | |
| islpc50:1048747:1048816 [0] NCCL INFO Channel 01/02 : 0 1 2 | |
| islpc50:1048747:1048816 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 | |
| islpc50:1048747:1048816 [0] NCCL INFO Setting affinity for GPU 0 to ffffff | |
| islpc50:1048748:1048817 [1] NCCL INFO Could not enable P2P between dev 1(=1a000) and dev 0(=19000) | |
| islpc50:1048748:1048817 [1] NCCL INFO Could not enable P2P between dev 1(=1a000) and dev 0(=19000) | |
| islpc50:1048748:1048817 [1] NCCL INFO Channel 00 : 1[1a000] -> 2[67000] via direct shared memory | |
| islpc50:1048748:1048817 [1] NCCL INFO Channel 01 : 1[1a000] -> 2[67000] via direct shared memory | |
| islpc50:1048749:1048818 [2] NCCL INFO Channel 00 : 2[67000] -> 0[19000] via direct shared memory | |
| islpc50:1048749:1048818 [2] NCCL INFO Channel 01 : 2[67000] -> 0[19000] via direct shared memory | |
| islpc50:1048747:1048816 [0] NCCL INFO Could not enable P2P between dev 0(=19000) and dev 1(=1a000) | |
| islpc50:1048747:1048816 [0] NCCL INFO Channel 00 : 0[19000] -> 1[1a000] via direct shared memory | |
| islpc50:1048747:1048816 [0] NCCL INFO Could not enable P2P between dev 0(=19000) and dev 1(=1a000) | |
| islpc50:1048747:1048816 [0] NCCL INFO Channel 01 : 0[19000] -> 1[1a000] via direct shared memory | |
| islpc50:1048749:1048818 [2] NCCL INFO Connected all rings | |
| islpc50:1048749:1048818 [2] NCCL INFO Channel 00 : 2[67000] -> 1[1a000] via direct shared memory | |
| islpc50:1048749:1048818 [2] NCCL INFO Channel 01 : 2[67000] -> 1[1a000] via direct shared memory | |
| islpc50:1048748:1048817 [1] NCCL INFO Connected all rings | |
| islpc50:1048747:1048816 [0] NCCL INFO Connected all rings | |
| islpc50:1048747:1048816 [0] NCCL INFO Could not enable P2P between dev 0(=19000) and dev 1(=1a000) | |
| islpc50:1048747:1048816 [0] NCCL INFO Could not enable P2P between dev 0(=19000) and dev 1(=1a000) | |
| islpc50:1048748:1048817 [1] NCCL INFO Could not enable P2P between dev 1(=1a000) and dev 0(=19000) | |
| islpc50:1048748:1048817 [1] NCCL INFO Channel 00 : 1[1a000] -> 0[19000] via direct shared memory | |
| islpc50:1048748:1048817 [1] NCCL INFO Could not enable P2P between dev 1(=1a000) and dev 0(=19000) | |
| islpc50:1048748:1048817 [1] NCCL INFO Channel 01 : 1[1a000] -> 0[19000] via direct shared memory | |
| islpc50:1048749:1048818 [2] NCCL INFO Connected all trees | |
| islpc50:1048749:1048818 [2] NCCL INFO threadThresholds 8/8/64 | 24/8/64 | 8/8/512 | |
| islpc50:1048749:1048818 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer | |
| islpc50:1048747:1048816 [0] NCCL INFO Connected all trees | |
| islpc50:1048747:1048816 [0] NCCL INFO threadThresholds 8/8/64 | 24/8/64 | 8/8/512 | |
| islpc50:1048747:1048816 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer | |
| islpc50:1048748:1048817 [1] NCCL INFO Connected all trees | |
| islpc50:1048748:1048817 [1] NCCL INFO threadThresholds 8/8/64 | 24/8/64 | 8/8/512 | |
| islpc50:1048748:1048817 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer | |
| islpc50:1048748:1048817 [1] NCCL INFO comm 0x7fafe8002fb0 rank 1 nranks 3 cudaDev 1 busId 1a000 - Init COMPLETE | |
| islpc50:1048749:1048818 [2] NCCL INFO comm 0x7f5734002fb0 rank 2 nranks 3 cudaDev 2 busId 67000 - Init COMPLETE | |
| islpc50:1048747:1048816 [0] NCCL INFO comm 0x7fc9d0002fb0 rank 0 nranks 3 cudaDev 0 busId 19000 - Init COMPLETE | |
| islpc50:1048747:1048747 [0] NCCL INFO Launch mode Parallel | |
| [islpc50:0/3] 2022-06-06 21:32:00,204 (trainer:280) INFO: 6/6epoch started | |
| [s3prl.upstream.experts] Warning: can not import s3prl.upstream.byol_a.expert: No module named 'easydict'. Pass. | |
| [s3prl.hub] Warning: can not import s3prl.upstream.byol_a.hubconf: No module named 'easydict'. Please see upstream/byol_a/README.md | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.quesst14_dtw.expert: No module named 'dtw'. Pass. | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.separation_stft.expert: No module named 'asteroid'. Pass. | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.enhancement_stft.expert: No module named 'asteroid'. Pass. | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.speech_commands.expert: No module named 'catalyst'. Pass. | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.a2a-vc-vctk.expert: No module named 'resemblyzer'. Pass. | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.voxceleb2_ge2e.expert: No module named 'sox'. Pass. | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.sv_voxceleb1.expert: No module named 'sox'. Pass. | |
| Using cache found in ./hub/s3prl_cache/1c76d6e88090f01736036b28dc995fef583f47f42662d55286332557f957609f | |
| for https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_vox_new.pt | |
| >> inserted adapters to the following layers: 0, 1, 2 | |
| * original model weights: 317,390,592 | |
| * new model weights - all: 319,757,184 | |
| * new model weights - trainable: 2,366,592 ( 0.75% of original model) | |
| [s3prl.upstream.experts] Warning: can not import s3prl.upstream.byol_a.expert: No module named 'easydict'. Pass. | |
| [s3prl.hub] Warning: can not import s3prl.upstream.byol_a.hubconf: No module named 'easydict'. Please see upstream/byol_a/README.md | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.quesst14_dtw.expert: No module named 'dtw'. Pass. | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.separation_stft.expert: No module named 'asteroid'. Pass. | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.enhancement_stft.expert: No module named 'asteroid'. Pass. | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.speech_commands.expert: No module named 'catalyst'. Pass. | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.a2a-vc-vctk.expert: No module named 'resemblyzer'. Pass. | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.voxceleb2_ge2e.expert: No module named 'sox'. Pass. | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.sv_voxceleb1.expert: No module named 'sox'. Pass. | |
| Using cache found in ./hub/s3prl_cache/1c76d6e88090f01736036b28dc995fef583f47f42662d55286332557f957609f | |
| for https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_vox_new.pt | |
| >> inserted adapters to the following layers: 0, 1, 2 | |
| * original model weights: 317,390,592 | |
| * new model weights - all: 319,757,184 | |
| * new model weights - trainable: 2,366,592 ( 0.75% of original model) | |
| [s3prl.upstream.experts] Warning: can not import s3prl.upstream.byol_a.expert: No module named 'easydict'. Pass. | |
| [s3prl.hub] Warning: can not import s3prl.upstream.byol_a.hubconf: No module named 'easydict'. Please see upstream/byol_a/README.md | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.quesst14_dtw.expert: No module named 'dtw'. Pass. | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.separation_stft.expert: No module named 'asteroid'. Pass. | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.enhancement_stft.expert: No module named 'asteroid'. Pass. | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.speech_commands.expert: No module named 'catalyst'. Pass. | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.a2a-vc-vctk.expert: No module named 'resemblyzer'. Pass. | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.voxceleb2_ge2e.expert: No module named 'sox'. Pass. | |
| [s3prl.downstream.experts] Warning: can not import s3prl.downstream.sv_voxceleb1.expert: No module named 'sox'. Pass. | |
| Using cache found in ./hub/s3prl_cache/1c76d6e88090f01736036b28dc995fef583f47f42662d55286332557f957609f | |
| for https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_vox_new.pt | |
| >> inserted adapters to the following layers: 0, 1, 2 | |
| * original model weights: 317,390,592 | |
| * new model weights - all: 319,757,184 | |
| * new model weights - trainable: 2,366,592 ( 0.75% of original model) | |
| [islpc50:0/3] 2022-06-06 21:32:07,213 (distributed:874) INFO: Reducer buckets have been rebuilt in this iteration. | |
| [islpc50:0/3] 2022-06-06 21:36:25,787 (trainer:678) INFO: 6epoch:train:1-1204batch: iter_time=2.363e-04, forward_time=0.086, loss_ctc=4.112, loss=4.112, backward_time=0.051, optim_step_time=0.004, optim0_lr0=1.552e-04, train_time=0.220 | |
| [islpc50:0/3] 2022-06-06 21:40:44,801 (trainer:678) INFO: 6epoch:train:1205-2408batch: iter_time=7.979e-05, forward_time=0.086, loss_ctc=3.838, loss=3.838, backward_time=0.052, optim_step_time=0.003, optim0_lr0=1.544e-04, train_time=0.215 | |
| [islpc50:0/3] 2022-06-06 21:45:04,292 (trainer:678) INFO: 6epoch:train:2409-3612batch: iter_time=8.071e-05, forward_time=0.086, loss_ctc=4.243, loss=4.243, backward_time=0.053, optim_step_time=0.003, optim0_lr0=1.537e-04, train_time=0.215 | |
| [islpc50:0/3] 2022-06-06 21:49:27,543 (trainer:678) INFO: 6epoch:train:3613-4816batch: iter_time=8.183e-05, forward_time=0.088, loss_ctc=4.037, loss=4.037, backward_time=0.052, optim_step_time=0.003, optim0_lr0=1.529e-04, train_time=0.218 | |
| [islpc50:0/3] 2022-06-06 21:53:45,627 (trainer:678) INFO: 6epoch:train:4817-6020batch: iter_time=8.053e-05, forward_time=0.085, loss_ctc=3.920, loss=3.920, backward_time=0.051, optim_step_time=0.003, optim0_lr0=1.522e-04, train_time=0.214 | |
| [islpc50:0/3] 2022-06-06 21:58:07,670 (trainer:678) INFO: 6epoch:train:6021-7224batch: iter_time=8.188e-05, forward_time=0.087, loss_ctc=3.811, loss=3.811, backward_time=0.052, optim_step_time=0.003, optim0_lr0=1.515e-04, train_time=0.217 | |
| [islpc50:0/3] 2022-06-06 22:02:28,898 (trainer:678) INFO: 6epoch:train:7225-8428batch: iter_time=8.150e-05, forward_time=0.086, loss_ctc=3.931, loss=3.931, backward_time=0.052, optim_step_time=0.004, optim0_lr0=1.508e-04, train_time=0.217 | |
| [islpc50:0/3] 2022-06-06 22:06:48,145 (trainer:678) INFO: 6epoch:train:8429-9632batch: iter_time=8.323e-05, forward_time=0.086, loss_ctc=4.285, loss=4.285, backward_time=0.050, optim_step_time=0.004, optim0_lr0=1.500e-04, train_time=0.215 | |
| [islpc50:0/3] 2022-06-06 22:11:09,779 (trainer:678) INFO: 6epoch:train:9633-10836batch: iter_time=8.026e-05, forward_time=0.086, loss_ctc=3.813, loss=3.813, backward_time=0.052, optim_step_time=0.003, optim0_lr0=1.494e-04, train_time=0.217 | |
| [islpc50:0/3] 2022-06-06 22:15:28,851 (trainer:678) INFO: 6epoch:train:10837-12040batch: iter_time=8.056e-05, forward_time=0.086, loss_ctc=3.816, loss=3.816, backward_time=0.052, optim_step_time=0.003, optim0_lr0=1.487e-04, train_time=0.215 | |
| [islpc50:0/3] 2022-06-06 22:19:51,154 (trainer:678) INFO: 6epoch:train:12041-13244batch: iter_time=8.233e-05, forward_time=0.087, loss_ctc=4.035, loss=4.035, backward_time=0.053, optim_step_time=0.003, optim0_lr0=1.480e-04, train_time=0.218 | |
| [islpc50:0/3] 2022-06-06 22:24:11,949 (trainer:678) INFO: 6epoch:train:13245-14448batch: iter_time=8.187e-05, forward_time=0.086, loss_ctc=3.966, loss=3.966, backward_time=0.052, optim_step_time=0.003, optim0_lr0=1.473e-04, train_time=0.216 | |
| [islpc50:0/3] 2022-06-06 22:28:32,228 (trainer:678) INFO: 6epoch:train:14449-15652batch: iter_time=8.346e-05, forward_time=0.086, loss_ctc=4.210, loss=4.210, backward_time=0.051, optim_step_time=0.004, optim0_lr0=1.467e-04, train_time=0.216 | |
| [islpc50:0/3] 2022-06-06 22:32:51,170 (trainer:678) INFO: 6epoch:train:15653-16856batch: iter_time=8.046e-05, forward_time=0.085, loss_ctc=3.757, loss=3.757, backward_time=0.052, optim_step_time=0.003, optim0_lr0=1.460e-04, train_time=0.215 | |
| [islpc50:0/3] 2022-06-06 22:37:11,808 (trainer:678) INFO: 6epoch:train:16857-18060batch: iter_time=7.927e-05, forward_time=0.086, loss_ctc=4.036, loss=4.036, backward_time=0.050, optim_step_time=0.003, optim0_lr0=1.454e-04, train_time=0.216 | |
| [islpc50:0/3] 2022-06-06 22:41:32,075 (trainer:678) INFO: 6epoch:train:18061-19264batch: iter_time=8.539e-05, forward_time=0.086, loss_ctc=3.985, loss=3.985, backward_time=0.052, optim_step_time=0.003, optim0_lr0=1.448e-04, train_time=0.216 | |
| [islpc50:0/3] 2022-06-06 22:45:55,830 (trainer:678) INFO: 6epoch:train:19265-20468batch: iter_time=8.007e-05, forward_time=0.088, loss_ctc=3.995, loss=3.995, backward_time=0.052, optim_step_time=0.003, optim0_lr0=1.441e-04, train_time=0.219 | |
| [islpc50:0/3] 2022-06-06 22:50:14,829 (trainer:678) INFO: 6epoch:train:20469-21672batch: iter_time=8.269e-05, forward_time=0.085, loss_ctc=3.735, loss=3.735, backward_time=0.052, optim_step_time=0.003, optim0_lr0=1.435e-04, train_time=0.215 | |
| [islpc50:0/3] 2022-06-06 22:54:36,835 (trainer:678) INFO: 6epoch:train:21673-22876batch: iter_time=8.108e-05, forward_time=0.087, loss_ctc=3.469, loss=3.469, backward_time=0.052, optim_step_time=0.004, optim0_lr0=1.429e-04, train_time=0.217 | |
| [islpc50:0/3] 2022-06-06 22:58:56,514 (trainer:678) INFO: 6epoch:train:22877-24080batch: iter_time=7.962e-05, forward_time=0.086, loss_ctc=3.565, loss=3.565, backward_time=0.053, optim_step_time=0.003, optim0_lr0=1.423e-04, train_time=0.215 | |
| [islpc50:0/3] 2022-06-06 23:00:18,310 (trainer:334) INFO: 6epoch results: [train] iter_time=8.914e-05, forward_time=0.086, loss_ctc=3.929, loss=3.929, backward_time=0.052, optim_step_time=0.003, optim0_lr0=1.485e-04, train_time=0.216, time=1 hour, 27 minutes and 1.34 seconds, total_count=144582, gpu_max_cached_mem_GB=6.926, [valid] loss_ctc=39.163, cer_ctc=0.184, loss_att=nan, acc=nan, cer=nan, wer=nan, loss=39.163, time=1 minute and 8.76 seconds, total_count=2784, gpu_max_cached_mem_GB=6.926, [att_plot] time=7.98 seconds, total_count=0, gpu_max_cached_mem_GB=6.926 | |
| [islpc50:0/3] 2022-06-06 23:00:48,850 (trainer:382) INFO: The best model has been updated: train.loss | |
| [islpc50:0/3] 2022-06-06 23:00:48,851 (trainer:454) INFO: The training was finished at 6 epochs | |
| /usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 7 leaked semaphore objects to clean up at shutdown | |
| warnings.warn('resource_tracker: There appear to be %d ' | |
| # Accounting: time=5384 threads=1 | |
| # Ended (code 0) at Mon Jun 6 23:00:51 EDT 2022, elapsed time 5384 seconds | |